Cody, Thanks for the message. I’ve been a fan of LINQ for a long time (you’ll see there is a module in Calcite called linq4j), LINQ Reactive Extensions, and RxJava. And I’m always interested how database-centric and progamming-language-centric meet.
If you do option 2, and work on the AST, you would be doing a lot of extra work. If you’re working bottom up you should at least go as far as Calcite’s relational algebra (RelNode/RexNode). SQL -> parser -> SqlNode -> validate -> sql2rel -> RelNode. This handles a good deal of the complexity of SQL. The logical RelNode tree will have relational operations (filter, project, aggregate) that probably map directly onto RxJava constructs. The biggest choice you have to make is whether you make an Rx Observable an adapter (and use an existing engine), or whether you use Rx as an engine. By “engine” I mean a framework that has implementations of all of the relational operators and SQL functions. Calcite’s interpreter is an engine (simple but not very efficient) and so are Samza, Flink and Storm/Trident; in future there will be Beam. The engines are not very complete yet, but that will change. With either approach you’d have to write a table adapter, so that Calcite’s SQL validator can see your Observable, know what columns it has, and so forth. If you made Rx an engine, then you would implement queries by generating Rx code. You’d end up writing classes such as RxFilter, RxProject, RxAggregate etc. They’d look fairly similar to EnumerableFilter, EnumerableProject etc. If you used another engine, say Calcite’s interpreter, then you’d just need to write an implementation of the Table adapter. It would operate similarly to CsvTest.testCsvStream. Did that help? Julian > On Jul 25, 2016, at 3:17 PM, Cody Rioux <[email protected]> wrote: > > Hey Calcite Developers, > > I've got a handful of data streams in RxJava Observables over which we're > attempting to provide SQL querying capabilities. I'm planning to implement a > limited subset of functionality to begin with, and Rx Observables support > many > SQL or SQL like operations. My original thought was to parse the SQL using > something like ANTRL4 / Calcite and then walk the returned AST / result > modifying an rx.Observable as I go, however I see that Calcite has the > concept > of an adapter to hook the SQL into various back ends. > > The objective is as follows: > > - Begin with an Observable data stream. > - Apply a series of transformations / filters / etc... specified by SQL > - Return the result to the end user as another Rx Observable which > can then be used seamlessly within this environment. > > In your opinion would it be more proper to: > > 1. Implement an RxObservable adapter to enable streaming SQL support on Rx > Observables. > > 2. Parse the SQL string using the Calcite parser and then manually walk the > SqlNode result to manipulate an rx.Observable representing the data > stream? > > > Option 1 sounds a lot like LINQ SQL from the C#/.NET stack, and I'm > surprised > I haven't found an SQL over Rx Observables implementation online though I > may > have overlooked something. I'm unaware of how much work this would take > though > looking over the streaming CSV stuff I found [0][1][2] it seems as though > I'd > be extending a handful of classes from the org.apache.calcite.streaming > namespace. Would it be wrong to assume that we get a lot of functionality > for > free by implementing an adapter? > > Option 2 appears more straightforward to me at this point in time, though > that is likely only due to both; the fact that I can easily envision how > to implement this, and I had intended to use the Rx operators to perform > the actual work. > > In either case I'm seeking advice about which road you think I should travel > and any prior work, documents, etc... to which I can refer to get started. > > Thanks! > > [0] > http://mail-archives.apache.org/mod_mbox/calcite-dev/201605.mbox/%[email protected]%3E > [1] > https://git1-us-west.apache.org/repos/asf?p=calcite.git;a=commit;h=65c1cec22d93a83b73bbdb8c73d08d8642db46e1 > [2] > https://github.com/apache/calcite/tree/master/example/csv/src/main/java/org/apache/calcite/adapter/csv
