Hello Hoppers, Converting SQL into distinct operations is something that was not just done in the glorious past of our tool but is happening in Apache Beam, Spark and other frameworks. Including the old code from said past is just not a good idea (HOP-2956).
I would suggest that we start having the discussion to (also) move to using Apache Calcite (great project from Julian btw) to achieve the following: 1. Implement the metadata ability to expose the output of a pipeline transform as a SQL table (or re-use the new Web Service metadata) 2. Implement a multi-table SQL parser with very basic (or no) function support 3. Implement the ability to use the output of the SQL parser to generate pipeline metadata and then to execute the generated pipeline to stream the data back to a SQL service on Hop Server. 4. Implement a JDBC driver on top of the SQL service. 5. Investigate how we can implement push-down optimisation on top of Calcite. Please think of your old use-cases. Did you ever use the old thin client? Do you have any specific requirements or tools to support? Let us know here or add them to HOP-2957. Thanks, Matt
