Thanks Julian. My understanding on Spark Adapter is it can be the computation engine on top of various data sources. For now, I am exploring whether we can use Spark Adapter to run query on a set of Parquet files. So, knowing what it can do is important.
Regards, Chin Wei On Wed, Jan 6, 2016 at 7:26 PM, Julian Hyde <[email protected]> wrote: > As you have noticed, we are using an old version of Spark. At some point > SparkAdapterTest worked — that is, we could execute queries with a VALUES > and WHERE clause — but wasn’t made part of the suite, and it stopped > working somewhere along the way. > > So, I think the first step would be to upgrade Spark. If there is interest > in using a Spark adapter I will help you to get it working; and we can > start adding additional relational operators. > > Julian > > > On Jan 6, 2016, at 2:05 AM, Chin Wei Low <[email protected]> wrote: > > > > Hi All, > > > > I am playing around Spark Adapter and my understanding of the adapter is > it > > will convert enumerable from other adapter like CSV Adapter to RDD for > > processing in Spark and later collect the result. > > > > I have a few issues and questions: > > > > issues: > > 1. Looks like the upgrade of the Jetty server has break that as Spark 0.9 > > (used by Calcite) still using the old Jetty server. Tried to revert that > > and there are further issues. > > 2. The Spark related rules does not used by planner, there is todo for > this. > > 3. The implementation of the child's rel is null, there is todo for this. > > > > questions: > > 1. Does the Spark Adapter working or complete implemented? > > 2. I am enabling Spark Adapter by specifying spark=true in connection > > string, is there any documentation on this? > > > > Regards, > > Chin Wei > >
