Hi Ajay, There is indeed an option to use Spark as the internal execution engine of Calcite but looking into the code it seems to be work in progress so the OOM does not surprise me. There are various places in the code indicating that the work is not yet finished and if I am not wrong nobody is actively working on this part of the codebase. Any contribution on this part is very welcomed.
Best, Stamatis On Sun, Jul 21, 2019 at 5:51 PM Ajay Babu Maguluri <[email protected]> wrote: > Hello Calcite Team, > > > > I have a requirement to query multiple data sources in single SQL query, I > saw Calcite was providing this with JDBC adaptor, So here I found some > challenges while execution like, > > > > 1. This execution (like join etc.) was happening over the memory, So > this will cause OOM when big data execution. > > 2. I saw their was spark option also, If we enable this option > execution will happen over spark? > > 3. If 2nd yes when I am trying with that option I am getting exception > like, > > Caused by: java.lang.NullPointerException > > at CalciteProgram162944.bind(Unknown Source) > > at > > org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePr > epare.java:355) > > at > > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionIm > pl.java:314) > > at > > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java > :506) > > at > > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java: > 497) > > at > > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:18 > 2) > > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) > > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) > > at > > org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.jav > a:667) > > at > > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.ja > va:566) > > at > > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(Avati > caConnection.java:675) > > at > > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement > .java:156) > > ... 2 more > > > > > > 4. And please let me know how to overcome OOM when big data execution. > > > > Thanks and Regards > > Ajay Babu Maguluri. > >
