Hi, Ajay AFAIK, the calcite-spark is not even applicable now, while the default Enumerables execution holds all the data in memory, so there is no surprise to trigger the OOM exception.
That means Calcite does not support big data set very well now, the most common way to use it is to do planning promotion or query the small data set. Best, Danny Chan 在 2019年7月21日 +0800 PM11:51，Ajay Babu Maguluri <[email protected]>，写道： > Hello Calcite Team, > > > > I have a requirement to query multiple data sources in single SQL query, I > saw Calcite was providing this with JDBC adaptor, So here I found some > challenges while execution like, > > > > 1. This execution (like join etc.) was happening over the memory, So > this will cause OOM when big data execution. > > 2. I saw their was spark option also, If we enable this option > execution will happen over spark? > > 3. If 2nd yes when I am trying with that option I am getting exception > like, > > Caused by: java.lang.NullPointerException > > at CalciteProgram162944.bind(Unknown Source) > > at > org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePr > epare.java:355) > > at > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionIm > pl.java:314) > > at > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java > :506) > > at > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java: > 497) > > at > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:18 > 2) > > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) > > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) > > at > org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.jav > a:667) > > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.ja > va:566) > > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(Avati > caConnection.java:675) > > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement > .java:156) > > ... 2 more > > > > > > 4. And please let me know how to overcome OOM when big data execution. > > > > Thanks and Regards > > Ajay Babu Maguluri. >
