Can you please log a JIRA case for this? If you can create a PR with a test case, even better.
> On Oct 6, 2016, at 11:14 AM, Shuyi Chen <[email protected]> wrote: > > Hi Julian, > > > This is how I use calcite and get non-heap memory leak. Could you please > take a look if I missed something? > > > Statement statement = > cassandraConnectionFactory.getCalciteConnection().createStatement(); > ResultSet resultSet = statement.executeQuery(query); > final JSONArray jsonArray = JSONUtil.*convertResultSetIntoJSON*(resultSet); > statement.close(); > > > Also, to reproduce the problem, you can follow this post ( > https://martinsdeveloperworld.wordpress.com/2015/08/24/apache-calcite-setting-up-your-own-in-memory-database-with-sql-interface/) > to reproduce the problem easily on ReflectiveSchema by increasing the > iteration count, and use jstat to observe. > > > Thanks a lot. > > Shuyi > > On Mon, Oct 3, 2016 at 1:17 PM, Julian Hyde <[email protected]> wrote: > >> I don’t know how the JVM stores JIT information but the most logical place >> to store it would be alongside the loaded class. If that is the case, if we >> are correctly using class loaders and are discarding classes when we no >> longer need them — and I believe we are — then we shouldn’t have a resource >> leak. >> >> Vladimir, As our resident expert on all things JVM, would you like to >> comment? >> >> Shuyi, Is it possible that you have a statement leak? I.e. you hold >> statements open after you have finished with them. If so, Calcite would be >> unable to free up generated classes and their associated data in the JVM, >> and I suppose that might give the symptoms you are seeing. >> >> Julian >> >>> On Oct 3, 2016, at 12:25 AM, Shuyi Chen <[email protected]> wrote: >>> >>> I think I might have a theory for the non-heap memory leak I observed. >>> Calcite uses code generation when working with adapters. For each issued >>> query, code will be generated at runtime and get compiled and loaded. >>> However, the JIT compiler data and runtime class loaded info are all >> stored >>> in JVM's non-heap area which rarely get gc unless they get unloaded >>> explicitly. Therefore, it causes non-heap memory explosion as queries >> come >>> in over time. I've verified the leak using jstat on both cassandra schema >>> and reflective schema. >>> >>> Could you please let me know if I am correct and how we can fix the leak? >>> Thanks a lot. >>> >>> Shuyi >>> >>> On Thu, Aug 25, 2016 at 4:02 PM, Julian Hyde <[email protected]> wrote: >>> >>>> Calcite doesn’t use off-heap memory (no native code, no direct >>>> ByteBuffers, no Unsafe). So if there are off-heap leaks, look to >> Calcite’s >>>> libraries (e.g. the Cassandra driver) or to Calcite adapters’ use of >> those >>>> libraries. >>>> >>>> Julian >>>> >>>> >>>>> On Aug 25, 2016, at 3:45 PM, Shuyi Chen <[email protected]> wrote: >>>>> >>>>> Hi all, I was using Calcite to query data in Cassandra. I found that >>>> there >>>>> is memory leak in non-heap space when doing calcite queries. Has >> someone >>>>> spot such issues? Below is an example code snippet I used. Thanks a >> lot. >>>>> >>>>> Statement statement = someCalciteConnection.createStatement(); >>>>> >>>>> ResultSet resultSet = statement.executeQuery(query); >>>>> final JSONArray jsonArray = JSONUtil.convertResultSetIntoJSON( >>>> resultSet); >>>>> statement.close(); >>>>> >>>>> >>>>> Shuyi >>>>> >>>>> -- >>>>> "So you have to trust that the dots will somehow connect in your >> future." >>>> >>>> >>> >>> >>> -- >>> "So you have to trust that the dots will somehow connect in your future." >> >> > > > -- > "So you have to trust that the dots will somehow connect in your future."
