I don’t know how the JVM stores JIT information but the most logical place to store it would be alongside the loaded class. If that is the case, if we are correctly using class loaders and are discarding classes when we no longer need them — and I believe we are — then we shouldn’t have a resource leak.
Vladimir, As our resident expert on all things JVM, would you like to comment? Shuyi, Is it possible that you have a statement leak? I.e. you hold statements open after you have finished with them. If so, Calcite would be unable to free up generated classes and their associated data in the JVM, and I suppose that might give the symptoms you are seeing. Julian > On Oct 3, 2016, at 12:25 AM, Shuyi Chen <[email protected]> wrote: > > I think I might have a theory for the non-heap memory leak I observed. > Calcite uses code generation when working with adapters. For each issued > query, code will be generated at runtime and get compiled and loaded. > However, the JIT compiler data and runtime class loaded info are all stored > in JVM's non-heap area which rarely get gc unless they get unloaded > explicitly. Therefore, it causes non-heap memory explosion as queries come > in over time. I've verified the leak using jstat on both cassandra schema > and reflective schema. > > Could you please let me know if I am correct and how we can fix the leak? > Thanks a lot. > > Shuyi > > On Thu, Aug 25, 2016 at 4:02 PM, Julian Hyde <[email protected]> wrote: > >> Calcite doesn’t use off-heap memory (no native code, no direct >> ByteBuffers, no Unsafe). So if there are off-heap leaks, look to Calcite’s >> libraries (e.g. the Cassandra driver) or to Calcite adapters’ use of those >> libraries. >> >> Julian >> >> >>> On Aug 25, 2016, at 3:45 PM, Shuyi Chen <[email protected]> wrote: >>> >>> Hi all, I was using Calcite to query data in Cassandra. I found that >> there >>> is memory leak in non-heap space when doing calcite queries. Has someone >>> spot such issues? Below is an example code snippet I used. Thanks a lot. >>> >>> Statement statement = someCalciteConnection.createStatement(); >>> >>> ResultSet resultSet = statement.executeQuery(query); >>> final JSONArray jsonArray = JSONUtil.convertResultSetIntoJSON( >> resultSet); >>> statement.close(); >>> >>> >>> Shuyi >>> >>> -- >>> "So you have to trust that the dots will somehow connect in your future." >> >> > > > -- > "So you have to trust that the dots will somehow connect in your future."
