Gerald's last comment/request says it all: > > If it turns out to be too complicated to make the memory consumption more predicatble or even bounded to some configurable limit, then I need at least as a workaround a way to turn off the HashJoin strategy completely: I did it by patching and building my own derby.jar, but if there would be an official solution with some kind of switch like a system property, it would be great! >
You'll be pleased to know its not complicated at all!!! -:) Please refer to the 'Tuning Derby' manual. The section 'About the Optimizer's Choice of Join Strategy' will explain why you are running out of memory. Also - did you notice this line in the OptimizerFactoryImpl code? 'protected int maxMemoryPerTable = 1048576;' Then around line 100 or so you will also notice that the 'boot' method looks for a system property 'Optimizer.MAX_MEMORY_PER_TABLE' that can be used to set the limit for a table. The comment block before this code states: /* ** This property determines the maximum size of memory (in KB) ** the optimizer can use for each table. If an access path takes ** memory larger than that size for a table, the access path is skipped. ** Default is 1024 (KB). */ Thus by combining the information found in the manual with the comments found in the code we can conclude the following: 1. If you don't change the setting from the default of 1MB the optimizer may choose a nested join strategy for a table. 2. If the table is actually bigger than 1MB you have fooled the optimizer and it may well run out of memory. My comment referring you to the tuning manual is not meant to be the type of RTFM comment you often see in other forums. There is a great team of people doing a lot of work on the documentation to make it even better and more useful. We want people to report problems like the one you reported because it may well indeed be a problem that needs fixed. As Jack implies though, when a problem is reported it is very helpful to have as much information as possible: stack trace, query, system information, etc. As Derby gets wider distribution and more people use it in different ways it may point out the need to highlight some of the system properties that can help people configure Derby for their use. Thanks for highlighting this issue! ----- Original Message ----- From: "Jack Klebanoff (JIRA)" <[email protected]> To: <[email protected]> Sent: Friday, December 17, 2004 9:18 AM Subject: [jira] Commented: (DERBY-106) HashJoinStrategy leads to java.lang.OutOfMemoryError > [ http://nagoya.apache.org/jira/browse/DERBY-106?page=comments#action_56830 ] > > Jack Klebanoff commented on DERBY-106: > -------------------------------------- > > Do you know whether the memory overflow occurred during compilation or execution? Do you have a stack trace from the OutOfMemoryError? Do you know the sizes of the tables being joined? > > > HashJoinStrategy leads to java.lang.OutOfMemoryError > > ---------------------------------------------------- > > > > Key: DERBY-106 > > URL: http://nagoya.apache.org/jira/browse/DERBY-106 > > Project: Derby > > Type: Bug > > Reporter: Gerald Khin > > > > > My application is running out of memory: I encounterd a java.lang.OutOfMemoryError. I used -Xmx256M. Unfortunatley, I cannot spend an arbitrary amount of JVM memory. > > Then, I commented out the line in class OptimizerFactoryImpl which was adding the HashJoinStrategy to the set of Join strategies: > > if (joinStrategySet == null) > > { > > // JoinStrategy[] jss = new JoinStrategy[2]; > > JoinStrategy[] jss = new JoinStrategy[1]; > > jss[0] = new NestedLoopJoinStrategy(); > > // jss[1] = new HashJoinStrategy(); > > joinStrategySet = jss; > > } > > And with these changes the OutOfMemoryError has gone away! And it works even with -Xmx128M!!! > > So I guess that there is a major memory issue with this HashJoin strategy implementation. > > If it turns out to be too complicated to make the memory consumption more predicatble or even bounded to some configurable limit, then I need at least as a workaround a way to turn off the HashJoin strategy completely: I did it by patching and building my own derby.jar, but if there would be an official solution with some kind of switch like a system property, it would be great! > > -- > This message is automatically generated by JIRA. > - > If you think it was sent incorrectly contact one of the administrators: > http://nagoya.apache.org/jira/secure/Administrators.jspa > - > If you want more information on JIRA, or have a bug to report see: > http://www.atlassian.com/software/jira >
