[jira] [Commented] (DRILL-4177) select * from table;Node ran out of Heap memory, exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded
[ https://issues.apache.org/jira/browse/DRILL-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940493#comment-15940493 ] Muhammad Gelbana commented on DRILL-4177: - [~arina], I followed your comment from [here|https://issues.apache.org/jira/browse/DRILL-4696?focusedCommentId=15351403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15351403]. (I understand that "pushing down query filters" means that Drill would let the datasource perform the data filtering, instead of receiving loads of data and performing the filtering by itself, please correct me if I'm wrong. Now you're saying that If MySQL (or any datasource) is configured to burst out huge amounts of data when selected, then Drill will attempt to push down query filters (and joins ?) ? I see you've only configured the MySQL connection to enable useCursorFetch and set the default fetch size to 10K. If that's what you meant, then how would that influence Drill to push down query filters ? Is there any other way that would configure Drill to push down query and join filters ? > select * from table;Node ran out of Heap memory, > exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded > --- > > Key: DRILL-4177 > URL: https://issues.apache.org/jira/browse/DRILL-4177 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.3.0 > Environment: drill1.3 jdk7 >Reporter: david_hudavy > Labels: patch > Original Estimate: 12h > Remaining Estimate: 12h > > 0: jdbc:drill:zk=local> select * from table; > Node ran out of Heap memory, exiting. > java.lang.OutOfMemoryError: GC overhead limit exceeded > at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308) > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) > at > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) > at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105) > at > org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230) > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4177) select * from table;Node ran out of Heap memory, exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded
[ https://issues.apache.org/jira/browse/DRILL-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353432#comment-15353432 ] Parth Chandra commented on DRILL-4177: -- [~arina] see also the Mysql driver guide - https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html (see the section on ResultSet) (and the JDBC API reference: http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setFetchSize(int)) > select * from table;Node ran out of Heap memory, > exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded > --- > > Key: DRILL-4177 > URL: https://issues.apache.org/jira/browse/DRILL-4177 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.3.0 > Environment: drill1.3 jdk7 >Reporter: david_hudavy > Labels: patch > Original Estimate: 12h > Remaining Estimate: 12h > > 0: jdbc:drill:zk=local> select * from table; > Node ran out of Heap memory, exiting. > java.lang.OutOfMemoryError: GC overhead limit exceeded > at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308) > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) > at > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) > at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105) > at > org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230) > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4177) select * from table;Node ran out of Heap memory, exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded
[
https://issues.apache.org/jira/browse/DRILL-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351280#comment-15351280
]
Arina Ielchiieva commented on DRILL-4177:
-
Adding configuration properties ?useCursorFetch=true=1 [1]
helps to solve OOM
Here is some guidelines:
1. Before setting up storage plugin for MySql user needs to add
mysql-connector-java-XXX-bin.jar to /jars/3rdparty/. Please mind cursor fetch configuration works from
5.0.0 version [2].
2. MySql version should be > 5.0.2.
3. Add properties to storage plugin configuration:
{
"type": "jdbc",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://host:port?useCursorFetch=true=1",
"username": "XXX",
"password": "XXX",
"enabled": true
}
Fetch size can be adjusted.
[1]
https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html
[2] https://dev.mysql.com/downloads/connector/j/
> select * from table;Node ran out of Heap memory,
> exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded
> ---
>
> Key: DRILL-4177
> URL: https://issues.apache.org/jira/browse/DRILL-4177
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.3.0
> Environment: drill1.3 jdk7
>Reporter: david_hudavy
> Labels: patch
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> 0: jdbc:drill:zk=local> select * from table;
> Node ran out of Heap memory, exiting.
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308)
> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463)
> at
> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032)
> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
> at
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> at
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> at
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101)
> at
> org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128)
> at
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40)
> at
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105)
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4177) select * from table;Node ran out of Heap memory, exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded
[ https://issues.apache.org/jira/browse/DRILL-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048308#comment-15048308 ] david_hudavy commented on DRILL-4177: - There are ten million rows in the table > select * from table;Node ran out of Heap memory, > exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded > --- > > Key: DRILL-4177 > URL: https://issues.apache.org/jira/browse/DRILL-4177 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.3.0 > Environment: drill1.3 jdk7 >Reporter: david_hudavy > Labels: patch > Original Estimate: 12h > Remaining Estimate: 12h > > 0: jdbc:drill:zk=local> select * from table; > Node ran out of Heap memory, exiting. > java.lang.OutOfMemoryError: GC overhead limit exceeded > at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308) > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) > at > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) > at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > at > org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101) > at > org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40) > at > org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131) > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174) > at > org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105) > at > org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230) > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
