[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-24 Thread Andrey (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022593#comment-16022593 ] Andrey commented on FLINK-6613: --- Hey! Looks like "fetch.max.bytes" option is what we need. I will try it.

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-23 Thread Tzu-Li (Gordon) Tai (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022285#comment-16022285 ] Tzu-Li (Gordon) Tai commented on FLINK-6613: +1 to what [~rmetzger] mentioned. The Kafka

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-23 Thread Robert Metzger (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021747#comment-16021747 ] Robert Metzger commented on FLINK-6613: --- There are some options in the Kafka consumer (that you can

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-18 Thread Andrey (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015715#comment-16015715 ] Andrey commented on FLINK-6613: --- Hi Dmytro, The problem is that Flink creates too much objects :) When I

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-18 Thread Dmytro Shkvyra (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015569#comment-16015569 ] Dmytro Shkvyra commented on FLINK-6613: --- Hi [~dernasherbrezon], first of all root cause of this

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-17 Thread Andrey (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014647#comment-16014647 ] Andrey commented on FLINK-6613: --- Hi Dmytro. There are several problems still exist: 1) I want to fail fast

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-17 Thread Dmytro Shkvyra (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014357#comment-16014357 ] Dmytro Shkvyra commented on FLINK-6613: --- Hi [~dernasherbrezon], I have assumed that you say that you

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-17 Thread Andrey (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014307#comment-16014307 ] Andrey commented on FLINK-6613: --- -Xmx2g and ParallelGC. Default for openjdk. > OOM during reading big

[jira] [Commented] (FLINK-6613) OOM during reading big messages from Kafka

2017-05-17 Thread Dmytro Shkvyra (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014155#comment-16014155 ] Dmytro Shkvyra commented on FLINK-6613: --- [~dernasherbrezon] What JVM option and GC do you use? >