Re: “Failed to construct kafka consumer” using Apache Drill
To increase your open file limits on your system, please refer to the content on below link. Let us know if that resolved it for you. https://easyengine.io/tutorials/linux/increase-open-files-limit/ Thanks, Khurram On Tue, Jun 11, 2019 at 10:45 PM Khurram Faraaz wrote: > Hi, > > By looking at the stack trace, the issue seems to not originate from > Drill, instead it is a Kafka issue, did you try to increase the limit on > the OS to avoid the Too many files open error ? > What version of Apache Drill, Kafka and OS are you trying this on ? > > Caused by: org.apache.kafka.common.KafkaException: > java.io.IOException: Too many open files > at org.apache.kafka.common.network.Selector.(Selector.java:129) > ~[kafka-clients-0.11.0.1.jar:na] > at org.apache.kafka.common.network.Selector.(Selector.java:156) > ~[kafka-clients-0.11.0.1.jar:na] > at org.apache.kafka.common.network.Selector.(Selector.java:160) > ~[kafka-clients-0.11.0.1.jar:na] > at > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:701) > ~[kafka-clients-0.11.0.1.jar:na] > ... 25 common frames omitted > Caused by: java.io.IOException: Too many open files > at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) > ~[na:1.8.0_181] > at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:130) > ~[na:1.8.0_181] > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:69) > ~[na:1.8.0_181] > at sun.nio.ch > .EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) > ~[na:1.8.0_181] > at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.8.0_181] > at org.apache.kafka.common.network.Selector.(Selector.java:127) > ~[kafka-clients-0.11.0.1.jar:na] > > Thanks, > Khurram > > On Tue, Jun 11, 2019 at 2:38 PM Aravind Voruganti < > aravind.vorugant...@gmail.com> wrote: > >> Hello Apache Drill support, >> >> I am Aravind Voruganti a Java developer and a big fan of Apache Drill. >> Below is the issue I was facing with my new application when using Drill >> over Kafka for querying. I have also detailed the issue in Stackoverflow >> which can found at: >> >> >> https://stackoverflow.com/questions/56549783/failed-to-construct-kafka-consumer-using-apache-drill >> >> I am using the Apache Drill (1.14) JDBC driver in my application which >> consumes the data from the Kafka. The application works just fine for some >> time and after few iterations it fails to execute due to the following Too >> many files open issue. I made sure there are no file handle leaks in my >> code but still nor sure why this issue is happening? >> >> It looks like the issue is happening from with-in the Apache drill >> libraries when constructing the Kafka consumer. Can any one please guide >> me >> help this problem fixed? >> >> The problem perishes when I restart my Apache drillbit but very soon it >> happens again. I did check the file descriptor count on my unix machine >> using ulimit -a | wc -l & lsof -a -p | wc -l before and after the >> drill process restart and it seems the drill process is considerably >> taking >> a lot of file descriptors. I tried increasing the file descriptor count on >> the system but still no luck. >> >> I have followed the Apache Drill storage plugin documentation in >> configuring the Kafka plugin into Apache Drill at >> https://drill.apache.org/docs/kafka-storage-plugin/ >> >> Any help on this issue is highly appreciated. Thanks. >> >> JDBC URL: jdbc:drill:drillbit=localhost:31010;schema=kafka >> >> NOTE: I am pushing down the filters in my query SELECT * FROM myKafkaTopic >> WHERE kafkaMsgTimestamp > 1560210931626 >> >> org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: >> Failed to fetch start/end offsets of the topic myKafkaTopic >> >> Failed to construct kafka consumer >> >> [Error Id: 73f896a7-09d4-425b-8cd5-f269c3a6e69a ] >> at >> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) >> ~[drill-common-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.store.kafka.KafkaGroupScan.init(KafkaGroupScan.java:198) >> [drill-storage-kafka-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.store.kafka.KafkaGroupScan.(KafkaGroupScan.java:98) >> [drill-storage-kafka-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.store.kafka.KafkaStoragePlugin.getPhysicalScan(KafkaStoragePlugin.java:83) >> [drill-storage-kafka-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:111) >> [drill-java-exec-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99) >> [drill-java-exec-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:89) >> [drill-java-exec-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:69) >> [drill-java-exec-1.14.0.jar:1.14.0] >> at >> org.apache.drill.exec.planner.logical.DrillScanRel
Re: “Failed to construct kafka consumer” using Apache Drill
Hi, By looking at the stack trace, the issue seems to not originate from Drill, instead it is a Kafka issue, did you try to increase the limit on the OS to avoid the Too many files open error ? What version of Apache Drill, Kafka and OS are you trying this on ? Caused by: org.apache.kafka.common.KafkaException: java.io.IOException: Too many open files at org.apache.kafka.common.network.Selector.(Selector.java:129) ~[kafka-clients-0.11.0.1.jar:na] at org.apache.kafka.common.network.Selector.(Selector.java:156) ~[kafka-clients-0.11.0.1.jar:na] at org.apache.kafka.common.network.Selector.(Selector.java:160) ~[kafka-clients-0.11.0.1.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:701) ~[kafka-clients-0.11.0.1.jar:na] ... 25 common frames omitted Caused by: java.io.IOException: Too many open files at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) ~[na:1.8.0_181] at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:130) ~[na:1.8.0_181] at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:69) ~[na:1.8.0_181] at sun.nio.ch .EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[na:1.8.0_181] at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.8.0_181] at org.apache.kafka.common.network.Selector.(Selector.java:127) ~[kafka-clients-0.11.0.1.jar:na] Thanks, Khurram On Tue, Jun 11, 2019 at 2:38 PM Aravind Voruganti < aravind.vorugant...@gmail.com> wrote: > Hello Apache Drill support, > > I am Aravind Voruganti a Java developer and a big fan of Apache Drill. > Below is the issue I was facing with my new application when using Drill > over Kafka for querying. I have also detailed the issue in Stackoverflow > which can found at: > > > https://stackoverflow.com/questions/56549783/failed-to-construct-kafka-consumer-using-apache-drill > > I am using the Apache Drill (1.14) JDBC driver in my application which > consumes the data from the Kafka. The application works just fine for some > time and after few iterations it fails to execute due to the following Too > many files open issue. I made sure there are no file handle leaks in my > code but still nor sure why this issue is happening? > > It looks like the issue is happening from with-in the Apache drill > libraries when constructing the Kafka consumer. Can any one please guide me > help this problem fixed? > > The problem perishes when I restart my Apache drillbit but very soon it > happens again. I did check the file descriptor count on my unix machine > using ulimit -a | wc -l & lsof -a -p | wc -l before and after the > drill process restart and it seems the drill process is considerably taking > a lot of file descriptors. I tried increasing the file descriptor count on > the system but still no luck. > > I have followed the Apache Drill storage plugin documentation in > configuring the Kafka plugin into Apache Drill at > https://drill.apache.org/docs/kafka-storage-plugin/ > > Any help on this issue is highly appreciated. Thanks. > > JDBC URL: jdbc:drill:drillbit=localhost:31010;schema=kafka > > NOTE: I am pushing down the filters in my query SELECT * FROM myKafkaTopic > WHERE kafkaMsgTimestamp > 1560210931626 > > org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: > Failed to fetch start/end offsets of the topic myKafkaTopic > > Failed to construct kafka consumer > > [Error Id: 73f896a7-09d4-425b-8cd5-f269c3a6e69a ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) > ~[drill-common-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.store.kafka.KafkaGroupScan.init(KafkaGroupScan.java:198) > [drill-storage-kafka-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.store.kafka.KafkaGroupScan.(KafkaGroupScan.java:98) > [drill-storage-kafka-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.store.kafka.KafkaStoragePlugin.getPhysicalScan(KafkaStoragePlugin.java:83) > [drill-storage-kafka-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:111) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:89) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:69) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:62) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:38) > [drill-java-exec-1.14.0.jar:1.14.0] > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanne
