Hi,
I am trying to query kafka using apache drill. Apache drill version is 1.16.0.
The authorization service is MIT Kerberos.
Query :
SHOW TABLES IN kafka
Kafka Storage plugin :
{
"type": "kafka",
"kafkaConsumerProps": {
"enable.auto.commit": "true",
"store.kafka.record.reader":
"org.apache.drill.exec.store.kafka.decoders.JsonMessageReader",
"streams.consumer.default.stream": "/twitter-data",
"value.deserializer":
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
"group.id": "drill-query-consumer-1",
"auto.offset.reset": "earliest",
"session.timeout.ms": "300",
"bootstrap.servers": "x.xx.x.xx:6667,xx.xx.xx.xx:6667",
"key.deserializer":
"org.apache.kafka.common.serialization.ByteArrayDeserializer"
},
"enabled": true
}
are there any configurations that are missing for kafka storage plugin in
kerberised cluster.
I dont know how it will read the client_jass.conf file how to provide kerberos
principle and keytab path.
Query Response :
UserRemoteException : DATA_READ ERROR: Failed to get tables information
org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: Failed
to get tables information
Failed to construct kafka consumer
Fragment 0:0
[Error Id: 73d70db9-0af7-4c9e-8ddd-0ae4f3da440f on xx.xxx.xx.xx.x.xx:31010]
Contents of drillbit.log file are :
2019-12-03 06:30:50,545 [2219fee4-8bc2-df23-2191-cf7facebac64:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
2219fee4-8bc2-df23-2191-cf7facebac64 issued by drill: show tables in kafka
2019-12-03 06:30:50,633 [2219fee4-8bc2-df23-2191-cf7facebac64:frag:0:0] INFO
o.a.d.e.s.k.s.KafkaMessageSchema - User Error Occurred: Failed to get tables
information (Failed to construct kafka consumer)
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Failed to
get tables information
Failed to construct kafka consumer
[Error Id: 73d70db9-0af7-4c9e-8ddd-0ae4f3da440f ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630)
~[drill-common-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.kafka.schema.KafkaMessageSchema.getTableNames(KafkaMessageSchema.java:81)
[drill-storage-kafka-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.AbstractSchema.getTableNamesAndTypes(AbstractSchema.java:299)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator$Tables.visitTables(InfoSchemaRecordGenerator.java:340)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:254)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:247)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:234)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146)
[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142)
[drill-java-exec-1.16.0.jar:1.16.0]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_151]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_151]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
[hadoop-common-2.7.4.jar:na]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:110)
[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:263)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.16.0.jar:1.16.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka
consumer
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:765)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:633)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:615)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.drill.exec.store.kafka.schema.KafkaMessageSchema.getTableNames(KafkaMessageSchema.java:77)
[drill-storage-kafka-1.16.0.jar:1.16.0]
... 24 common frames omitted
Caused by: java.lang.IllegalArgumentException: Heartbeat must be set lower than
the session timeout
at
org.apache.kafka.clients.consumer.internals.Heartbeat.<init>(Heartbeat.java:39)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.<init>(AbstractCoordinator.java:142)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.<init>(ConsumerCoordinator.java:113)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:722)
~[kafka-clients-0.11.0.1.jar:na]
... 27 common frames omitted
2019-12-03 06:30:50,633 [2219fee4-8bc2-df23-2191-cf7facebac64:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2219fee4-8bc2-df23-2191-cf7facebac64:0:0:
State change requested AWAITING_ALLOCATION --> FAILED
2019-12-03 06:30:50,633 [2219fee4-8bc2-df23-2191-cf7facebac64:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2219fee4-8bc2-df23-2191-cf7facebac64:0:0:
State change requested FAILED --> FINISHED
2019-12-03 06:30:50,635 [2219fee4-8bc2-df23-2191-cf7facebac64:frag:0:0] WARN
o.a.d.exec.rpc.control.WorkEventBus - Fragment
2219fee4-8bc2-df23-2191-cf7facebac64:0:0 manager is not found in the work bus.
2019-12-03 06:30:50,638 [qtp289525290-53] ERROR
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed: {}
org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: Failed
to get tables information
Failed to construct kafka consumer
Fragment 0:0
[Error Id: 73d70db9-0af7-4c9e-8ddd-0ae4f3da440f on xx.xx.xx.xx:31010]
at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:126)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:90)
~[drill-java-exec-1.16.0.jar:1.16.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jersey-common-2.25.1.jar:na]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
[jersey-server-2.25.1.jar:na]
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
[jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
[jersey-container-servlet-core-2.25.1.jar:na]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
[jersey-container-servlet-core-2.25.1.jar:na]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
[jersey-container-servlet-core-2.25.1.jar:na]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
[jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
[jetty-servlet-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[jetty-servlet-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
[jetty-security-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.apache.drill.exec.server.rest.auth.DrillHttpSecurityHandlerProvider.handle(DrillHttpSecurityHandlerProvider.java:151)
[drill-java-exec-1.16.0.jar:1.16.0]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
[jetty-servlet-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.server.Server.handle(Server.java:539)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[jetty-io-9.3.25.v20180904.jar:9.3.25.v20180904]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[jetty-io-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[jetty-io-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[jetty-util-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[jetty-util-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[jetty-util-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[jetty-util-9.3.25.v20180904.jar:9.3.25.v20180904]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[jetty-util-9.3.25.v20180904.jar:9.3.25.v20180904]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:765)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:633)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:615)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.drill.exec.store.kafka.schema.KafkaMessageSchema.getTableNames(KafkaMessageSchema.java:77)
~[drill-storage-kafka-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.AbstractSchema.getTableNamesAndTypes(AbstractSchema.java:299)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator$Tables.visitTables(InfoSchemaRecordGenerator.java:340)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:254)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:247)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:234)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
~[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146)
~[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142)
~[drill-java-exec-1.16.0.jar:1.16.0]
at .......(:0) ~[na:na]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
~[hadoop-common-2.7.4.jar:na]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:110)
~[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:263)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
~[drill-common-1.16.0.jar:1.16.0]
at .......(:0) ~[na:na]
Caused by: java.lang.IllegalArgumentException: Heartbeat must be set lower than
the session timeout
at
org.apache.kafka.clients.consumer.internals.Heartbeat.<init>(Heartbeat.java:39)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.<init>(AbstractCoordinator.java:142)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.<init>(ConsumerCoordinator.java:113)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:722)
~[kafka-clients-0.11.0.1.jar:na]
... 24 common frames omitted
2019-12-03 06:30:50,643 [2219fee4-8bc2-df23-2191-cf7facebac64:frag:0:0] WARN
o.a.d.e.w.f.QueryStateProcessor - Dropping request to move to COMPLETED state
as query is already at FAILED state (which is terminal).
contents of drillbit_queries.json :
{"queryId":"2219d5ff-d7f2-54ce-7609-8c6a730dc22d","schema":"","queryText":"show
tables in
kafka","start":1575365119214,"finish":1575365119246,"outcome":"FAILED","username":"drill","remoteAddress":"xx.xx.xx.xxx:51570"}
Let me know if i am missing any configurations for kafka storage plugin. It
will be a big help.
Thanks and Regards
Rameshwar Mane
