[
https://issues.apache.org/jira/browse/HIVE-19636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552208#comment-16552208
]
Vineet Garg commented on HIVE-19636:
------------------------------------
I have been bumping into TestMiniDruidCliDriver timeout. This test is run in
three batches and the batch with druidmini_dynamic_partition is timing out. I
looked at corresponding hive logs (attached) and i see the following error
while running {{SELECT sum(cint), sum(cbigint) FROM druid_max_size_partition}}
{noformat}
2018-07-21T05:35:50,587 WARN [HttpClient-Netty-Boss-0]
channel.SimpleChannelUpstreamHandler: EXCEPTION, please implement
org.apache.hive.druid.org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught()
for proper handling.
java.net.ConnectException: Connection refused: localhost/127.0.0.1:8082
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[?:1.8.0_102]
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_102]
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
~[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.druid.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[hive-druid-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
2018-07-21T05:35:50,593 ERROR [ee745c13-27f8-4940-a347-c8307a2da8be main]
CliDriver: Failed with exception
java.io.IOException:org.apache.hive.druid.io.druid.java.util.common.RE: Failure
getting results for
query[TimeseriesQuery{dataSource='default.druid_max_size_partition',
querySegmentSpec=LegacySegmentSpec{intervals=[1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z]},
descending=false, virtualColumns=[], dimFilter=null,
granularity='AllGranularity',
aggregatorSpecs=[LongSumAggregatorFactory{fieldName='cint', expression='null',
name='$f0'}, LongSumAggregatorFactory{fieldName='cbigint', expression='null',
name='$f1'}], postAggregatorSpecs=[], context={skipEmptyBuckets=true}}]
url[http://localhost:8082/druid/v2/] because of
[org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel
in resource pool]
java.io.IOException: org.apache.hive.druid.io.druid.java.util.common.RE:
Failure getting results for
query[TimeseriesQuery{dataSource='default.druid_max_size_partition',
querySegmentSpec=LegacySegmentSpec{intervals=[1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z]},
descending=false, virtualColumns=[], dimFilter=null,
granularity='AllGranularity',
aggregatorSpecs=[LongSumAggregatorFactory{fieldName='cint', expression='null',
name='$f0'}, LongSumAggregatorFactory{fieldName='cbigint', expression='null',
name='$f1'}], postAggregatorSpecs=[], context={skipEmptyBuckets=true}}]
url[http://localhost:8082/druid/v2/] because of
[org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel
in resource pool]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2722)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1339)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1313)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:171)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: org.apache.hive.druid.io.druid.java.util.common.RE: Failure getting
results for
query[TimeseriesQuery{dataSource='default.druid_max_size_partition',
querySegmentSpec=LegacySegmentSpec{intervals=[1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z]},
descending=false, virtualColumns=[], dimFilter=null,
granularity='AllGranularity',
aggregatorSpecs=[LongSumAggregatorFactory{fieldName='cint', expression='null',
name='$f0'}, LongSumAggregatorFactory{fieldName='cbigint', expression='null',
name='$f1'}], postAggregatorSpecs=[], context={skipEmptyBuckets=true}}]
url[http://localhost:8082/druid/v2/] because of
[org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel
in resource pool]
at
org.apache.hadoop.hive.druid.serde.DruidQueryRecordReader$JsonParserIterator.init(DruidQueryRecordReader.java:268)
at
org.apache.hadoop.hive.druid.serde.DruidQueryRecordReader$JsonParserIterator.hasNext(DruidQueryRecordReader.java:207)
at
org.apache.hadoop.hive.druid.serde.DruidTimeseriesQueryRecordReader.nextKeyValue(DruidTimeseriesQueryRecordReader.java:47)
at
org.apache.hadoop.hive.druid.serde.DruidTimeseriesQueryRecordReader.next(DruidTimeseriesQueryRecordReader.java:72)
at
org.apache.hadoop.hive.druid.serde.DruidTimeseriesQueryRecordReader.next(DruidTimeseriesQueryRecordReader.java:33)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569)
... 50 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel
in resource pool
at
org.apache.hive.druid.com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:186)
at
org.apache.hadoop.hive.druid.serde.DruidQueryRecordReader$JsonParserIterator.init(DruidQueryRecordReader.java:245)
... 55 more
Caused by: org.apache.hive.druid.org.jboss.netty.channel.ChannelException:
Faulty channel in resource pool
at
org.apache.hive.druid.com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:143)
at
org.apache.hive.druid.com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14)
at
org.apache.hadoop.hive.druid.serde.DruidQueryRecordReader.initialize(DruidQueryRecordReader.java:116)
at
org.apache.hadoop.hive.druid.serde.DruidQueryRecordReader.initialize(DruidQueryRecordReader.java:123)
at
org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat.getRecordReader(DruidQueryBasedInputFormat.java:295)
at
org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:776)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:344)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
... 50 more
Caused by: java.net.ConnectException: Connection refused:
localhost/127.0.0.1:8082
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.apache.hive.druid.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.hive.druid.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
After this all queries are getting connection refused from druid.
[~nishantbangarwa] [~bslim] any idea what could be the reason? How should I
further debug this?
> Fix druidmini_dynamic_partition.q slowness
> ------------------------------------------
>
> Key: HIVE-19636
> URL: https://issues.apache.org/jira/browse/HIVE-19636
> Project: Hive
> Issue Type: Sub-task
> Affects Versions: 3.1.0
> Reporter: Prasanth Jayachandran
> Priority: Major
> Attachments: hive.12762.logs.log
>
>
> druidmini_dynamic_partition.q runs for >5 mins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
