[jira] [Commented] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator
[
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277844#comment-16277844
]
ASF GitHub Bot commented on DRILL-5922:
---
Github user ilooner commented on the issue:
https://github.com/apache/drill/pull/1023
@paul-rogers Can you take another pass at reviewing this? I have resolved
conflicts with the graceful shutdown work. The graceful shutdown tests were
also timing out on my laptop so I fixed them.
> Intermittent Memory Leaks in the ROOT allocator
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> child allocators.
> Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
> child allocators: 1
> Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf)
> 1048576/0/0/268435456 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> ledgers: 0
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157)
> [test-classes/:na]
> at
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:148)
> [test-classes/:na]
> at
> org.apache.drill.exec.DrillSeparatePlanningTest.getFragmentsHelper(DrillSeparatePlanningTest.java:185)
> [test-classes/:na]
> at
> org.apache.drill.exec.DrillSeparatePlanningTest.testMultiMinorFragmentComplexQuery(DrillSeparatePlanningTest.java:108)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.interna
[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.
[
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277597#comment-16277597
]
Arina Ielchiieva commented on DRILL-6003:
-
Could be that tests on Abhishek environment are run in parallel and they delete
jars for those that are currently executed? Initially each test had its own
folders.
> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Affects Versions: 1.12.0
>Reporter: Abhishek Girish
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)):
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
> ~[classes/:na]
> at
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567)
> ~[classes/:na]
> at
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338)
> ~[test-classes/:na]
> at
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147)
> ~[test-classes/:na]
> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139)
> ~[test-classes/:na]
> at
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
> ~[test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireCh
[jira] [Commented] (DRILL-4708) connection closed unexpectedly
[
https://issues.apache.org/jira/browse/DRILL-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277565#comment-16277565
]
Chun Chang commented on DRILL-4708:
---
[~karthikm] Yes, the test case provided above frequently fails in our
automation run.
> connection closed unexpectedly
> --
>
> Key: DRILL-4708
> URL: https://issues.apache.org/jira/browse/DRILL-4708
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
>Affects Versions: 1.7.0
>Reporter: Chun Chang
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Attachments: data.tgz
>
>
> Running DRILL functional automation, we often see query failed randomly due
> to the following unexpected connection close error.
> {noformat}
> Execution Failures:
> /root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/10rows/filter5.q
> Query:
> select * from dfs.ctas_flatten.`filter5_10rows_ctas`
> Failed with exception
> java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <-->
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly.
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
> at
> oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
> at
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
> at
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION
> ERROR: Connection /10.10.100.171:36185 <-->
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly.
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
> at
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> at
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.
[
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277542#comment-16277542
]
Timothy Farkas commented on DRILL-6003:
---
[~arina] The temp directories are created at the beginning of the
TestDynamicUDFSupport class . The directories themselves are not deleted, but
they are cleared after each test run on line 119.
> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Affects Versions: 1.12.0
>Reporter: Abhishek Girish
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)):
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
> ~[classes/:na]
> at
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567)
> ~[classes/:na]
> at
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338)
> ~[test-classes/:na]
> at
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147)
> ~[test-classes/:na]
> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139)
> ~[test-classes/:na]
> at
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
> ~[test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractCha
[jira] [Commented] (DRILL-5702) Jdbc Driver Class not found
[
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277410#comment-16277410
]
Vitalii Diravka commented on DRILL-5702:
Thank you [~korlawulki] for finding this issue.
I have reproduced the issue with pre-built Drill jdbc driver. And also noticed
that with manual built driver the issue is gone.
The problem is related to the lack of
[oadd.|https://github.com/apache/drill/blob/master/exec/jdbc-all/pom.xml#L544]
prefix for different profiles except default one.
> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Cannot connect to drill cluster after upgrade to new Jar
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
> not found
> at
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (DRILL-5702) Jdbc Driver Class not found
[
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277204#comment-16277204
]
Holger Kiel edited comment on DRILL-5702 at 12/4/17 6:29 PM:
-
Please have a look at file "profile.props" in drill-jdbc-all-1.12.0.jar. This
is part of the content:
{code}
package.namespace.prefix=${package.namespace.prefix}
{code}
When I'm building everything from the repository, my jdbc Driver has the
following instead:
{code}
package.namespace.prefix=oadd.
{code}
The latter works for me.
was (Author: korlawulki):
Please have a look at file "profile.props" in drill-jdbc-all-1.12.0.jar. This
is part of the content:
{code}
package.namespace.prefix=${package.namespace.prefix}
{code}
When I'm building everything from the repository, my jdbc Driver hat the
following instead:
{code}
package.namespace.prefix=oadd.
{code}
The latter works for me.
> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Cannot connect to drill cluster after upgrade to new Jar
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
> not found
> at
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5702) Jdbc Driver Class not found
[
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277204#comment-16277204
]
Holger Kiel commented on DRILL-5702:
Please have a look at file "profile.props" in drill-jdbc-all-1.12.0.jar. This
is part of the content:
{code}
package.namespace.prefix=${package.namespace.prefix}
{code}
When I'm building everything from the repository, my jdbc Driver hat the
following instead:
{code}
package.namespace.prefix=oadd.
{code}
The latter works for me.
> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Cannot connect to drill cluster after upgrade to new Jar
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
> not found
> at
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.
[
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277152#comment-16277152
]
Arina Ielchiieva commented on DRILL-6003:
-
It looks like concurrency issue. It might be that folder got deleted when
another unit test from {{TestDynamicUDFSupport}} family finished execution.
[~timothyfarkas] since you have changed folders for the tests, could you please
confirm that each test has its unique folder for staging / temporary and
registry areas?
> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Affects Versions: 1.12.0
>Reporter: Abhishek Girish
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)):
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
> ~[classes/:na]
> at
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567)
> ~[classes/:na]
> at
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338)
> ~[test-classes/:na]
> at
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147)
> ~[test-classes/:na]
> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139)
> ~[test-classes/:na]
> at
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
> ~[test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abs
[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.
[
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277136#comment-16277136
]
Abhishek Girish commented on DRILL-6003:
[~vitalii], can you please check if the stack trace shared above is helpful? I
did not set the verbose flag, but found this on a different run and looks
verbose enough :)
> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Affects Versions: 1.12.0
>Reporter: Abhishek Girish
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)):
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException:
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
> ~[classes/:na]
> at
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567)
> ~[classes/:na]
> at
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338)
> ~[test-classes/:na]
> at
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
> ~[test-classes/:na]
> at
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147)
> ~[test-classes/:na]
> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139)
> ~[test-classes/:na]
> at
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
> ~[test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR:
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102)
> ~[classes/:na]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
> ~[drill-rpc-1.12.0.jar:1.12.0]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fi
[jira] [Updated] (DRILL-6000) Graceful Shutdown Unit Tests Should Not Be Run On Travis
[ https://issues.apache.org/jira/browse/DRILL-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6000: Fix Version/s: (was: 1.13.0) 1.12.0 > Graceful Shutdown Unit Tests Should Not Be Run On Travis > > > Key: DRILL-6000 > URL: https://issues.apache.org/jira/browse/DRILL-6000 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.12.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas > Labels: ready-to-commit > Fix For: 1.12.0 > > > Currently these tests fail on Travis. But since they are heavy weight tests > which test a corner case, they should not be run as part of the smoke tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.
[
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277134#comment-16277134
]
Abhishek Girish commented on DRILL-6003:
A different stack trace
{code}
Running
org.apache.drill.TestDynamicUDFSupport#testLazyInitWhenDynamicUdfSupportIsDisabled
Nov 27, 2017 3:48:32 AM org.apache.calcite.sql.validate.SqlValidatorException
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: No match found
for function signature custom_lower()
Nov 27, 2017 3:48:32 AM org.apache.calcite.runtime.CalciteException
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column
8 to line 1, column 24: No match found for function signature
custom_lower()
Nov 27, 2017 3:48:33 AM org.apache.calcite.sql.validate.SqlValidatorException
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: No match found
for function signature custom_lower()
Nov 27, 2017 3:48:33 AM org.apache.calcite.runtime.CalciteException
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column
8 to line 1, column 24: No match found for function signature
custom_lower()
Running org.apache.drill.vector.TestToNullable
Running org.apache.drill.vector.TestToNullable#testNullable
03:48:33.057 [25e4046e-3de0-aab2-f524-74e5eb4473a3:foreman] ERROR
o.a.d.e.e.f.FunctionImplementationRegistry - Problem during remote functions
load from DrillUDF-1.0.jar
org.apache.drill.exec.exception.JarValidationException: Marker file
drill-module.conf is missing in DrillUDF-1.0.jar
at
org.apache.drill.exec.expr.fn.FunctionImplementationRegistry.scan(FunctionImplementationRegistry.java:443)
~[classes/:na]
at
org.apache.drill.exec.expr.fn.FunctionImplementationRegistry.syncWithRemoteRegistry(FunctionImplementationRegistry.java:367)
~[classes/:na]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:81)
[classes/:na]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1007)
[classes/:na]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:279)
[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_131]
Running org.apache.drill.vector.TestToNullable#testVariableWidth
Running org.apache.drill.vector.TestToNullable#testFixedWidth
Nov 27, 2017 3:48:33 AM org.apache.calcite.sql.validate.SqlValidatorException
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: No match found
for function signature custom_lower()
Nov 27, 2017 3:48:33 AM org.apache.calcite.runtime.CalciteException
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column
8 to line 1, column 24: No match found for function signature
custom_lower()
{code}
{code}
03:48:34.015 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 B(1
B), h: 220.5 MiB(509.2 MiB), nh: -51.4 KiB(72.8 MiB)):
testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
org.apache.drill.exec.rpc.RpcException:
org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: From
line 1, column 8 to line 1, column 24: No match found for function signature
custom_lower()
SQL Query null
[Error Id: 7aa44c96-6f6c-4112-88ce-d98d2021a535 on 10.10.104.57:31010]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
~[drill-rpc-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:858)
~[classes/:na]
at
org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:560)
~[classes/:na]
at
org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338)
~[test-classes/:na]
at
org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
~[test-classes/:na]
at
org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
~[test-classes/:na]
at
org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
~[test-classes/:na]
at
org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147)
~[test-classes/:na]
at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139)
~[test-classes/:na]
at
org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_131]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0
[jira] [Updated] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
[ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] salim achouche updated DRILL-5846: -- Reviewer: Parth Chandra > Improve Parquet Reader Performance for Flat Data types > --- > > Key: DRILL-5846 > URL: https://issues.apache.org/jira/browse/DRILL-5846 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.11.0 >Reporter: salim achouche >Assignee: salim achouche > Labels: performance > Fix For: 1.13.0 > > > The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to > further improve the Parquet Reader performance as several users reported that > Parquet parsing represents the lion share of the overall query execution. It > tracks Flat Data types only as Nested DTs might involve functional and > processing enhancements (e.g., a nested column can be seen as a Document; > user might want to perform operations scoped at the document level that is no > need to span all rows). Another JIRA will be created to handle the nested > columns use-case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5996) Unable to re-run queries from Profiles tab with impersonation and without authentication
[ https://issues.apache.org/jira/browse/DRILL-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5996: Fix Version/s: 1.13.0 > Unable to re-run queries from Profiles tab with impersonation and without > authentication > - > > Key: DRILL-5996 > URL: https://issues.apache.org/jira/browse/DRILL-5996 > Project: Apache Drill > Issue Type: Bug > Components: Client - CLI >Affects Versions: 1.12.0 >Reporter: Krystal >Assignee: Arina Ielchiieva > Fix For: 1.13.0 > > > On drillbit, enable impersonation without authentication. From WebUI, click > on a query from Profiles link -> click on the "Edit Query" tab -> click on > the "Re-Run Query" button. The query would fail with the following error: > "HTTP ERROR 412 > Problem accessing /query. Reason: > User-Name header is not set" > We should also provide a username dialog box for this tab. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DRILL-5996) Unable to re-run queries from Profiles tab with impersonation and without authentication
[ https://issues.apache.org/jira/browse/DRILL-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5996: --- Assignee: Arina Ielchiieva > Unable to re-run queries from Profiles tab with impersonation and without > authentication > - > > Key: DRILL-5996 > URL: https://issues.apache.org/jira/browse/DRILL-5996 > Project: Apache Drill > Issue Type: Bug > Components: Client - CLI >Affects Versions: 1.12.0 >Reporter: Krystal >Assignee: Arina Ielchiieva > > On drillbit, enable impersonation without authentication. From WebUI, click > on a query from Profiles link -> click on the "Edit Query" tab -> click on > the "Re-Run Query" button. The query would fail with the following error: > "HTTP ERROR 412 > Problem accessing /query. Reason: > User-Name header is not set" > We should also provide a username dialog box for this tab. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277006#comment-16277006
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154122358
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276997#comment-16276997
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154114792
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -221,11 +200,15 @@ public QueryManager getQueryManager() {
}
/**
- * Cancel the query. Asynchronous -- it may take some time for all
remote fragments to be
- * terminated.
+ * Cancel the query. Asynchronous -- it may take some time for all
remote fragments to be terminated.
+ * For planning and enqueued states we cancel immediately since these
states are done locally.
+ *
+ * Note this can be called from outside of run() on another thread, or
after run() completes
*/
public void cancel() {
-// Note this can be called from outside of run() on another thread, or
after run() completes
+if (QueryState.PLANNING == state || QueryState.ENQUEUED == state) {
+ moveToState(QueryState.CANCELLATION_REQUESTED, null);
--- End diff --
Fixed.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277001#comment-16277001
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154116755
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan)
throws ExecutionSetupExcep
queryManager.setTotalCost(plan.totalCost());
work.applyPlan(drillbitContext.getPlanReader());
logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId,
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
-logger.debug("Submitting fragments to run.");
-
-// set up the root fragment first so we'll have incoming buffers
available.
-setupRootFragment(rootPlanFragment, work.getRootOperator());
-
-setupNonRootFragments(planFragments);
-
-moveToState(QueryState.RUNNING, null);
-logger.debug("Fragments running.");
+admit();
}
- private void admit(QueryWorkUnit work) throws ForemanSetupException {
+ private void admit() throws ForemanSetupException {
queryManager.markPlanningEndTime();
-try {
- queryRM.admit();
-} catch (QueueTimeoutException e) {
- throw UserException
- .resourceError()
- .message(e.getMessage())
- .build(logger);
-} catch (QueryQueueException e) {
- throw new ForemanSetupException(e.getMessage(), e);
-} finally {
- queryManager.markQueueWaitEndTime();
-}
-moveToState(QueryState.STARTING, null);
+planningQueries.dec();
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNI
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276995#comment-16276995
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153461906
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277002#comment-16277002
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153458672
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan)
throws ExecutionSetupExcep
queryManager.setTotalCost(plan.totalCost());
work.applyPlan(drillbitContext.getPlanReader());
logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId,
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
--- End diff --
1. Agree. I will create single function to accept all of these.
2. `work` object is applicable only when we run query / prepared statement
/ plans. For the cases when run fragment, work object is not used at all [1].
Featured introduced in DRILL-4132 [2].
[1]
https://github.com/arina-ielchiieva/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java#L473
[2] https://issues.apache.org/jira/browse/DRILL-4132
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once que
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277009#comment-16277009
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154119336
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276996#comment-16276996
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153456718
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -256,12 +239,14 @@ public void run() {
// track how long the query takes
queryManager.markStartTime();
-enqueuedQueries.dec();
runningQueries.inc();
+queryText = queryRequest.getPlan();
+recordNewState(QueryState.PLANNING);
--- End diff --
`getPlan` is simple getter and cannot throw en exception. It was moved up
to display query string on Web UI when we have entered PLANNING and other
states rather then having empty string, please see screenshot in Jira.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277012#comment-16277012
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154679303
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
---
@@ -288,8 +287,10 @@ public void setCost(double cost) {
}
@Override
-public void admit() throws QueueTimeoutException, QueryQueueException {
- lease = rm.queue().enqueue(foreman.getQueryId(), queryCost);
+public void admit() {
+ QueueAcquirer queueAcquirer = new QueueAcquirer(rm.queue(), foreman,
queryCost);
+ queueAcquirerThread = new Thread(queueAcquirer);
--- End diff --
Admit method is moved back to be blocking.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277008#comment-16277008
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154117721
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277005#comment-16277005
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154117281
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
}
}
- private void moveToState(final QueryState newState, final Exception
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState,
- exception);
+ public synchronized void moveToState(final QueryState newState, final
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState, exception);
switch (state) {
-case ENQUEUED:
- switch (newState) {
- case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+ case PLANNING:
+switch (newState) {
+ case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
+return;
+ case CANCELLATION_REQUESTED:
+assert exception == null;
+recordNewState(newState);
+foremanResult.setCompleted(QueryState.CANCELED);
+foremanResult.close();
+return;
+ case FAILED:
+assert exception != null;
+recordNewState(newState);
+foremanResult.setFailed(exception);
+foremanResult.close();
+return;
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLA
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277013#comment-16277013
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154680084
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
---
@@ -314,13 +315,49 @@ public void exit() {
lease = null;
}
+@Override
+public void cancel() {
+ if (queueAcquirerThread != null) {
+queueAcquirerThread.interrupt();
+ }
+ foreman.moveToState(QueryState.CANCELED, null);
+}
+
@Override
public boolean hasQueue() { return true; }
@Override
public String queueName() {
return lease == null ? null : lease.queueName();
}
+
+/**
+ * Is used to start query enqueue process in separate thread.
+ * Changes query state depending on the result.
+ */
+private class QueueAcquirer implements Runnable {
+
+ private final QueryQueue queryQueue;
+ private final Foreman foreman;
+ private final double queryCost;
+
+ QueueAcquirer(QueryQueue queryQueue, Foreman foreman, double
queryCost) {
+this.queryQueue = queryQueue;
+this.foreman = foreman;
+this.queryCost = queryCost;
+ }
+
+ @Override
+ public void run() {
+try {
+ queryQueue.enqueue(foreman.getQueryId(), queryCost);
+ foreman.moveToState(QueryState.STARTING, null);
+} catch (Exception e) {
--- End diff --
Logic is removed.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277004#comment-16277004
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154114531
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -177,9 +155,10 @@ public Foreman(final WorkerBee bee, final
DrillbitContext drillbitContext,
queryManager = new QueryManager(queryId, queryRequest,
drillbitContext.getStoreProvider(),
drillbitContext.getClusterCoordinator(), this);
-recordNewState(QueryState.ENQUEUED);
-enqueuedQueries.inc();
+recordNewState(QueryState.PREPARING);
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277003#comment-16277003
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154117496
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
}
}
- private void moveToState(final QueryState newState, final Exception
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState,
- exception);
+ public synchronized void moveToState(final QueryState newState, final
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState, exception);
switch (state) {
-case ENQUEUED:
- switch (newState) {
- case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+ case PLANNING:
+switch (newState) {
+ case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
+return;
+ case CANCELLATION_REQUESTED:
+assert exception == null;
+recordNewState(newState);
+foremanResult.setCompleted(QueryState.CANCELED);
+foremanResult.close();
+return;
+ case FAILED:
+assert exception != null;
+recordNewState(newState);
+foremanResult.setFailed(exception);
+foremanResult.close();
+return;
+}
+break;
+ case ENQUEUED:
+enqueuedQueries.dec();
+queryManager.markQueueWaitEndTime();
+switch (newState) {
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277011#comment-16277011
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154120750
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276998#comment-16276998
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153462132
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276994#comment-16276994
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153460494
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276999#comment-16276999
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153462719
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java
---
@@ -280,29 +280,29 @@ public void interrupted(final InterruptedException
ex) {
}
void updateEphemeralState(final QueryState queryState) {
- // If query is already in zk transient store, ignore the transient
state update option.
- // Else, they will not be removed from transient store upon
completion.
- if (!inTransientStore &&
-
!foreman.getQueryContext().getOptions().getOption(ExecConstants.QUERY_TRANSIENT_STATE_UPDATE))
{
-return;
- }
+// If query is already in zk transient store, ignore the transient
state update option.
+// Else, they will not be removed from transient store upon completion.
+if (!inTransientStore &&
!foreman.getQueryContext().getOptions().getOption(ExecConstants.QUERY_TRANSIENT_STATE_UPDATE))
{
+ return;
+}
- switch (queryState) {
+switch (queryState) {
+ case PREPARING:
+ case PLANNING:
case ENQUEUED:
case STARTING:
case RUNNING:
case CANCELLATION_REQUESTED:
runningProfileStore.put(stringQueryId, getQueryInfo()); // store
as ephemeral query profile.
inTransientStore = true;
break;
-
case COMPLETED:
case CANCELED:
case FAILED:
try {
runningProfileStore.remove(stringQueryId);
inTransientStore = false;
-} catch(final Exception e) {
+} catch (final Exception e) {
--- End diff --
Fixed. I believe it was meant to be `stored`.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states wer
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277010#comment-16277010
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154116803
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
}
}
- private void moveToState(final QueryState newState, final Exception
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState,
- exception);
+ public synchronized void moveToState(final QueryState newState, final
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState, exception);
switch (state) {
-case ENQUEUED:
- switch (newState) {
- case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+ case PLANNING:
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276993#comment-16276993
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153454792
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/QueryWorkUnit.java ---
@@ -112,4 +115,46 @@ public void applyPlan(PhysicalPlanReader reader)
throws ForemanSetupException {
fragments.add(defn.applyPlan(reader));
}
}
+
+ /**
+ * Converts list of stores fragments into json,
+ * in case of exception returns text indicating that json was malformed.
+ * Is used for debugging purposes.
+ *
+ * @return fragments json
+ */
+ public String convertFragmentToJson() {
+StringBuilder stringBuilder = new StringBuilder();
+final int fragmentCount = fragments.size();
+int fragmentIndex = 0;
+for (final PlanFragment planFragment : fragments) {
+ final ExecProtos.FragmentHandle fragmentHandle =
planFragment.getHandle();
+ stringBuilder.append("PlanFragment(");
--- End diff --
Agree. Renamed.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277007#comment-16277007
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r154117233
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
}
}
- private void moveToState(final QueryState newState, final Exception
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState,
- exception);
+ public synchronized void moveToState(final QueryState newState, final
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}",
state, newState, exception);
switch (state) {
-case ENQUEUED:
- switch (newState) {
- case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+ case PLANNING:
+switch (newState) {
+ case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
--- End diff --
Done.
> Canceling a query hung in planning state, leaves the query in ENQUEUED state
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit:
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ','
> AS CSV_DATA from (values(1));
> +--+
> | |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root
> cause is that during planning time DrillFuncHolderExpr return type is
> extensively used to determine matching function, matching type etc. Though
> this type is retrieved via
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
> in reality complex logic is executed beaneath it. For example for [concat
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
> Since function return type can not be changes during DrillFuncHolderExpr
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING /
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking,
> making foreman wait till enqueueing is done Making it non-blocking will
> prevent consuming threads that just sit idle in a busy system and also is
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued,
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
[
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277000#comment-16277000
]
ASF GitHub Bot commented on DRILL-5963:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1051#discussion_r153461286
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and
remotely).
+ */
+public class FragmentsRunner {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+ private static final ControlsInjector injector =
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+ private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+ private final WorkerBee bee;
+ private final UserClientConnection initiatingClient;
+ private final DrillbitContext drillbitContext;
+ private final Foreman foreman;
+
+ private List planFragments;
+ private PlanFragment rootPlanFragment;
+ private FragmentRoot rootOperator;
+
+ public FragmentsRunner(WorkerBee bee, UserClientConnection
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+ }
+
+ public WorkerBee getBee() {
+return bee;
+ }
+
+ public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+ }
+
+ public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+ }
+
+ public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+ }
+
+ /**
+ * Submits root and non-root fragments fragments for running.
+ * In
[jira] [Commented] (DRILL-6006) Label current is missing on Web UI since the same drillbits with different status are considered to be different
[
https://issues.apache.org/jira/browse/DRILL-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276737#comment-16276737
]
Arina Ielchiieva commented on DRILL-6006:
-
FYI [~Paul.Rogers], [~priteshm] [~vdonapati]
> Label current is missing on Web UI since the same drillbits with different
> status are considered to be different
>
>
> Key: DRILL-6006
> URL: https://issues.apache.org/jira/browse/DRILL-6006
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
> Fix For: 1.13.0
>
>
> After DRILL-4286 label current is missing on Web UI indicating current
> drillbit.
> This happens because when creating DrillbitInfo object, we check [if current
> drillbit is the same as
> available|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java#L197].
> But {{dbContext.getEndpoint()}} returns drillbit with status {{startup}} and
> {{work.getContext().getAvailableBits()}} returns drillbit with status
> {{online}}, thus isCurrent variable is set as false but should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-6006) Label current is missing on Web UI since the same drillbits with different status are considered to be different
Arina Ielchiieva created DRILL-6006:
---
Summary: Label current is missing on Web UI since the same
drillbits with different status are considered to be different
Key: DRILL-6006
URL: https://issues.apache.org/jira/browse/DRILL-6006
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.12.0
Reporter: Arina Ielchiieva
Fix For: 1.13.0
After DRILL-4286 label current is missing on Web UI indicating current drillbit.
This happens because when creating DrillbitInfo object, we check [if current
drillbit is the same as
available|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java#L197].
But {{dbContext.getEndpoint()}} returns drillbit with status {{startup}} and
{{work.getContext().getAvailableBits()}} returns drillbit with status
{{online}}, thus isCurrent variable is set as false but should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276652#comment-16276652
]
Arina Ielchiieva edited comment on DRILL-6005 at 12/4/17 12:48 PM:
---
FYI [~Paul.Rogers], [~priteshm]
was (Author: arina):
FYI [~Paul.Rogers]
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
> Fix For: 1.13.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276652#comment-16276652
]
Arina Ielchiieva commented on DRILL-6005:
-
FYI [~Paul.Rogers]
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
> Fix For: 1.13.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
Arina Ielchiieva created DRILL-6005:
---
Summary: Fix TestGracefulShutdown tests to skip check for loopback
address usage in distributed mode
Key: DRILL-6005
URL: https://issues.apache.org/jira/browse/DRILL-6005
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.12.0
Reporter: Arina Ielchiieva
Assignee: Venkata Jyothsna Donapati
Fix For: 1.13.0
After DRILL-4286 changes some of the newly added unit tests fail with
{noformat}
Drillbit is disallowed to bind to loopback address in distributed mode.
{noformat}
List of failed tests:
{noformat}
Tests in error:
TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
set...
TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture setup
...
TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
fail...
TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
se...
{noformat}
This can be fixed if {{/etc/hosts}} file is edited.
Source -
https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
Though these changes are required on production during running unit tests I
don't think this check should be enforced.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5845) Columns returned by select with "ORDER BY" and "LIMIT" clauses are not in correct order.
[
https://issues.apache.org/jira/browse/DRILL-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276611#comment-16276611
]
Vitalii Diravka commented on DRILL-5845:
[~rhou] Dir0/dir1/dir2 should appear as the first few columns:
https://drill.apache.org/docs/querying-directories/
But is other columns ordering correct? Could you provide test case? But anyway
if column ordering is wrong for querying directories it can be other issue,
since the bug will be placed in other operator.
> Columns returned by select with "ORDER BY" and "LIMIT" clauses are not in
> correct order.
>
>
> Key: DRILL-5845
> URL: https://issues.apache.org/jira/browse/DRILL-5845
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Column order is proper for queries with only one clause: ORDER BY or LIMIT.
> For queries with both these clauses column order isn't preserved.
> Test case for reproduce:
> {code}
> 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` limit 1;
> +--+--+--+--+
> | n_nationkey | n_name | n_regionkey | n_comment
> |
> +--+--+--+--+
> | 0| ALGERIA | 0| haggle. carefully final deposits
> detect slyly agai |
> +--+--+--+--+
> 1 row selected (0.181 seconds)
> 0: jdbc:drill:zk=local> select * from cp.`tpch/nation.parquet` order by
> n_name limit 1;
> +--+--+--+--+
> | n_comment | n_name |
> n_nationkey | n_regionkey |
> +--+--+--+--+
> | haggle. carefully final deposits detect slyly agai | ALGERIA | 0
>| 0|
> +--+--+--+--+
> 1 row selected (0.154 seconds)
> {code}
> For json files the column ordering is not preserved as well:
> {code}
> select * from cp.`employee.json` limit 1;
> select * from cp.`employee.json` order by full_name limit 1;
> {code}
> Perhaps the wrong operator for sorting is used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
