[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408654#comment-16408654
]
Sahil Takiar commented on HIVE-18533:
-------------------------------------
[~vanzin] I'm seeing a race condition in one of the tests that invoke the
{{SparkLauncher}}
{code}
2018-03-21T14:11:26,064 ERROR [Driver-RPC-Handler-0] util.Utils: Uncaught
exception in thread Driver-RPC-Handler-0
java.lang.IllegalStateException: Disconnected.
at
org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
~[spark-launcher_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.launcher.LauncherConnection.send(LauncherConnection.java:81)
~[spark-launcher_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.launcher.LauncherBackend.setState(LauncherBackend.scala:77)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.scheduler.local.LocalSchedulerBackend.org$apache$spark$scheduler$local$LocalSchedulerBackend$$stop(LocalSchedulerBackend.scala:161)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.scheduler.local.LocalSchedulerBackend.stop(LocalSchedulerBackend.scala:137)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:508)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1752)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1924)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357)
[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.SparkContext.stop(SparkContext.scala:1923)
[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.api.java.JavaSparkContext.stop(JavaSparkContext.scala:654)
[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.hive.spark.client.JobContextImpl.stop(JobContextImpl.java:81)
[classes/:?]
at
org.apache.hive.spark.client.RemoteDriver.shutdown(RemoteDriver.java:223)
[classes/:?]
at
org.apache.hive.spark.client.RemoteDriver.access$200(RemoteDriver.java:71)
[classes/:?]
at
org.apache.hive.spark.client.RemoteDriver$DriverProtocol.handle(RemoteDriver.java:286)
[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_92]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_92]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_92]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
at
org.apache.hive.spark.client.rpc.RpcDispatcher.handleCall(RpcDispatcher.java:121)
[classes/:?]
at
org.apache.hive.spark.client.rpc.RpcDispatcher.channelRead0(RpcDispatcher.java:80)
[classes/:?]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[netty-all-4.1.17.Final.jar:4.1.17.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
{code}
Any ideas whats causing it? We invoke {{SparkAppHandle#disconnect}} in the
client as soon as the app goes into a {{RUNNING}} state. It seems that
{{#disconnect}} just closes the connection so any attempt by the
{{LauncherBackend}} to send something on the connection will fail.
> Add option to use InProcessLauncher to submit spark jobs
> --------------------------------------------------------
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch,
> HIVE-18533.3.patch, HIVE-18533.4.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a
> HoS session + debuggability (no need launch a separate process to run a Spark
> app).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
