Sahil Takiar reassigned HIVE-18684:

> Race condition in RemoteSparkJobMonitor
> ---------------------------------------
>                 Key: HIVE-18684
>                 URL: https://issues.apache.org/jira/browse/HIVE-18684
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
> There is a race condition in {{RemoteSparkJobMonitor}}. Sometimes the info in 
> {{RemoteSparkJobMonitor#startMonitor.STARTED}} gets printed out, sometimes it 
> doesn't. This can be easily verified by running a qtest on 
> {{TestMiniSparkOnYarnCliDriver}} and counting the number of times {{Query 
> Hive on Spark job}} is printed vs. the number of times {{Finished 
> successfully in}} gets printed.
> The issue is that {{RemoteSparkJobMonitor}} runs every one second, and checks 
> the state of {{JobHandle}}. Depending on the state, it prints out some 
> logging info. The content of the logs contain an implicit assumption that 
> logs in the {{STARTED}} state are printed before the logs in the 
> {{SUCCEEDED}} state. However, this isn't always the case. The state 
> transitions are driven by how long the remote Spark job takes to run, and it 
> it finishes within one second then the logs in the {{STARTED}} state never 
> printed.
> This can be confusing to users, and there is key debugging information that 
> is printed in the {{STARTED}} state.

This message was sent by Atlassian JIRA

Reply via email to