> On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote: > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, > > line 687 > > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line687> > > > > Since we have 2 redirectors maybe also log out which one we are in.
The log contains the name of the actual thread (stdout-redir-1 or stderr-redir-1). > On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote: > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, > > line 695 > > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line695> > > > > Since we have 2 redirectors maybe also log out which one we are in. The log contains the name of the actual thread (stdout-redir-1 or stderr-redir-1). > On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote: > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, > > line 708 > > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line708> > > > > Wouldn't lineBuilder.indexOf(String.valueOf('\n')) work as well? We have to search for '\n' all the time so it might worth having a bit more complex code for performance. What do you think? > On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote: > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, > > line 671 > > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line671> > > > > What would happen if the child process is killed while we are inside > > this while loop (so after the BufferedReader#ready check)? Wouldn't we get > > a stream closed exception on line 674? It is a good point. I'll add a code part to the catch clause of run() so in any case we will flush the lines from the buffer. - Gabor ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51895/#review149035 ----------------------------------------------------------- On Sept. 14, 2016, 4:54 p.m., Gabor Szadovszky wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51895/ > ----------------------------------------------------------- > > (Updated Sept. 14, 2016, 4:54 p.m.) > > > Review request for hive, Chaoyu Tang, Naveen Gangam, and Barna Zsombor Klara. > > > Repository: hive-git > > > Description > ------- > > HIVE-14714 - Finishing Hive on Spark causes "java.io.IOException: Stream > closed" > > > Diffs > ----- > > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java > e8ca42aa22f0b312e009bea19e39adc8bd31e2b4 > > Diff: https://reviews.apache.org/r/51895/diff/ > > > Testing > ------- > > As the modification result is related to logging and the spark job submission > it would require too much efforts to create unit tests. > > Tested manually by "highjacking" $SPARK_HOME/bin/spark-submit script to > reproduce the following scenarios: > - The submit process does not exit after the RemoteDriver stopped > - Generating some output for less time than the actual redirector timeout > - Generating output for more time than the actual redirector timeout > - The submit process ends properly after the RemoteDriver stopped > > Expected behavior: After ending the actual session the client exits > immediately (beeline). All the stdout/stderr of the RemoteDriver are captured > properly in the hive.log until the redirector timeout. > > > Thanks, > > Gabor Szadovszky > >
