[
https://issues.apache.org/jira/browse/LIVY-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898354#comment-16898354
]
Gustavo Martin commented on LIVY-619:
-------------------------------------
I do not think this issue is a bug.
LineBufferedStream is in charge of reading the stdout (in Unix) from the forked
process that runs spark-submit. This is done by means of a pipe between Livy
and the forked process. Without reading from stdout Livy wouldn't be able to
show logs related to the submitted application and also the forked process
could hang forever because the pipe between the forked process and Livy can
only buffer some bytes.
When spark-submit ends, the forked process dies and it closes its file
descriptor (stdout, stdin and stderr) Because stdout is attached to
LineBufferedStream by means of a pipe there is always a chance of seeing some
of those fds in status closed.
It is an ugly exception that could be handle in a different way but I do not
think it should be a bug.
> Getting java.io.IOException: Stream closed very regularly in livy logs
> ----------------------------------------------------------------------
>
> Key: LIVY-619
> URL: https://issues.apache.org/jira/browse/LIVY-619
> Project: Livy
> Issue Type: Bug
> Components: Server
> Affects Versions: 0.6.0
> Reporter: Parveen
> Priority: Major
>
> We are using livy submitting spark applications through Livys rest interface.
> Getting this exception quite frequently :
> Exception in thread "Thread-110709" java.io.IOException: Stream closed
> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:40)
>
>
> Stack is almost same in almost all of the circumstances.
>
> Have gone through the code and it seems this is harmless as this code only
> prints logs after getting it from yarn but we are getting it with a very high
> frequency.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
