We are using a recent build post 0.81 so 0.82?
Restarting the interpreter helps in 75% of the cases, but the remaining few
just won’t recover until we shut everything down and restart all if zeppelin. I
suspect there is some other process that we aren’t identifying that needs
killing.
PAUL BRENNER
Head of Data Science
pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com
twitter @placeiq linkedin /placeiq
On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zjf...@gmail.com>, wrote:
> Which version do you use ? And does restarting interpreter work for you ?
>
> > Paul Brenner <pbren...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
> > > We run into the error pasted below semi-regularly and often the only
> > > solution is to do a full restart of zeppelin to get the user/notebook
> > > back up and running. I’m at a loss of where to even look to try to debug
> > > this or address it. Anyone have any ideas or any way to point me in the
> > > right direction?
> > >
> > > This is what the notebook shows:
> > > org.apache.thrift.transport.TTransportException at
> > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
> > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at
> > > org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> > > org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > > This is what I pulled from the log:
> > >
> > > ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) -
> > > Job failed
> > > java.lang.RuntimeException:
> > > org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > at
> > > org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > ... 11 more
> > > INFO [2019-08-30 15:41:56,954] ({Exec Default Executor}
> > > RemoteInterpreterManagedProcess.java[onProcessComplete]:243) -
> > > Interpreter process exited 0
> > > DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24}
> > > RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857,
> > > org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > at
> > > org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > > ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> > > NotebookServer.java[afterStatusChange]:2308) - Error
> > > java.lang.RuntimeException:
> > > org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > at
> > > org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > ... 11 more
> > > WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> > > NotebookServer.java[afterStatusChange]:2316) - Job
> > > 20190821-204527_149556857 is finished, status: ERROR, exception:
> > > java.lang.RuntimeException:
> > > org.apache.thrift.transport.TTransportException, result: %text
> > > org.apache.thrift.transport.TTransportException
> > > at
> > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > at
> > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > at
> > > org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > at
> > > org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > >
> > > PAUL BRENNER
> > > Head of Data Science
> > > pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > twitter @placeiq linkedin /placeiq
>
>
> --
> Best Regards
>
> Jeff Zhang
