Hi,
I show the all log on this error.There are 2 type error. The 1 is at the
first run. The 2 is at the second run. Seems like could not established to
Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift
server?
The notebook script was just
%pyspark
print 'aaa'
Any advice appreciate.
1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
Job 20180302-135047_99102973 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
pyspark
RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
[2018-03-05 13:11:14,010] ({pool-2-thread-2}
InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
13:11:14,023] ({pool-2-thread-2}
RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
-d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
-p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
20180302-135047_99102973 is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
pyspark
RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job
failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused), result:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
2018-03-02 15:07 GMT+09:00 小野圭二 <onoke...@gmail.com>:
> Hi Jeff,
>
> No one graved 63039 so far on Mac.
> Yes, this issue has been reproduced as changing the port NO randomly.
>
> Ono
>
>
> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
>>
>> It is possible that 63039 is used by other processes, as Mac can not
>> detect whether this port is used or not from my experience.
>>
>> And can you reproduce this issue consistently ? Because each time
>> zeppelin will chose a random port as long as you don't specify the
>> portRange, so if you reproduce it consistently, it might due to other
>> issues.
>>
>>
>>
>>
>> 小野圭二 <onoke...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>
>>> Hi there,
>>>
>>> I am so confusing now, let me know why if You know this, please.
>>>
>>> The env:
>>> Mac: High Sierra ver.10.13.3
>>> Zep: 0.7.3
>>> Spark: 2.2.0
>>> Java: 1.8
>>> notebook script: %pyspark print 'aaa' (just it for
>>> test)
>>>
>>> In this env, i got the error in log as
>>>
>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>>
>>>
>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>> prepares a Port for callback to Thrift.
>>>
>>>
>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>> - Thrift server for callback will start. Port: 63039
>>>
>>>
>>> I guess, the reason why the error was Zeppelin could not attache to the
>>> Thrift PORT.
>>>
>>>
>>> And my confusion is, when i run the same module on CentOS, it worked
>>> fine.
>>> CentOS: 7
>>> Zep: same
>>> Spark: same
>>> Java: same
>>> notebook script: same
>>>
>>> And there was no the Thift port description in the log file. From this
>>> consequence, the fine working was cause of did not use Thrift port on
>>> CentOS.
>>>
>>>
>>> Do You know actually there is different behavior between Mac and CentOS?
>>> And i am so appreciated if You know the workaround on Mac.
>>>
>>>
>>> Best,
>>> Ono
>>>
>>
>
