Hey all, I am running into an interesting problem and I think I am getting
to the end of my ability to troubleshoot so I thought I'd list things out
here and see if anyone has more ideas for next steps in troubleshooting.
I am running a Docker Container I built in Mesos. I can get things up and
running, things seem happy and healthy until I try to run a command with an
interpreter. At that point I am getting strange errors about connections
refused. I put the errors below (from the Notebook, log files and from Std
Err) for clarity. But the basic thing I saw was "connection refused". So I
put tcpdump on the container and went to trouble shoot what was happening.
(TCPdump below too) it looks like it's trying to connect to localhost 36365
which is the port the interpreter was started on, but after the initial
syn, it's getting a rst-ack. I've validated in netstat, and that port IS
listening on all interfaces, so I am not sure why it's providing the rst
ack.
One hunch is around the hostname that interpreter is listening. The
Hostname I connect to in the webui is zeppelin.marathon.mesos (I am using
mesos dns and haproxy-bridge) however perhaps that is causing the thrift
server to deny something that said, it's connecting to local host, and it's
not even getting to the app level (just SYN -> RST/ACK) so I am not sure
how or why that would be occurring.
I guess based on what I have seen, this SHOULD work. i.e. even though I've
only exposed the UI and the web sockets port to the client, the docker
container should be able to connect locally to any newly opened ports. The
interpreter is starting fine.. so I guess are there any other steps I
should take to try and trouble shoot?
Thanks
John
Only thing in hive interpreter log:
INFO [2015-06-16 13:07:07,150] ({Thread-0}
RemoteInterpreterServer.java[run]:95) - Starting remote interpreter server
on port 36365
tcpdump from container:
13:06:50.967696 IP 127.0.0.1.38133 > 127.0.0.1.36365: Flags [S], seq
340951329, win 65535, options [mss 65495,sackOK,TS val 300975824 ecr
0,nop,wscale 7], length 0
.R.!.........0.........
............
13:06:50.967716 IP 127.0.0.1.36365 > 127.0.0.1.38133: Flags [R.], seq 0,
ack 340951330, win 0, length 0
.......R."P....V.....
13:06:51.468191 IP 127.0.0.1.38137 > 127.0.0.1.36365: Flags [S], seq
3821372812, win 65535, options [mss 65495,sackOK,TS val 300975949 ecr
0,nop,wscale 7], length 0
.............0.........
...M........
13:06:51.468216 IP 127.0.0.1.36365 > 127.0.0.1.38137: Flags [R.], seq 0,
ack 3821372813, win 0, length 0
..........P...%t.....
13:06:51.968677 IP 127.0.0.1.38142 > 127.0.0.1.36365: Flags [S], seq
2630719687, win 65535, options [mss 65495,sackOK,TS val 300976074 ecr
0,nop,wscale 7], length 0
.............0.........
............
13:06:51.968693 IP 127.0.0.1.36365 > 127.0.0.1.38142: Flags [R.], seq 0,
ack 2630719688, win 0, length 0
..........P...Y,.....
13:06:52.469035 IP 127.0.0.1.38146 > 127.0.0.1.36365: Flags [S], seq
976891692, win 65535, options [mss 65495,sackOK,TS val 300976199 ecr
0,nop,wscale 7], length 0
::/,.........0.........
...G........
13:06:52.469052 IP 127.0.0.1.36365 > 127.0.0.1.38146: Flags [R.], seq 0,
ack 976891693, win 0, length 0
......::/-P...%W.....
Error in Logs:
INFO [2015-06-16 13:06:50,953] ({pool-1-thread-2}
SchedulerFactory.java[jobStarted]:132) - Job
paragraph_1434047295030_-1730740540 started by scheduler
remoteinterpreter_236878590
INFO [2015-06-16 13:06:50,954] ({pool-1-thread-2}
Paragraph.java[jobRun]:194) - run paragraph 20150611-132815_546208121 using
hive org.apache.zeppelin.interpreter.LazyOpenInterpreter@12aa010c
INFO [2015-06-16 13:06:50,966] ({pool-1-thread-2}
RemoteInterpreterProcess.java[reference]:107) - Run interpreter process
/zeppelin/bin/interpreter.sh -d /zeppelin/interpreter/hive -p 36365
ERROR [2015-06-16 13:06:56,023] ({Thread-35}
RemoteScheduler.java[getStatus]:226) - Can't get status information
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:138)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:224)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:183)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 8 more
Error in Std Err:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:135)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:249)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:202)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:296)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:138)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:133)
... 12 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 19 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 20 more
Error in the Notebook:
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:135)
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:249)
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:202)
org.apache.zeppelin.scheduler.Job.run(Job.java:170)
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:296)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
