Hi Willie, Can you please continue the discussion on Github? Here's a link: https://github.com/jupyter-incubator/sparkmagic/issues/339
Thanks! On Wednesday, March 8, 2017 at 2:48:37 PM UTC-8, William Kupersanin wrote: > > Hello All, > > I am trying to debug an issue in a notebook running the Pyspark kernel > where the notebook will execute some cells but then freeze after a certain > point. I think that messaging is getting screwed up between Pyspark and > Livy. When the last cell is executed, I will see this on the client side. > > > 2017-03-08 22:24:48,505 INFO EventsHandler InstanceId: > 0e1c8fd2-047e-4337-b264-5b64ba74de5a,EventName: > notebookStatementExecutionStart,Timestamp: 2017-03-08 > 22:24:48.504920,SessionGuid: 03d14478-6adc-4b > 34-abef-b9b6fd400543,LivyKind: pyspark,SessionId: 8,StatementGuid: > f1933b11-b767-4a18-b311-c48901ad8369 > 2017-03-08 22:24:48,788 DEBUG Command Status of statement 8 is running. > 2017-03-08 22:24:50,920 DEBUG Command Status of statement 8 is running. > > ...and it never comes back. > > On the livy end, I see > > 17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO > scheduler.DAGScheduler: ResultStage 17 (collect at <stdin>:5) finished in > 1.521 s > 17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO > scheduler.DAGScheduler: Job 8 finished: collect at <stdin>:5, took 3.729078 > s > 17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Registered > outstanding rpc 230 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult). > 17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type > com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes) > 17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type > com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult (91 bytes) > 17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type > com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes) > 17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type > com.cloudera.livy.rsc.rpc.Rpc$NullMessage (2 bytes) > 17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Received RPC > message: type=REPLY id=230 payload=com.cloudera.livy.rsc.rpc.Rpc$NullMessage > 17/03/08 17:26:28 DEBUG RpcDispatcher: [ClientProtocol] Registered > outstanding rpc 231 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult). > > ad infinitum > > So, with my limited knowledge, it looks to me that Livy thinks it has sent > a result to a finished job, but pyspark hasn't received it. > Anyone seen this before? Any thoughts? > > Thanks! > --Willie > > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/6a45ed8d-9529-4073-9f29-a71aec5ae52a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
