Re: Heartbeat timeout with Python spout

[cogumelosmaravilha]

Yes, but if you submit to Storm, it's ok, with or without an empty Kafka.

On 23-06-2016 10:40, ram kumar wrote:

Yes, The same topology, when I run in single node cluster, it works. And when there is no data to consume from kafka, it shows heartbeat timeout error.

Here, I am testing in multi node cluster and kafka server is in different node

On Thu, Jun 23, 2016 at 1:48 PM, cogumelosmaravilha <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>> wrote:

    Hi,

    It happens to me to, but only when my kafka is empy.
    I'm using Petrel because the generated jar file is really small
    like 300k Petrel vs 16M Streamparse.


    On 23-06-2016 08:05, ram kumar wrote:

    Hi,
    *
    *
    *Version:
    *
        Storm : 0.10.0
        Streamparse : 2.1.4


    I am running a storm topology with a python streamparse "sparse run".
    This topology stops executing in the middle and throw an exception

        158031 [pool-37-thread-1] ERROR b.s.s.ShellSpout - Halting
        process: ShellSpout died.
        java.lang.RuntimeException: *subprocess heartbeat timeout*
            at
        
backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261)
        [storm-core-0.10.0.jar:0.10.0]
            at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [?:1.8.0_40]
            at
        java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [?:1.8.0_40]
            at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
        158036 [pool-37-thread-1] ERROR b.s.d.executor -
        java.lang.RuntimeException: subprocess heartbeat timeout
            at
        
backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261)
        [storm-core-0.10.0.jar:0.10.0]
            at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [?:1.8.0_40]
            at
        java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [?:1.8.0_40]
            at
        
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [?:1.8.0_40]
            at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]



    This occurs randomly
    I can't able to trace back to the problem

    Maybe if spout takes too long to process, then streamparse can't
    acknowledge the heartbeat in time

    changed "supervisor.worker.timeout.secs" from 30 to 600

    Still the topology breaks. Is there any other options here?


    Thanks,
    Ram.