unrelated probably but my 1.0.2 topology closes itself down whenever the
console times out. This may be because I installed it in /home/ubuntu user
dir. Not certain if that causes the problem or not. Unlike you I'm not
using kafka at all but after any inactivity period I return to find
supervisor, drpc, and nimbus shutdown with no errors in logs. When I
restart those processes the topology workers resume. Since mine is only a
demo it is not a crucial issue but quite annoying

On Sun, Jul 16, 2017 at 11:38 PM, Ambud Sharma <asharma52...@gmail.com>
wrote:

> Please check if you have orphan workers. Orphan workers happen when a
> topology is redeployed in a short period of time and the old workers
> haven't yet been cleaned up.
>
> Check this running ps aux|grep java or specific jar keyword if you have
> one.
>
> On Jul 14, 2017 11:17 PM, "Sreeram" <sreera...@gmail.com> wrote:
>
>> Thank you Taylor for replying.
>>
>> I checked the worker logs and there were no messages that get printed
>> once the worker goes into freeze( I had the worker logs set at ERROR
>> level ).
>>
>> Regarding the components, I have kafka spout in the topology and bolts
>> that write to HBase.
>>
>> I had been rebalancing the topology with wait time of 30seconds. Is it
>> recommended to match it with topology.message.timeout.secs ?
>>
>> Please let me know if you need any specific info.
>>
>> Thanks,
>> Sreeram
>>
>> On Sat, Jul 15, 2017 at 12:39 AM, P. Taylor Goetz <ptgo...@gmail.com>
>> wrote:
>> >> Supervisor log at the time of freeze looks like below
>> >>
>> >> 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >
>> >
>> > There are two situations where you would see those messages: When a
>> topology is first deployed, and when a worker has died and is being
>> restarted.
>> >
>> > I suspect the latter. Have you looked at the worker logs for any
>> indication that the workers might be crashing and what might be causing it?
>> >
>> > What components are involved in you’re topology?
>> >
>> > -Taylor
>> >
>> >
>> >> On Jul 12, 2017, at 5:26 AM, Sreeram <sreera...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I am observing that my storm topology intermediately freezes and does
>> >> not continue to process tuples from Kafka. This happens frequently and
>> >> when it happens this freeze lasts for 5 to 15 minutes. No content is
>> >> written to any of the worker log files during this time.
>> >>
>> >> The version of storm I use is 1.0.2 and Kafka version is 0.9.0.
>> >>
>> >> Any suggestions to solve the issue ?
>> >>
>> >> Thanks,
>> >> Sreeram
>> >>
>> >> Supervisor log at the time of freeze looks like below
>> >>
>> >> 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:47.212 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:47.712 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:48.213 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:48.713 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:49.213 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:49.713 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:50.214 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >> 2017-07-12 14:38:50.714 o.a.s.d.supervisor [INFO]
>> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
>> >>
>> >>
>> >> Thread stacks (sample)
>> >> Most of worker threads during this freeze period look like one of the
>> >> below two stack traces.
>> >>
>> >> Thread 104773: (state = BLOCKED)
>> >> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
>> >> information may be imprecise)
>> >> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
>> >> long) @bci=20, line=215 (Compiled frame)
>> >> - java.util.concurrent.SynchronousQueue$TransferStack.
>> awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode,
>> >> boolean, long) @bci=160, line=460 (Compil
>> >> ed frame)
>> >> - java.util.concurrent.SynchronousQueue$TransferStack.
>> transfer(java.lang.Object,
>> >> boolean, long) @bci=102, line=362 (Compiled frame)
>> >> - java.util.concurrent.SynchronousQueue.poll(long,
>> >> java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
>> >> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134,
>> >> line=1066 (Compiled frame)
>> >> - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.
>> concurrent.ThreadPoolExecutor$Worker)
>> >> @bci=26, line=1127 (Compiled frame)
>> >> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5,
>> >> line=617 (Compiled frame)
>> >> - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
>> >>
>> >> Thread 147495: (state = IN_NATIVE)
>> >> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
>> >> (Compiled frame; information may be imprecise)
>> >> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
>> frame)
>> >> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93
>> (Compiled frame)
>> >> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86
>> >> (Compiled frame)
>> >> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled
>> frame)
>> >> - org.apache.kafka.common.network.Selector.select(long) @bci=35,
>> >> line=425 (Compiled frame)
>> >> - org.apache.kafka.common.network.Selector.poll(long) @bci=81,
>> >> line=254 (Compiled frame)
>> >> - org.apache.kafka.clients.NetworkClient.poll(long, long) @bci=84,
>> >> line=270 (Compiled frame)
>> >> - org.apache.kafka.clients.producer.internals.Sender.run(long)
>> >> @bci=343, line=216 (Compiled frame)
>> >> - org.apache.kafka.clients.producer.internals.Sender.run() @bci=27,
>> >> line=128 (Interpreted frame)
>> >> - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
>> >
>>
>

Reply via email to