i tried to use debug . got in the worker.log.err
2017-07-27 21:47:48,868 FATAL Unable to register shutdown hook because JVM
is shutting down.
and this lines from worker.log
2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-1:27, stream: __ack_ack, id: {},
[3247365064986003851 -431522470795602124]
2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-1:27, stream: __ack_ack, id: {}, [3247365064986003851
-431522470795602124]
2017-07-27 21:47:48.811 b.s.d.executor [INFO] Execute done TUPLE source:
b-1:27, stream: __ack_ack, id: {}, [3247365064986003851
-431522470795602124] TASK: 1 DELTA: 0
2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-1:29, stream: __ack_ack, id: {},
[3247365064986003851 -6442207219333745818]
2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-1:29, stream: __ack_ack, id: {}, [3247365064986003851
-6442207219333745818]
2017-07-27 21:47:48.811 b.s.d.executor [INFO] Execute done TUPLE source:
b-1:29, stream: __ack_ack, id: {}, [3247365064986003851
-6442207219333745818] TASK: 1 DELTA: 0
2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 5263752373603294688]
2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
5263752373603294688]
2017-07-27 21:47:48.868 b.s.d.worker [INFO] Shutting down worker
top-1-1501184820 9adf5f4c-dc5b-47b5-a458-40defe84fe9e 6703
2017-07-27 21:47:48.868 b.s.d.worker [INFO] Shutting down receive thread
2017-07-27 21:47:48.869 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-1:31, stream: __ack_ack, id: {}, [3247365064986003851
4288963968930353157]
2017-07-27 21:47:48.872 b.s.d.executor [INFO] Execute done TUPLE source:
b-1:31, stream: __ack_ack, id: {}, [3247365064986003851
4288963968930353157] TASK: 1 DELTA: 60
2017-07-27 21:47:48.872 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 5240959063117469257]
2017-07-27 21:47:48.872 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
5240959063117469257]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source:
b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
5240959063117469257] TASK: 1 DELTA: 1
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 7583382518734849127]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
7583382518734849127]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source:
b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
7583382518734849127] TASK: 1 DELTA: 0
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 6840644970823833210]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
6840644970823833210]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source:
b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
6840644970823833210] TASK: 1 DELTA: 0
2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 -6463368911496394080]
2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
-6463368911496394080]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source:
b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
-6463368911496394080] TASK: 1 DELTA: 1
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {},
[3247365064986003851 764549587969230513]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851
764549587969230513]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source:
b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 764549587969230513]
TASK: 1 DELTA: 0
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-5:35, stream: __ack_ack, id: {},
[3247365064986003851 -4632707886455738545]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851
-4632707886455738545]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source:
b-5:35, stream: __ack_ack, id: {}, [3247365064986003851
-4632707886455738545] TASK: 1 DELTA: 0
2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message
FOR 1 TUPLE: source: b-5:35, stream: __ack_ack, id: {},
[3247365064986003851 2993206175355277727]
2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0
TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851
2993206175355277727]
2017-07-27 21:47:48.875 b.s.d.executor [INFO] Execute done TUPLE source:
b-5:35, stream: __ack_ack, id: {}, [3247365064986003851
2993206175355277727] TASK: 1 DELTA: 1
2017-07-27 21:47:48.898 b.s.m.n.Client [INFO] creating Netty Client,
connecting to lenovo:6703, bufferSize: 5242880
2017-07-27 21:47:48.902 b.s.m.loader [INFO] Shutting down receiving-thread:
[top-1-1501184820, 6703]
2017-07-27 21:47:48.902 b.s.m.n.Client [INFO] closing Netty Client
Netty-Client-lenovo/192.168.1.5:6703
2017-07-27 21:47:48.902 b.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-lenovo/192.168.1.5:6703
2017-07-27 21:47:48.902 b.s.m.loader [INFO] Waiting for
receiving-thread:[top-1-1501184820, 6703] to die
2017-07-27 21:47:48.903 b.s.m.loader [INFO] Shutdown receiving-thread:
[top-1-1501184820, 6703]
2017-07-27 21:47:48.904 b.s.d.worker [INFO] Shut down receive thread
2017-07-27 21:47:48.904 b.s.d.worker [INFO] Terminating messaging context
2017-07-27 21:47:48.904 b.s.d.worker [INFO] Shutting down executors
2017-07-27 21:47:48.904 b.s.d.executor [INFO] Shutting down executor b-0:[8
8]
2017-07-27 21:47:48.905 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.905 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.906 b.s.d.executor [INFO] Shut down executor b-0:[8 8]
2017-07-27 21:47:48.906 b.s.d.executor [INFO] Shutting down executor
b-8:[47 47]
2017-07-27 21:47:48.907 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.907 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shut down executor b-8:[47 47]
2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shutting down executor
b-0:[12 12]
2017-07-27 21:47:48.908 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.908 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shut down executor b-0:[12 12]
2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shutting down executor
b-8:[54 54]
2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shut down executor b-8:[54 54]
2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shutting down executor b-0:[2
2]
2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shut down executor b-0:[2 2]
2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shutting down executor
b-2:[32 32]
2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.910 b.s.util [INFO] Async loop interrupted!
2017-07-27 21:47:48.910 b.s.d.executor [INFO] Shut down executor b-2:[32 32]
2017-07-27 21:47:48.910 b.s.d.executor [INFO] Shutting down executor
b-8:[41 41]
2017-07-27 21:47:48.910 b.s.util [INFO] Asy
On Thu, Jul 27, 2017 at 3:11 PM, Stig Rohde Døssing <[email protected]> wrote:
> Yes, there is topology.message.timeout.secs for setting how long the
> topology has to process a message after it is emitted from the spout, and
> topology.enable.message.timeouts if you want to disable timeouts
> entirely. I'm assuming that's what you're asking?
>
> 2017-07-27 15:03 GMT+02:00 sam mohel <[email protected]>:
>
>> Thanks for your patience and time. I will use debug now . But is there
>> any settings or configurations about the time for spout? How can I increase
>> it to try ?
>>
>> On Thursday, July 27, 2017, Stig Rohde Døssing <[email protected]> wrote:
>> > Last message accidentally went to you directly instead of the mailing
>> list.
>> >
>> > Never mind what I wrote about worker slots. I think you should check
>> that all tuples are being acked first. Then you might want to try enabling
>> debug logging. You should also verify that your spout is emitting all the
>> expected tuples. Since you're talking about a result file, I'm assuming
>> your spout output is limited.
>> >
>> > 2017-07-27 10:36 GMT+02:00 Stig Rohde Døssing <[email protected]>:
>> >>
>> >> Okay. Unless you're seeing out of memory errors or know that your
>> garbage collector is thrashing, I don't know why changing your xmx would
>> help. Without knowing more about your topology it's hard to say what's
>> going wrong. I think your best bet is to enable debug logging and try to
>> figure out what happens when the topology stops writing to your result
>> file. When you run your topology on a distributed cluster, you can use
>> Storm UI to verify that all your tuples are being acked, maybe your tuple
>> trees are not being acked correctly?
>> >>
>> >> Multiple topologies shouldn't be interfering with each other, the only
>> thing I can think of is if you have too few worker slots and some of your
>> topology's components are not being assigned to a worker. You can see this
>> as well in Storm UI.
>> >>
>> >> 2017-07-27 <20%2017%2007%2027> 8:11 GMT+02:00 sam mohel <
>> [email protected]>:
>> >>>
>> >>> Yes I tried 2048 and 4096 to make worker more size but same problem .
>> >>>
>> >>> I have result file . It should contains the result of my processing .
>> The size of this file should be 7 mb but what I got after sumbit the
>> topology 50 kb only .
>> >>>
>> >>> I submitted this toplogy before . Since 4 months . But when I
>> submitted it now I got this problem .
>> >>>
>> >>> How the toplogy working well before but now not ?
>> >>>
>> >>> Silly question and sorry for that
>> >>> I submitted three topology except that one . Is that make memory weak
>> ? Or should I clean something after that
>> >>>
>> >>> On Thursday, July 27, 2017, Stig Rohde Døssing <[email protected]>
>> wrote:
>> >>> > As far as I can tell the default xmx for workers in 0.10.2 is 768
>> megs (https://github.com/apache/storm/blob/v0.10.2/conf/defaults.
>> yaml#L134), your supervisor logs shows the following:
>> >>> > "Launching worker with command: <snip> -Xmx2048m". Is this the
>> right configuration?
>> >>> >
>> >>> > Regarding the worker log, it looks like the components are
>> initialized correctly, all the bolts report that they're done running
>> prepare(). Could you explain what you expect the logs to look like and what
>> you expect to happen when you run the topology?
>> >>> >
>> >>> > It's sometimes helpful to enable debug logging if your topology
>> acts strange, consider trying that by setting
>> >>> > Config conf = new Config();
>> >>> > conf.setDebug(true);
>> >>> >
>> >>> > 2017-07-27 <20%2017%2007%2027> 1:43 GMT+02:00 sam mohel <
>> [email protected]>:
>> >>> >>
>> >>> >> Same problem with distributed mode . I tried to submit toplogy in
>> distributed with localhost and attached log files of worker and supervisor
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> On Thursday, July 27, 2017, sam mohel <[email protected]> wrote:
>> >>> >> > I submit my topology by this command
>> >>> >> > mvn package
>> >>> >> > mvn compile exec:java -Dexec.classpathScope=compile
>> -Dexec.mainClass=trident.Topology
>> >>> >> > and i copied those lines
>> >>> >> > 11915 [Thread-47-b-4] INFO b.s.d.executor - Prepared bolt
>> b-4:(40)
>> >>> >> > 11912 [Thread-111-b-2] INFO b.s.d.executor - Prepared bolt
>> b-2:(14)
>> >>> >> > 11934 [Thread-103-b-5] INFO b.s.d.executor - Prepared bolt
>> b-5:(45)
>> >>> >> > sam@lenovo:~/first-topology$
>> >>> >> > from what i saw in terminal . I checked the size of the result
>> file and found it's 50 KB each time i submit it .
>> >>> >> > what should i check ?
>> >>> >> > On Wed, Jul 26, 2017 at 9:05 PM, Bobby Evans <
>> [email protected]> wrote:
>> >>> >> >>
>> >>> >> >> Local mode is totally separate and there are no processes
>> launched except the original one. Those values are ignored in local mode.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> - Bobby
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> On Wednesday, July 26, 2017, 2:01:52 PM CDT, sam mohel <
>> [email protected]> wrote:
>> >>> >> >>
>> >>> >> >> Thanks so much for replying , i tried to submit topology in
>> local mode ... i increased size of worker like
>> >>> >> >> conf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx4096m" );
>> >>> >> >>
>> >>> >> >> but got in terminal
>> >>> >> >> 11920 [Thread-121-b-4] INFO b.s.d.executor - Preparing bolt
>> b-4:(25)
>> >>> >> >> 11935 [Thread-121-b-4] INFO b.s.d.executor - Prepared bolt
>> b-4:(25)
>> >>> >> >> 11920 [Thread-67-b-5] INFO b.s.d.executor - Preparing bolt
>> b-5:(48)
>> >>> >> >> 11936 [Thread-67-b-5] INFO b.s.d.executor - Prepared bolt
>> b-5:(48)
>> >>> >> >> 11919 [Thread-105-b-2] INFO b.s.d.executor - Prepared bolt
>> b-2:(10)
>> >>> >> >> 11915 [Thread-47-b-4] INFO b.s.d.executor - Prepared bolt
>> b-4:(40)
>> >>> >> >> 11912 [Thread-111-b-2] INFO b.s.d.executor - Prepared bolt
>> b-2:(14)
>> >>> >> >> 11934 [Thread-103-b-5] INFO b.s.d.executor - Prepared bolt
>> b-5:(45)
>> >>> >> >> sam@lenovo:~/first-topology$
>> >>> >> >> and didn't complete processing . the size of the result is 50
>> KB . This topology was working well without any problems . But when i tried
>> to submit it now , i didn't get the full result
>> >>> >> >>
>> >>> >> >> On Wed, Jul 26, 2017 at 8:35 PM, Bobby Evans <
>> [email protected]> wrote:
>> >>> >> >>
>> >>> >> >> worker.childops is the default value that is set by the system
>> administrator in storm.yaml on each of the supervisor nodes.
>> topology.worker.childopts is what you set in your topology conf if you want
>> to add something more to the command line.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> - Bobby
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> On Tuesday, July 25, 2017, 11:50:04 PM CDT, sam mohel <
>> [email protected]> wrote:
>> >>> >> >>
>> >>> >> >> i'm using 0.10.2 version . i tried to write in the code
>> >>> >> >> conf.put(Config.WORKER_ CHILDOPTS, "-Xmx4g");
>> >>> >> >> conf.put(Config.SUPERVISOR_ CHILDOPTS, "-Xmx4g");
>> >>> >> >>
>> >>> >> >> but i didn't touch any affect . Did i write the right
>> configurations ?
>> >>> >> >> Does this value is the largest ?
>> >>> >> >>
>> >>> >> >
>> >>> >> >
>> >>> >
>> >
>> >
>>
>
>
