Hi Eduardo,
Thanks for the tests and questions,
I can't tell if this is a problem of Storm / Nimbus or Samoa.
Have you tried a smaller stream, i.e., with a smaller number of events, or
smaller number of attributes?
Perhaps the DSPE is shutting down due to memory error or other resource
issue?
Try with different sizes and let us know the status of the machine before
it fails.
Thanks,
Nicolas
On Mon, Sep 12, 2016 at 4:25 AM, Eduardo Costa <eduardoc...@gmail.com>
wrote:
> Dear,
> I'm executanto the Samoa with the storm in local mode (command line:
> bin/samoa storm target/SAMOA-Storm-0.4.0-incubating-SNAPSHOT.jar
> "PrequentialEvaluation -i -1 -f 1000 -d /tmp/cenario10-2classes-
> ordenado-trat_entdv-sem-label-puro.csv -l (classifiers.ensemble.
> AdaptiveBagging
> -s 1) -s (org.apache.samoa.streams.ArffFileStream -s HDFSFileStreamSource
> -f /cenario10-2classes-ordenado-trat_entdv-sem-label-puro.csv.arff)").
>
> I am using a dataset of 1,300.00 instances, but this samoa stopping at
> 480,000 when I use Adaptive Bagging. After a while the following message
> appears:
>
> ------------------------------------------------------------
> ---------------------------------------------------
>
> org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor -
> Change detected, resetting the classifier
> 66025 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66025 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66026 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66027 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66027 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 224126 [main] INFO backtype.storm.daemon.nimbus - Delaying event :remove
> for 30 secs for Prequential_20160911213510-1-1473640527
> 224132 [main] INFO backtype.storm.daemon.nimbus - Updated
> Prequential_20160911213510-1-1473640527 with status {:type :killed,
> :kill-time-secs 30}
> 224132 [main] INFO backtype.storm.daemon.nimbus - Shutting down master
> 224136 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50001
> 224137 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
> org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /0:0:0:0:0:0:0:1:50353 which had sessionid 0x1571bd1c9c50001
> 224137 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50001 closed
> 224137 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224142 [main] INFO backtype.storm.daemon.nimbus - Shut down master
> 224155 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50003
> 224159 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50003 closed
> 224159 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
> org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /0:0:0:0:0:0:0:1:50355 which had sessionid 0x1571bd1c9c50003
> 224159 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224163 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50005
> 224164 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
> org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /127.0.0.1:49569 which had sessionid 0x1571bd1c9c50005
> 224165 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224165 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50005 closed
> 224166 [main] INFO backtype.storm.daemon.supervisor - Shutting down
> 9957d6b3-05c3-46d4-bb6f-c37751a40026:37ed88ae-36f0-4b87-b128-cd16c1fa21f7
> 224166 [main] INFO backtype.storm.process-simulator - Killing process
> 3a39d80e-10b5-4db7-823d-a48ca5d85109
> 224166 [main] INFO backtype.storm.daemon.worker - Shutting down worker
> Prequential_20160911213510-1-1473640527 9957d6b3-05c3-46d4-bb6f-
> c37751a40026
> 1024
> 224167 [main] INFO backtype.storm.daemon.worker - Shutting down receive
> thread
> 224167 [main] INFO backtype.storm.messaging.loader - Shutting down
> receiving-thread: [Prequential_20160911213510-1-1473640527, 1024]
> 224167 [main] INFO backtype.storm.messaging.loader - Waiting for
> receiving-thread:[Prequential_20160911213510-1-1473640527, 1024] to die
> 224167 [Thread-10-worker-receiver-thread-0] INFO
> backtype.storm.messaging.loader
> - Receiving-thread:[Prequential_20160911213510-1-1473640527, 1024]
> received
> shutdown notice
> 224168 [main] INFO backtype.storm.messaging.loader - Shutdown
> receiving-thread: [Prequential_20160911213510-1-1473640527, 1024]
> 224168 [main] INFO backtype.storm.daemon.worker - Shut down receive thread
> 224169 [main] INFO backtype.storm.daemon.worker - Terminating messaging
> context
> 224169 [main] INFO backtype.storm.daemon.worker - Shutting down executors
> 224169 [main] INFO backtype.storm.daemon.executor - Shutting down executor
> org.apache.samoa.evaluation.EvaluatorProcessor_1:[2 2]
> 224169 [Thread-12-org.apache.samoa.evaluation.EvaluatorProcessor_1] INFO
> backtype.storm.util - Async loop interrupted!
> 224170 [Thread-11-disruptor-executor[2 2]-send-queue] INFO
> backtype.storm.util - Async loop interrupted!
> 224170 [main] INFO backtype.storm.daemon.executor - Shut down executor
> org.apache.samoa.evaluation.EvaluatorProcessor_1:[2 2]
> 224170 [main] INFO backtype.storm.daemon.executor - Shutting down executor
> org.apache.samoa.learners.classifiers.ensemble.
> BaggingDistributorProcessor_1:[3 3]
> ------------------------------------------------------------
> --------------------------------------------------------
> What can it be?
>
> Regards,
> Eduardo Costa.
>
--
Nicolas Kourtellis
