[ https://issues.apache.org/jira/browse/STREAMS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864712#comment-15864712 ]
Steve Blackmon edited comment on STREAMS-183 at 2/13/17 11:53 PM: ------------------------------------------------------------------ The basic strategy should be: runtime calls close() on component when stream is ready to shutdown component isRunning() should returns false once it has finished its work runtime waits a maximum of 'taskTimeoutMs' before shutting stream down regardless of component status was (Author: steveblackmon): The basic strategy should be: runtime calls close() on component component isRunning() returns false once it has flushed runtime waits a maximum of 'taskTimeoutMs' before shutting stream down regardless of component status > streams-local closes persist-writers before datums in-flight are written > ------------------------------------------------------------------------ > > Key: STREAMS-183 > URL: https://issues.apache.org/jira/browse/STREAMS-183 > Project: Streams > Issue Type: Bug > Components: Runtimes > Reporter: Steve Blackmon > > I have observed this behavior when writing datums from a readCurrent provider > through to ElasticsearchPersistWriter and ElasticsearchPersistDeleter. > The log looks like: > 421777 [pool-2-thread-7] INFO o.a.s.e.ElasticsearchPersistReader - > PersistReader done > 469271 [main] INFO o.a.s.l.builders.LocalStreamBuilder - Attempting to > shutdown tasks > 476319 [pool-2-thread-3] ERROR o.a.s.e.ElasticsearchPersistWriter - There was > an error sending the batch: No node available > 485275 [main] WARN o.a.s.l.builders.LocalStreamBuilder - Task > org.apache.streams.local.tasks.StreamsPersistWriterTask@6add646d failed to > terminate in allotted timeframe > 495276 [pool-2-thread-3] WARN o.a.s.e.ElasticsearchPersistWriter - Catchup > was interrupted. Data may be lost > followed by additional No Node Available exceptions. > When the same stream is executed as a perpetual stream, this problem does not > occur. > Possible solutions: > a) localstreambuilder waits a while between detecting providers have > completed and stopping downstream components > b) localstreambuilder does not stop downstream components until all datums > have been processed/written > c) writers refuse to close until all datums have been written -- This message was sent by Atlassian JIRA (v6.3.15#6346)