I have issued a pull request that allows local streams to shutdown if a provider, processor, or writer tasks stops execution due to an unhandled throwable. This should be a step forward in making streams more transparent and reliable on the local level. The Throwable will appear in the logs and it will note which Runnable class threw the error.
This is intended to only be a temporary fix. This will illuminate many streams operations that have bugs, and need better testing. Please create tickets for all exceptions that are caused by bugs and issue pull requests (with appropriate tests and documentation) to fix them. In the future we can relaunch threads that exit due to error, and reprocess the failed datum. Until we implemented better datum tacking this is not possible, but it is on the road map. Please create tickets or notify the dev list of bugs you encounter. This push should expose bugs that previously were causing streams to fail silently or stall/hang. We need to have big push for better tested code in the project. Note: Threads that exit due to error that are not started/handled by the local stream builder will still fail silently. The fact that so many providers, processors, and writers have to launch their own threads means we should work on the local frame work to remove those needs push almost all thread management responsibility to the framework so we can ensure consistent monitoring/error handling of threads. Thanks, Ryan Ebanks
