GitHub user smashew opened a pull request:
https://github.com/apache/incubator-streams/pull/16
Overhaul of LocalStreamsBuilder
Commit Fixes:
https://issues.apache.org/jira/browse/STREAMS-81
https://issues.apache.org/jira/browse/STREAMS-80
https://issues.apache.org/jira/browse/STREAMS-82
https://issues.apache.org/jira/browse/STREAMS-83
https://issues.apache.org/jira/browse/STREAMS-84
https://issues.apache.org/jira/browse/STREAMS-85
Very large overhaul, pardon the large commit. The only major addition to
this is to allow for the 'concept' of a known finite stream. There are many
streams that are listened to and will be opened for a long time, however, there
are other streams, that have a known termination point.
IE: Reading 1,000,000 records from an index in ElasticSearch to HDFS
There is a flag, now in StreamsResultSet, that when set to true, changes
the 'queue' from being read as a Collection<E> of items to an actual shared
queue. This has many benefits, mainly, the queue is respected as a queue
instead of being treated as a batch. In paradigms where items can be read in a
stream, via a queue, this is an excellent choice.
There were many bugs that prevented the execution of this software from
being run inside of a currently running JVM as it would cause memory and thread
leaks. This has been updated and repaired in this version.
+++ tests +++
The tests that were in LocalStreamsBuilder have been restored and updated
to be accurate (many of the tests were not accurate). This helps to ensure
backward compatibility with the previous implementation. In addition, many (20)
tests were added for edge cases and various amounts of behavior that could
disrupt the execution of a stream. (RuntimeExceptions, excessive delays in
processing, etc...) All tests have been written and are currently passing.
Thank you for your time!
Smashew
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smashew/incubator-streams master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-streams/pull/16.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16
----
commit c21e2b8687738753ff51ff2a3e595a1e4b5a439c
Author: [email protected] <[email protected]>
Date: 2014-05-13T16:19:45Z
Merge pull request #1 from apache/master
f
commit 5c6f6560c2a9d90ad1406733f0dee80b1f418804
Author: Matthew Hager <[email protected]>
Date: 2014-05-15T16:19:39Z
STREAMS-81 - Fix for: There is an issue shutting down the local
streambuilder, this is the fix.
commit 1c30164a2d58d141b15d7ea181405542eda766e2
Author: Matthew Hager <[email protected]>
Date: 2014-05-15T16:32:21Z
I noticed these threads running to, causing an issue.
commit ff6bbd579594a20893ae33a64b11c1cb7e085b38
Author: [email protected] <[email protected]>
Date: 2014-05-15T20:39:14Z
Merge pull request #2 from apache/master
merge
commit 49511e8022627e8ee1e2257372a354c753477868
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T01:01:59Z
added some more tests, re
commit bf6e4627957b845d74b19e50306f6c3168d8a62f
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T15:39:38Z
Added more tests... still working on getting them all to pass and adding
some more delay tests
commit 4e5e78d550559235e47c9fc7f3b4bb9816825e66
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T15:40:26Z
Changes to url
commit 2683fc8617edbd86b1258e48d5cb873f29e9fe31
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T16:41:54Z
moved some tests around and some other stuff
commit 6112b54e6e9e92fe8fb36c949237a7b418219aa4
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T17:28:44Z
Took out the monitor thread, replaced with a schedule. The monitor thread
because of the Thread.sleep(#) was causing the shutdown behavior to be slow and
masking issues that resulted from improper handling of the queues.
commit e4725e6f5f78115a261e39f9d97e0fc43d5009e8
Author: Robert Douglas <[email protected]>
Date: 2014-05-16T19:18:03Z
Added in additional delayed tests and removed one redundant
LocalStreamBuilderTest
commit a3061d5c597e65217c6d18f03e58955f2b1ee451
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T19:21:03Z
pushed more items down into the base class for simplicity and readability.
Many things are now marked as 'final' which shouldn't be changed during
operation.
commit dd23158d824426047933a7ad8c6b3e76e3187823
Author: Robert Douglas <[email protected]>
Date: 2014-05-16T22:39:49Z
Added tests to ensure that proper behavior occurs when errors are thrown at
runtime in various stages of the stream
commit 323d9c196af719605e2a55ce2572558ebcfbd98d
Author: Matthew Hager <[email protected]>
Date: 2014-05-17T01:45:03Z
All tests have been added, everything is working, many different delays,
and use-cases represented. All work and tests, save the delay tets, are
extremely fast in the ms to produce and shutdown properly.
commit 2a1928e1bb65d13c345c28537fb050eec6a398f9
Author: Matthew Hager <[email protected]>
Date: 2014-05-16T19:21:03Z
pushed more items down into the base class for simplicity and readability.
Many things are now marked as 'final' which shouldn't be changed during
operation.
commit cbf223d7baf1106767d59218820b46dca748ed7b
Author: Matthew Hager <[email protected]>
Date: 2014-05-17T01:45:03Z
All tests have been added, everything is working, many different delays,
and use-cases represented. All work and tests, save the delay tets, are
extremely fast in the ms to produce and shutdown properly.
commit 40a258b21f641675233a8c84663bf2b7c7e5dd6b
Author: Matthew Hager <[email protected]>
Date: 2014-05-17T03:50:48Z
Merge branch 'robdouglas-Unit-Tests'
Conflicts:
streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/builders/LocalStreamBuilderTest.java
streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/test/processors/PassThroughStaticCounterProcessor.java
streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/test/writer/DatumCounterWriter.java
commit aa2a0253f8a764b2687b8b4691af08e39322253d
Author: Matthew Hager <[email protected]>
Date: 2014-05-17T04:45:22Z
Completed overhaul of LocalStreamsbuilder. Much faster backwards compatible
and now, complete with over 30 working tests.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
