GitHub user clharris opened a pull request:
https://github.com/apache/incubator-pirk/pull/74
PIRK-4 [WIP] Add Streaming Implementation for Apache Storm
This is an initial implementation for a streaming version of Pirk to run on
Apache Storm. I am leaving it temporarily as WIP so people have a chance to
look over it and add feedback. Right now there is only one integration test
which runs the Storm topology 4 times with each of the different significant
configuration possibilities. I wanted to unit test the bolts, but it seems not
very straightforward with the way that the Pirk processing works. I'll try to
add some documentation (at the minimum a diagram of the Pirk Storm topology) by
early next week.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/clharris/incubator-pirk pirk-4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-pirk/pull/74.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #74
----
commit 577fb33d04ce9061d2e88a27595737f30b9c446d
Author: charris <[email protected]>
Date: 2016-07-19T14:33:34Z
Temporarily using Cloudera pom.
commit 41f9042b81a5b683ecbd2c718f98712283acacf9
Author: charris <[email protected]>
Date: 2016-07-19T14:37:36Z
PIRK-4 Initial commit.
commit 0c147c36b377c730c7d6489a82b71958e59cdb2d
Author: charris <[email protected]>
Date: 2016-07-19T14:38:27Z
Adding toMap method and constructor.
commit a30f1a9de6e02dbf55bdc4e771d673b429ac331f
Author: charris <[email protected]>
Date: 2016-07-19T14:39:14Z
Adding functionality for streaming processing.
commit 8d27e5d3f2ff1fec9f2fffd29052189f567d51f0
Author: charris <[email protected]>
Date: 2016-07-19T14:40:17Z
Broke out method to checkDNSHostNameQueryResults.
commit 4df0e31836ec26da00662d92c5eac081c61419b2
Author: charris <[email protected]>
Date: 2016-07-19T15:04:07Z
Added Storm parameters.
commit 126361c31426d8285b4202a1134a653dd210dd4b
Author: eawilliams <[email protected]>
Date: 2016-07-22T14:55:28Z
initial modifications to storm code
commit 9f78a94dec84c7fcb5c09f13ae62598a7700adf4
Author: eawilliams <[email protected]>
Date: 2016-07-22T14:55:57Z
initial modifications to storm code - removing logs
commit 4bdd0c1f2461fdf0a1254bd6369db70b05b3e12a
Author: eawilliams <[email protected]>
Date: 2016-07-22T15:52:06Z
splitPartitions update to storm implementation
commit 180c56161ceffbe7c2d24719f8f9e5071ca632cb
Author: eawilliams <[email protected]>
Date: 2016-07-23T16:44:24Z
changed splitPartitions to false in test
commit 2520fbe8e0e6eb6a9ee8d259e0a5f10f965e916e
Author: charris <[email protected]>
Date: 2016-07-25T15:02:00Z
splitPartitions allows data element partitions to be sent individually.
commit e4e69e0119df4c0659a55bee7d6a5c55a016c9ff
Author: charris <[email protected]>
Date: 2016-07-30T19:41:11Z
Moved hashing into the spout.
commit 3262f0fdd52d62c050c6dc7fb6c6b41c34e99173
Author: charris <[email protected]>
Date: 2016-07-30T19:41:55Z
Added some functionality useful for Storm.
commit 290404531eaaa4442ef611252f4d24b2be7a6371
Author: charris <[email protected]>
Date: 2016-07-30T19:42:12Z
Fixed based on refactoring.
commit ba81c6814d0f0e01974fb934a0625c44676a316e
Author: charris <[email protected]>
Date: 2016-07-31T02:21:55Z
Added Apache headers.
commit 987d2f5254e359aa54ec5969935dda124b4d85a1
Author: charris <[email protected]>
Date: 2016-07-31T20:34:34Z
Changed Scala version of Spark and added curator-test.
commit 3b617c1b6c5ccdc560a2e32e58815b8732f18e58
Author: charris <[email protected]>
Date: 2016-07-31T23:17:34Z
Merging.
commit 22b946cbb361f96f4ecb67fb317d44be1c1b1498
Author: charris <[email protected]>
Date: 2016-07-31T23:18:24Z
Merge branch 'master' of https://github.com/apache/incubator-pirk into storm
commit 204495f8a75d38fd0b51a0970352e2204367e6ff
Author: eawilliams <[email protected]>
Date: 2016-08-01T00:34:32Z
fixed logging and scala version issues via pom modifications; refactored
storm test package
commit 8f8113727be5ae7de97adf82d8953abb2efbde63
Author: eawilliams <[email protected]>
Date: 2016-08-01T01:57:04Z
Merging - closes clharris/incubator-pirk#1
commit 1f8b42fdbf2326114f0cbce5fdfb7bd9c53306c3
Author: Chris Harris <[email protected]>
Date: 2016-08-01T12:22:36Z
Merge pull request #1 from ellisonanne/chris-storm
fixed logging and scala version issues via pom modifications; refactoâ¦
commit a61c9765bff4018041201d3adc7f81038f5499c6
Author: charris <[email protected]>
Date: 2016-08-01T12:39:00Z
Merging.
commit 21c2a3c0166574da7f05576853733a93d4826a65
Author: charris <[email protected]>
Date: 2016-08-16T15:31:35Z
Debugging.
commit bec45e7f39eb6fb73f54c0744382a725897bd3ae
Author: charris <[email protected]>
Date: 2016-08-17T18:49:15Z
Merging. Storm works when jar not minimized.
commit 3bbc4d8e4c4b7d80b6c478b96e7947e673580995
Author: charris <[email protected]>
Date: 2016-08-18T12:32:40Z
Fixing logging issues.
commit 6a9ebcbacd0c0a4354db4cc309309ca7fba228ce
Author: charris <[email protected]>
Date: 2016-08-19T12:42:40Z
Merge branch 'master' of https://github.com/apache/incubator-pirk into storm
commit 78a7d3a8fff29781d3c336d1f97fc1d0ae1b5389
Author: charris <[email protected]>
Date: 2016-08-19T13:17:45Z
Merging pom with unminimized jar.
commit 275b1db125c0c7055f4b5dfb42e675e7fc0a2bd1
Author: charris <[email protected]>
Date: 2016-08-19T15:13:48Z
Topology runs from ResponderDriver
commit 2d2abe1b90891737204bbded9d9d6f7b3fa2f1ff
Author: charris <[email protected]>
Date: 2016-08-19T18:44:25Z
Streamlined integration test to run all 4 configurations.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
