GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/310
METRON-495: Upgrade Storm to 1.0.x
***** INCOMPLETE DO NOT MERGE *****
Putting the PR out, so people can take a look and potentially fix/improve
things.
Initial start at upgrading the Storm version to 1.0.1, and I'm going to hop
off of this for right now. Provided in case anyone wants to take a stab at
fixing / cleaning up the various issues with it, and so feedback can be
provided. If you do make improvements, feel free to grab the branch and submit
a new PR.
What changes occurred
- The move over to org.apache.storm from backtype.storm
- API changes from byte[] to ByteArray are handled
- Kafka also updated to 0.10 out of necessity, storm-kafka is still used
(storm-kafka-client seemed like significantly more work given the amount of
changes to the underpinnings). I would not expect this to cause issues, given
backwards compatibility, but since tests still fail this hasn't been validated.
- Some of the various classpath issues have been taken care of (things move
between jars, and a couple test items now need a test jar which has been pulled
in and scoped appropriately on various modules)
- A couple simple Scala wrappers about some of the Storm Kafka handling in
regards to Zookeeper has been added (including to the pom) to avoid having to
call direct Scala only testing tools that use pretty Scala specific stuff. The
wrappers just take in the args we were using and use defaults for the rest.
- Changes to the full dev platform to account for using HDP 2.5 to match
the new version. This is entirely untested given test failures.
What works
- Most unit tests. Interestingly, metron-platform/metron-management fails
on CLI, but succeeds in IntelliJ. I'm guessing classpath issue as a result of
attempting to fix integration tests broke this on the CLI as it used to work,
but unsure.
- Most integration tests. This involved moving.
What doesn't work
- Both some unit and some integration tests
- Dev platforms are untested.
What's left
- Fixing tests
- Testing full dev
- Building quick dev image and testing it
- End to End testing. Given the scope of the changes, this needs to be
pretty thorough.
- Ambari MPack testing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron storm_1.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/310.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #310
----
commit dbf7ff19072cb44df66928920a70e0f1f1f8e4e8
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-11T13:58:39Z
Initial move to Storm 1.0.1
commit 9796bf36fc4a19fbed5276166c305cb8fb815e00
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-12T15:13:45Z
fixing more imports
commit 044b9d7113f3b7d2d367f2f58e40872ea372d75a
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-13T19:56:27Z
changes to kakfa. WIP
commit ec157af245cfcac3a8eda390d145328d62dfb0ab
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-13T19:56:43Z
Missing changes from last commit
commit 90321655055e9a290f64724c8e3bd8a013deebfc
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-14T12:51:17Z
WIP changes with scala wrapper
commit f418f50958d9826c9307858914253f304a18d066
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-14T16:22:59Z
More integration tests working
commit f445bfb9fef0583bbce444406227f96a4c506706
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-14T19:24:41Z
cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
