[ 
https://issues.apache.org/jira/browse/METRON-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576577#comment-15576577
 ] 

ASF GitHub Bot commented on METRON-495:
---------------------------------------

GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/310

    METRON-495: Upgrade Storm to 1.0.x

    ***** INCOMPLETE DO NOT MERGE ***** 
    
    Putting the PR out, so people can take a look and potentially fix/improve 
things.
    
    Initial start at upgrading the Storm version to 1.0.1, and I'm going to hop 
off of this for right now.  Provided in case anyone wants to take a stab at 
fixing / cleaning up the various issues with it, and so feedback can be 
provided. If you do make improvements, feel free to grab the branch and submit 
a new PR.
    
    What changes occurred
    - The move over to org.apache.storm from backtype.storm
    - API changes from byte[] to ByteArray are handled
    - Kafka also updated to 0.10 out of necessity, storm-kafka is still used 
(storm-kafka-client seemed like significantly more work given the amount of 
changes to the underpinnings).  I would not expect this to cause issues, given 
backwards compatibility, but since tests still fail this hasn't been validated.
    - Some of the various classpath issues have been taken care of (things move 
between jars, and a couple test items now need a test jar which has been pulled 
in and scoped appropriately on various modules)
    - A couple simple Scala wrappers about some of the Storm Kafka handling in 
regards to Zookeeper has been added (including to the pom) to avoid having to 
call direct Scala only testing tools that use pretty Scala specific stuff.  The 
wrappers just take in the args we were using and use defaults for the rest.
    - Changes to the full dev platform to account for using HDP 2.5 to match 
the new version.  This is entirely untested given test failures.
    
    What works
    - Most unit tests.  Interestingly, metron-platform/metron-management fails 
on CLI, but succeeds in IntelliJ.  I'm guessing classpath issue as a result of 
attempting to fix integration tests broke this on the CLI as it used to work, 
but unsure.
    - Most integration tests. This involved moving.
    
    What doesn't work
    - Both some unit and some integration tests
    - Dev platforms are untested.
    
    What's left
    - Fixing tests
    - Testing full dev
    - Building quick dev image and testing it
    - End to End testing.  Given the scope of the changes, this needs to be 
pretty thorough.
    - Ambari MPack testing.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron storm_1.0

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #310
    
----
commit dbf7ff19072cb44df66928920a70e0f1f1f8e4e8
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-11T13:58:39Z

    Initial move to Storm 1.0.1

commit 9796bf36fc4a19fbed5276166c305cb8fb815e00
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-12T15:13:45Z

    fixing more imports

commit 044b9d7113f3b7d2d367f2f58e40872ea372d75a
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-13T19:56:27Z

    changes to kakfa.  WIP

commit ec157af245cfcac3a8eda390d145328d62dfb0ab
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-13T19:56:43Z

    Missing changes from last commit

commit 90321655055e9a290f64724c8e3bd8a013deebfc
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-14T12:51:17Z

    WIP changes with scala wrapper

commit f418f50958d9826c9307858914253f304a18d066
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-14T16:22:59Z

    More integration tests working

commit f445bfb9fef0583bbce444406227f96a4c506706
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-10-14T19:24:41Z

    cleanup

----


> Upgrade Storm to 1.0.x
> ----------------------
>
>                 Key: METRON-495
>                 URL: https://issues.apache.org/jira/browse/METRON-495
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Justin Leet
>            Assignee: Justin Leet
>
> As listed at https://storm.apache.org/2016/04/12/storm100-released.html, 
> there's a variety of improvements with Storm 1.0.  The obvious and likely 
> most important improvement is to performance, but a variety of other 
> improvements are noted on that link.
> There are several changes that have to occur in order to make this upgrade.  
> As noted in http://storm.apache.org/releases/current/index.html, Storm's code 
> packages moved from backtype.storm to org.apache.storm, meaning all 
> topologies have to be recompiled if that change.  There is a runtime 
> converter to run things in place with backtype.storm, but this doesn't appear 
> to be enough for our case because a couple interfaces change from byte[] to 
> ByteBuffer (somewhat, but not entirely, related to 
> https://issues.apache.org/jira/browse/STORM-1449). Even without this issue, 
> the long term solution is to use the new package naming.
> In addition, our dev instances right now spin up an HDP 2.4 instance, which 
> matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to migrate 
> to Storm 1.0.x, I'd prefer to match that version, rather than going to 1.0.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to