Oscar Boykin created STORM-1537:
-----------------------------------
Summary: Upgrade to Kryo 3
Key: STORM-1537
URL: https://issues.apache.org/jira/browse/STORM-1537
Project: Apache Storm
Issue Type: Improvement
Reporter: Oscar Boykin
In storm, Kryo (2.21) is used for serialization:
https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231
The user must use the same version storm does, or there will be a java class
error at runtime.
Storm depends on a quasi-abandoned library: carbonite:
https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210
which depends on Kryo 2.21 and Twitter chill 0.3.6:
https://github.com/sritchie/carbonite/blob/master/project.clj#L8
Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3:
https://github.com/twitter/chill/pull/245
because Spark, also depending on chill, would like to upgrade for performance
improvements and bugfixes.
https://issues.apache.org/jira/browse/SPARK-11416
Unfortunately, summingbird depends on storm:
https://github.com/twitter/summingbird/blob/develop/build.sbt#L34
so, if chill is upgraded, and that gets on the classpath, summingbird will
break at runtime.
I propose:
1) copy the carbonite code into storm. It is likely the only consumer.
2) bump the storm kryo dependency after chill upgrades: recall that storm
actually depends on chill-java. A dependency that could possibly be removed
after you pull carbonite in.
3) once a new version of storm is published, summingbird (and scalding) can
upgrade to the latest chill.
Also, I hope for:
4) we as a JVM community get better about classpath isolation and versioning.
Diamonds like this in one big classpath make large codebases very fragile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
