[
https://issues.apache.org/jira/browse/METRON-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936424#comment-15936424
]
ASF GitHub Bot commented on METRON-793:
---------------------------------------
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107430400
--- Diff: pom.xml ---
@@ -67,20 +67,44 @@
<!-- base project versions -->
<base_storm_version>1.0.1</base_storm_version>
<base_flux_version>1.0.1</base_flux_version>
- <base_kafka_version>0.10.0.1</base_kafka_version>
+ <base_kafka_version>0.10.0</base_kafka_version>
<base_hadoop_version>2.7.1</base_hadoop_version>
<base_hbase_version>1.1.1</base_hbase_version>
- <global_accumulo_version>1.8.0</global_accumulo_version>
<base_flume_version>1.5.2</base_flume_version>
<!-- full dependency versions -->
+ <global_accumulo_version>1.8.0</global_accumulo_version>
<global_antlr_version>4.5</global_antlr_version>
<global_opencsv_version>3.7</global_opencsv_version>
<global_curator_version>2.7.1</global_curator_version>
<global_classindex_version>3.3</global_classindex_version>
- <global_storm_version>${base_storm_version}</global_storm_version>
+ <global_storm_version>1.0.3</global_storm_version>
+ <!--
+ This bears some explanation. storm-kafka-client is our kafka
spout.
+ If we ever hope to support kerberos, this provides the
capability to do so
+ in apache. Unfortunately, it also does not support, as of
Storm 1.0.x
+ Kafka 0.10.x (see
https://issues.apache.org/jira/browse/STORM-2091).
+ The consumer libraries (not to be confused with the protocol)
on the JVM
+ are binary incompatible. Note the discussion on
https://issues.apache.org/jira/browse/KAFKA-3006,
+ the main issue is the move to Collection over List. While
this would be polymorphically
+ a non-issue, it would require a recompile of
storm-kafka-client against Kafka 0.10.x.
+
+ Since a targeted platform is HDP 2.5.x, which ships only
kafka 0.10.x, we need
+ to support kafka 0.10.x. Therefore, if we are to use this,
then we would need
+ to support both Kafka 0.9.x and 0.10.x. Unfortunately, this
would require us
+ to fork some of the internal projects because the 0.9.x API
has shifted
+ (e.g. the Admin functions have different parameters) and
behaves
+ differently than 0.10.x in subtle ways (e.g. KAFKA_GET
doesn't work as implemented).
+
+ Rather than do this, we chose to depend on the HDP version of
storm-kafka because
+ it is compiled against 0.10.x and therefore would allow us to
not fork our support
+ for kafka. I do not like this bleeding of the HDP profile
dependency into the default,
+ but I justify it by noting that this should be able to be
removed when we migrate to
+ Storm 1.1.x, which properly supports Kafka 0.10.x.
+ -->
+
<global_storm_kafka_version>1.0.1.2.5.0.0-1245</global_storm_kafka_version>
--- End diff --
The comment above this line is worth pointing out.
> Migrate to storm-kafka-client kafka spout from storm-kafka
> ----------------------------------------------------------
>
> Key: METRON-793
> URL: https://issues.apache.org/jira/browse/METRON-793
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
>
> In order to eventually support kerberos, the suggested path is to migrate to
> the new kafka spout (org.apache.storm:storm-kafka-client) which uses the new
> consumer API.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
