[
https://issues.apache.org/jira/browse/SPARK-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967525#comment-13967525
]
Ted Malaska commented on SPARK-1478:
------------------------------------
After reviewing the code and I found that Spark Streaming is focused on Flume
1.2.0 and Avro-ipc 1.6.3.
Cloudera, Hortenworks, and even MapR are using Flume 1.4.0 in their current
releases and that version of Flume uses Avro-ipc 1.7.3 which allows for us to
apply our own ChannelPipeLine, which allows us to use compression and
encryption.
I don't think I can complete do this Jira until I get permission to up the
versions of Flume in the pom file from 1.2.0 to at least 1.3.0. Flume version
1.3 is more then 2 years old so that should be ok for Spark Users and Flume
1.3.0 uses Avro-ipc 1.7.3, which is what we need.
As for now, I will continue as if I had the permission, and I will do a pull
request with the change from Flume from 1.2.0 to 1.3.0.
Thanks
> Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915
> ---------------------------------------------------------------
>
> Key: SPARK-1478
> URL: https://issues.apache.org/jira/browse/SPARK-1478
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Ted Malaska
> Priority: Minor
>
> Flume-1915 added support for compression over the wire from avro sink to avro
> source. I would like to add this functionality to the FlumeReceiver.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
