Re: Remove Flume support in 3.0.0?
Yea, I thought we are already going to remove this out. +1 for removing it anyway. 2018년 10월 12일 (금) 오전 1:44, Wenchen Fan 님이 작성: > Note that, it was deprecated in 2.3.0 already: > https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html > > On Fri, Oct 12, 2018 at 12:46 AM Reynold Xin wrote: > >> Sounds like a good idea... >> >> > On Oct 11, 2018, at 6:40 PM, Sean Owen wrote: >> > >> > Yep, that already exists as Bahir. >> > Also, would anyone object to declaring Flume support at least >> > deprecated in 2.4.0? >> >> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke >> wrote: >> >> >> >> I think it makes sense to remove it. >> >> If it is not too much effort and the architecture of the flume source >> is not considered as too strange one may extract it as a separate project >> and put it on github in a dedicated non-supported repository. This would >> enable distributors and other companies to continue to use it with minor >> adaptions in case their architecture depends on it. Furthermore, if there >> is a growing interest then one could pick it up and create a clean >> connector based on the current Spark architecture to be available as a >> dedicated connector or again in later Spark versions. >> >> >> >> That being said there are also „indirect“ ways to use Flume with Spark >> (eg via Kafka), so i believe people would not be affected so much by a >> removal. >> >> >> >> (Non-Voting just my opinion) >> >> >> >>> Am 10.10.2018 um 22:31 schrieb Sean Owen : >> >>> >> >>> Marcelo makes an argument that Flume support should be removed in >> >>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 >> >>> >> >>> I tend to agree. Is there an argument that it needs to be supported, >> >>> and can this move to Bahir if so? >> >>> >> >>> - >> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>> >> > >> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: Remove Flume support in 3.0.0?
Note that, it was deprecated in 2.3.0 already: https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html On Fri, Oct 12, 2018 at 12:46 AM Reynold Xin wrote: > Sounds like a good idea... > > > On Oct 11, 2018, at 6:40 PM, Sean Owen wrote: > > > > Yep, that already exists as Bahir. > > Also, would anyone object to declaring Flume support at least > > deprecated in 2.4.0? > >> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke > wrote: > >> > >> I think it makes sense to remove it. > >> If it is not too much effort and the architecture of the flume source > is not considered as too strange one may extract it as a separate project > and put it on github in a dedicated non-supported repository. This would > enable distributors and other companies to continue to use it with minor > adaptions in case their architecture depends on it. Furthermore, if there > is a growing interest then one could pick it up and create a clean > connector based on the current Spark architecture to be available as a > dedicated connector or again in later Spark versions. > >> > >> That being said there are also „indirect“ ways to use Flume with Spark > (eg via Kafka), so i believe people would not be affected so much by a > removal. > >> > >> (Non-Voting just my opinion) > >> > >>> Am 10.10.2018 um 22:31 schrieb Sean Owen : > >>> > >>> Marcelo makes an argument that Flume support should be removed in > >>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 > >>> > >>> I tend to agree. Is there an argument that it needs to be supported, > >>> and can this move to Bahir if so? > >>> > >>> - > >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >>> > > > > - > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: Remove Flume support in 3.0.0?
Sounds like a good idea... > On Oct 11, 2018, at 6:40 PM, Sean Owen wrote: > > Yep, that already exists as Bahir. > Also, would anyone object to declaring Flume support at least > deprecated in 2.4.0? >> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke wrote: >> >> I think it makes sense to remove it. >> If it is not too much effort and the architecture of the flume source is not >> considered as too strange one may extract it as a separate project and put >> it on github in a dedicated non-supported repository. This would enable >> distributors and other companies to continue to use it with minor adaptions >> in case their architecture depends on it. Furthermore, if there is a growing >> interest then one could pick it up and create a clean connector based on the >> current Spark architecture to be available as a dedicated connector or again >> in later Spark versions. >> >> That being said there are also „indirect“ ways to use Flume with Spark (eg >> via Kafka), so i believe people would not be affected so much by a removal. >> >> (Non-Voting just my opinion) >> >>> Am 10.10.2018 um 22:31 schrieb Sean Owen : >>> >>> Marcelo makes an argument that Flume support should be removed in >>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 >>> >>> I tend to agree. Is there an argument that it needs to be supported, >>> and can this move to Bahir if so? >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Remove Flume support in 3.0.0?
Yep, that already exists as Bahir. Also, would anyone object to declaring Flume support at least deprecated in 2.4.0? On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke wrote: > > I think it makes sense to remove it. > If it is not too much effort and the architecture of the flume source is not > considered as too strange one may extract it as a separate project and put it > on github in a dedicated non-supported repository. This would enable > distributors and other companies to continue to use it with minor adaptions > in case their architecture depends on it. Furthermore, if there is a growing > interest then one could pick it up and create a clean connector based on the > current Spark architecture to be available as a dedicated connector or again > in later Spark versions. > > That being said there are also „indirect“ ways to use Flume with Spark (eg > via Kafka), so i believe people would not be affected so much by a removal. > > (Non-Voting just my opinion) > > > Am 10.10.2018 um 22:31 schrieb Sean Owen : > > > > Marcelo makes an argument that Flume support should be removed in > > 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 > > > > I tend to agree. Is there an argument that it needs to be supported, > > and can this move to Bahir if so? > > > > - > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Remove Flume support in 3.0.0?
Yup was thinking the same. It is legacy too at this point. On Wed, Oct 10, 2018, 3:19 PM Marcelo Vanzin wrote: > BTW, although I did not file a bug for that, I think we should also > consider getting rid of the kafka-0.8 connector. > > That would leave only kafka-0.10 as the single remaining dstream > connector in Spark, though. (If you ignore kinesis which we can't ship > in binary form or something like that?) > On Wed, Oct 10, 2018 at 1:32 PM Sean Owen wrote: > > > > Marcelo makes an argument that Flume support should be removed in > > 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 > > > > I tend to agree. Is there an argument that it needs to be supported, > > and can this move to Bahir if so? > > > > - > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > > -- > Marcelo >
Re: Remove Flume support in 3.0.0?
BTW, although I did not file a bug for that, I think we should also consider getting rid of the kafka-0.8 connector. That would leave only kafka-0.10 as the single remaining dstream connector in Spark, though. (If you ignore kinesis which we can't ship in binary form or something like that?) On Wed, Oct 10, 2018 at 1:32 PM Sean Owen wrote: > > Marcelo makes an argument that Flume support should be removed in > 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 > > I tend to agree. Is there an argument that it needs to be supported, > and can this move to Bahir if so? > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Remove Flume support in 3.0.0?
I think it makes sense to remove it. If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions. That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal. (Non-Voting just my opinion) > Am 10.10.2018 um 22:31 schrieb Sean Owen : > > Marcelo makes an argument that Flume support should be removed in > 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 > > I tend to agree. Is there an argument that it needs to be supported, > and can this move to Bahir if so? > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Remove Flume support in 3.0.0?
Marcelo makes an argument that Flume support should be removed in 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598 I tend to agree. Is there an argument that it needs to be supported, and can this move to Bahir if so? - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
