Re: [DISCUSS] Removing code related to `FireHose`
I am in favor of immediately removing FiniteFirehoseFactory and marking EventReceiverFirehoseFactory deprecated. Then, later on we can remove InputRowParser and EventReceiverFirehoseFactory. On Fri, Jun 24, 2022 at 4:41 AM Abhishek Agarwal wrote: > I didn’t include them (RealtimeIndexTask and > AppenderatorDriverRealtimeIndexTask) in my previous email because they have > not been marked deprecated yet. We should mark them deprecated officially > in the next release and remove them in the release after that. > > So looks like the classes that we can definitely remove are implementations > of `FiniteFirehoseFactory` and mark the `Firehose` interface deprecated. > > On Fri, 24 Jun 2022 at 4:36 AM, Clint Wylie wrote: > > > If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask > > then we can remove EventReceiverFirehoseFactory. The former was > > primarily used by tranquility which has been sunset, the latter I'm > > not sure was ever used for anything. I'm personally in favor of > > removing both of them since push based ingestion is very fragile in my > > experience, but I think some of the oldest integration tests use > > RealtimeIndexTask and so would need to be removed/updated/rewritten to > > use something else as appropriate. > > > > I don't think we can completely remove InputRowParser until we drop > > Hadoop support (or modify Hadoop ingestion to use > > InputSource/InputFormat?), since it still relies on using the older > > spec. As far as I know, Thrift is the only data format that has not > > been fully migrated to use InputFormat, though there is an old PR that > > is mostly done here https://github.com/apache/druid/pull/11360. > > > > On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal > > wrote: > > > > > > Hello, > > > The `FiniteFirehoseFactory` and `InputRowParser` classes were > deprecated > > in > > > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of > > > `InputSource`. 0.17.0 was released more than 2 years ago in Jan 2020. > > > > > > I think it is about time that we remove this code entirely. Removing > > > `InputRowParser` may not be as trivial as > `EventReceiverFirehoseFactory` > > > depends on it. I didn't find any alternatives for > > > `EventReceiverFirehoseFactory` and it is not marked deprecated as well. > > > > > > But we can still remove `FiniteFirehoseFactory` and the implementations > > > safely as there are alternatives available. > > > > > > Thoughts/Suggestions? > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > >
Re: [DISCUSS] Removing code related to `FireHose`
@Clint Wylie - what do you think of the above proposal? On Fri, Jun 24, 2022 at 5:11 PM Abhishek Agarwal wrote: > I didn’t include them (RealtimeIndexTask and > AppenderatorDriverRealtimeIndexTask) in my previous email because they have > not been marked deprecated yet. We should mark them deprecated officially > in the next release and remove them in the release after that. > > So looks like the classes that we can definitely remove are > implementations of `FiniteFirehoseFactory` and mark the `Firehose` > interface deprecated. > > On Fri, 24 Jun 2022 at 4:36 AM, Clint Wylie wrote: > >> If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask >> then we can remove EventReceiverFirehoseFactory. The former was >> primarily used by tranquility which has been sunset, the latter I'm >> not sure was ever used for anything. I'm personally in favor of >> removing both of them since push based ingestion is very fragile in my >> experience, but I think some of the oldest integration tests use >> RealtimeIndexTask and so would need to be removed/updated/rewritten to >> use something else as appropriate. >> >> I don't think we can completely remove InputRowParser until we drop >> Hadoop support (or modify Hadoop ingestion to use >> InputSource/InputFormat?), since it still relies on using the older >> spec. As far as I know, Thrift is the only data format that has not >> been fully migrated to use InputFormat, though there is an old PR that >> is mostly done here https://github.com/apache/druid/pull/11360. >> >> On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal >> wrote: >> > >> > Hello, >> > The `FiniteFirehoseFactory` and `InputRowParser` classes were >> deprecated in >> > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of >> > `InputSource`. 0.17.0 was released more than 2 years ago in Jan 2020. >> > >> > I think it is about time that we remove this code entirely. Removing >> > `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory` >> > depends on it. I didn't find any alternatives for >> > `EventReceiverFirehoseFactory` and it is not marked deprecated as well. >> > >> > But we can still remove `FiniteFirehoseFactory` and the implementations >> > safely as there are alternatives available. >> > >> > Thoughts/Suggestions? >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org >> For additional commands, e-mail: dev-h...@druid.apache.org >> >>
Re: [DISCUSS] Removing code related to `FireHose`
I didn’t include them (RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask) in my previous email because they have not been marked deprecated yet. We should mark them deprecated officially in the next release and remove them in the release after that. So looks like the classes that we can definitely remove are implementations of `FiniteFirehoseFactory` and mark the `Firehose` interface deprecated. On Fri, 24 Jun 2022 at 4:36 AM, Clint Wylie wrote: > If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask > then we can remove EventReceiverFirehoseFactory. The former was > primarily used by tranquility which has been sunset, the latter I'm > not sure was ever used for anything. I'm personally in favor of > removing both of them since push based ingestion is very fragile in my > experience, but I think some of the oldest integration tests use > RealtimeIndexTask and so would need to be removed/updated/rewritten to > use something else as appropriate. > > I don't think we can completely remove InputRowParser until we drop > Hadoop support (or modify Hadoop ingestion to use > InputSource/InputFormat?), since it still relies on using the older > spec. As far as I know, Thrift is the only data format that has not > been fully migrated to use InputFormat, though there is an old PR that > is mostly done here https://github.com/apache/druid/pull/11360. > > On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal > wrote: > > > > Hello, > > The `FiniteFirehoseFactory` and `InputRowParser` classes were deprecated > in > > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of > > `InputSource`. 0.17.0 was released more than 2 years ago in Jan 2020. > > > > I think it is about time that we remove this code entirely. Removing > > `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory` > > depends on it. I didn't find any alternatives for > > `EventReceiverFirehoseFactory` and it is not marked deprecated as well. > > > > But we can still remove `FiniteFirehoseFactory` and the implementations > > safely as there are alternatives available. > > > > Thoughts/Suggestions? > > - > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
Re: [DISCUSS] Removing code related to `FireHose`
If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask then we can remove EventReceiverFirehoseFactory. The former was primarily used by tranquility which has been sunset, the latter I'm not sure was ever used for anything. I'm personally in favor of removing both of them since push based ingestion is very fragile in my experience, but I think some of the oldest integration tests use RealtimeIndexTask and so would need to be removed/updated/rewritten to use something else as appropriate. I don't think we can completely remove InputRowParser until we drop Hadoop support (or modify Hadoop ingestion to use InputSource/InputFormat?), since it still relies on using the older spec. As far as I know, Thrift is the only data format that has not been fully migrated to use InputFormat, though there is an old PR that is mostly done here https://github.com/apache/druid/pull/11360. On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal wrote: > > Hello, > The `FiniteFirehoseFactory` and `InputRowParser` classes were deprecated in > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of > `InputSource`. 0.17.0 was released more than 2 years ago in Jan 2020. > > I think it is about time that we remove this code entirely. Removing > `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory` > depends on it. I didn't find any alternatives for > `EventReceiverFirehoseFactory` and it is not marked deprecated as well. > > But we can still remove `FiniteFirehoseFactory` and the implementations > safely as there are alternatives available. > > Thoughts/Suggestions? - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
[DISCUSS] Removing code related to `FireHose`
Hello, The `FiniteFirehoseFactory` and `InputRowParser` classes were deprecated in 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of `InputSource`. 0.17.0 was released more than 2 years ago in Jan 2020. I think it is about time that we remove this code entirely. Removing `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory` depends on it. I didn't find any alternatives for `EventReceiverFirehoseFactory` and it is not marked deprecated as well. But we can still remove `FiniteFirehoseFactory` and the implementations safely as there are alternatives available. Thoughts/Suggestions?
