Re: delayed emit (timer) in py-beam?
FYI: the reference DirectRunner implementation of the Python user state and timers API is out for review: https://github.com/apache/beam/pull/6304 On Mon, Jul 30, 2018 at 3:57 PM Austin Bennett wrote: > Fantastic; thanks, Charles! > > > > On Mon, Jul 30, 2018 at 3:49 PM, Charles Chen wrote: > >> Hey Austin, >> >> This API is not yet implemented in the Python SDK. I am working on this >> feature: the next step from my end is to finish a reference implementation >> in the local DirectRunner. As you note, the doc at >> https://s.apache.org/beam-python-user-state-and-timers describes the >> design. >> >> You can track progress on the mailing list thread here: >> https://lists.apache.org/thread.html/51ba1a00027ad8635bc1d2c0df805ce873995170c75d6a08dfe21997@%3Cdev.beam.apache.org%3E >> >> Best, >> Charles >> >> On Mon, Jul 30, 2018 at 3:34 PM Austin Bennett < >> whatwouldausti...@gmail.com> wrote: >> >>> What's going on with timers and python? >>> >>> Am looking at building a pipeline (assuming another group in my company >>> will grant access to the Kafka topic): >>> >>> Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a >>> record. If I read things correctly that's not currently possible in python >>> on beam. What all is needed? (trying to figure out whether that is >>> something that I am capable of and there is room for me to help with). >>> Looking for similar functionality to >>> https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ >>> (though don't need alternate routing, nor is that example in python). >>> >>> >>> For example, I see: >>> https://beam.apache.org/blog/2017/08/28/timely-processing.html >>> >>> and tickets like: https://issues.apache.org/jira/browse/BEAM-4594 >>> >>> >>> >
Re: delayed emit (timer) in py-beam?
Fantastic; thanks, Charles! On Mon, Jul 30, 2018 at 3:49 PM, Charles Chen wrote: > Hey Austin, > > This API is not yet implemented in the Python SDK. I am working on this > feature: the next step from my end is to finish a reference implementation > in the local DirectRunner. As you note, the doc at > https://s.apache.org/beam-python-user-state-and-timers describes the > design. > > You can track progress on the mailing list thread here: > https://lists.apache.org/thread.html/51ba1a00027ad8635bc1d2c0df805c > e873995170c75d6a08dfe21997@%3Cdev.beam.apache.org%3E > > Best, > Charles > > On Mon, Jul 30, 2018 at 3:34 PM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> What's going on with timers and python? >> >> Am looking at building a pipeline (assuming another group in my company >> will grant access to the Kafka topic): >> >> Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a >> record. If I read things correctly that's not currently possible in python >> on beam. What all is needed? (trying to figure out whether that is >> something that I am capable of and there is room for me to help with). >> Looking for similar functionality to https://www.rabbitmq.com/blog/ >> 2015/04/16/scheduling-messages-with-rabbitmq/ (though don't need >> alternate routing, nor is that example in python). >> >> >> For example, I see: https://beam.apache.org/blog/ >> 2017/08/28/timely-processing.html >> >> and tickets like: https://issues.apache.org/jira/browse/BEAM-4594 >> >> >>
Re: delayed emit (timer) in py-beam?
Hey Austin, This API is not yet implemented in the Python SDK. I am working on this feature: the next step from my end is to finish a reference implementation in the local DirectRunner. As you note, the doc at https://s.apache.org/beam-python-user-state-and-timers describes the design. You can track progress on the mailing list thread here: https://lists.apache.org/thread.html/51ba1a00027ad8635bc1d2c0df805ce873995170c75d6a08dfe21997@%3Cdev.beam.apache.org%3E Best, Charles On Mon, Jul 30, 2018 at 3:34 PM Austin Bennett wrote: > What's going on with timers and python? > > Am looking at building a pipeline (assuming another group in my company > will grant access to the Kafka topic): > > Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a > record. If I read things correctly that's not currently possible in python > on beam. What all is needed? (trying to figure out whether that is > something that I am capable of and there is room for me to help with). > Looking for similar functionality to > https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ > (though don't need alternate routing, nor is that example in python). > > > For example, I see: > https://beam.apache.org/blog/2017/08/28/timely-processing.html > > and tickets like: https://issues.apache.org/jira/browse/BEAM-4594 > > >
delayed emit (timer) in py-beam?
What's going on with timers and python? Am looking at building a pipeline (assuming another group in my company will grant access to the Kafka topic): Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a record. If I read things correctly that's not currently possible in python on beam. What all is needed? (trying to figure out whether that is something that I am capable of and there is room for me to help with). Looking for similar functionality to https://www.rabbitmq.com/blog/ 2015/04/16/scheduling-messages-with-rabbitmq/ (though don't need alternate routing, nor is that example in python). For example, I see: https://beam.apache.org/blog/ 2017/08/28/timely-processing.html and tickets like: https://issues.apache.org/jira/browse/BEAM-4594