Re: [Proposal] => JMSIO dynamic topic publishing
All JMS related properties can be null (@Nullable). Good point about breaking change. The part on which I'm not big fan in your proposal is this getDynamic() method. Maybe we can mimic what I did in JdbcIO with a fn we can inject to define the destination. But, ok, I would be happy to review a PR (I can comment on the PR though). Regards JB On Thu, Feb 24, 2022 at 5:16 PM BALLADA Vincent wrote: > > Hi Jean Baptiste > > > > Thank you for your feedback, it is interesting. > > > > In your proposal, the write transform would take a PCollection< JmsRecord>. > > JmsRecord has a lot of property that are related to the read operation > (jmsRedelivered, correlationId, etc…) > > As a result I think it won’t be easy to create it with all relevant > properties at late stage of the pipeline, before the write operation. > > It would introduce also a breaking change as the current implementation takes > a Pcollection as input. > > Plus the new standard for IO implementation seems to be parameterized > read/write (see annotated document). > > > > Thanks, > > > > Regards > > > > Vincent BALLADA > > > > De : Jean-Baptiste Onofré > Date : jeudi, 24 février 2022 à 16:32 > À : dev@beam.apache.org > Objet : Re: [Proposal] => JMSIO dynamic topic publishing > > [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] > > Hi Vincent, > > It's Jean-Baptiste, not Jean-François, but it works as well ;) > > I got your point, however, I think we can achieve the same with > JmsRecord. You can always use a DoFn at any part of your pipeline > (after reading from kafka, redis, whatever), where you can create > JmsRecord. If JmsRecord contains a property with destination, it can > be populated dynamically JmsDestination. > > So basically, it means that we can add JmsDestinationType on the > JmsRecord and use it in the sink. > > Something lie: > > pipeline > .apply(...) // returns PCollection // JmsReccord can be > created/populated with destination and destination type > .apply(JmsIO.write().withConnectionFactory(myConnectionFactory)) > > I think it's less invasive. > > It's basically the same that you propose, but reusing JmsRecord and > avoiding getDynamic() hook. > > Again, I'm not against with your proposal, but we can have something > more concise. > > Just my €0.01 ;) > > Regards > JB > > On Thu, Feb 24, 2022 at 3:42 PM BALLADA Vincent > wrote: > > > > Hi Jean François > > > > > > > > Please to hear of the author of JmsIO . > > > > Many thanks for your suggestion. > > > > JmsRecord is used at read time, in most use cases we will use a mapper to > > provide an object that will be used in several transform in the pipeline. > > > > The destination won’t necessary be included in the read JmsRecord, as for > > instance in many pipelines we do data enrichment (form redis, database, > > etc…) and the dynamic part of the topic would be extracted at any stage of > > the pipeline. > > > > So for me it would be more flexible to not be tied with JmsRecord. > > > > > > > > Regards > > > > > > > > Vincent BALLADA > > > > > > > > De : Jean-Baptiste Onofré > > Date : samedi, 19 février 2022 à 06:40 > > À : dev@beam.apache.org > > Objet : Re: [Proposal] => JMSIO dynamic topic publishing > > > > [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] > > > > Hi Vincent, > > > > It looks interesting. Another possible approach is to have some > > implicit (instead of being explicit) but defining the destination on > > the JmsRecord. If the JmsRecord contains the destination (that could > > be "dynamic"), we use it, overriding the destination provided on the > > IO configuration. > > Thoughts ? > > > > Regards > > JB > > > > NB: I'm the original author of JmsIO ;) > > > > On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent > > wrote: > > > > > > Hi all > > > > > > > > > > > > Here is a proposal to implement the ability to publish on dynamic topics > > > with JMSIO: > > > > > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing > > > > > > > > > > > > There is also a JIRA issue: > > > > > > https://issues.apache.org/jira/browse/BEAM-13608 > > > > > > > > > > > > Best regards > > > > > > > > > &g
Re: [Proposal] => JMSIO dynamic topic publishing
Hi Jean Baptiste Thank you for your feedback, it is interesting. In your proposal, the write transform would take a PCollection< JmsRecord>. JmsRecord has a lot of property that are related to the read operation (jmsRedelivered, correlationId, etc…) As a result I think it won’t be easy to create it with all relevant properties at late stage of the pipeline, before the write operation. It would introduce also a breaking change as the current implementation takes a Pcollection as input. Plus the new standard for IO implementation seems to be parameterized read/write (see annotated document). Thanks, Regards Vincent BALLADA De : Jean-Baptiste Onofré Date : jeudi, 24 février 2022 à 16:32 À : dev@beam.apache.org Objet : Re: [Proposal] => JMSIO dynamic topic publishing [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] Hi Vincent, It's Jean-Baptiste, not Jean-François, but it works as well ;) I got your point, however, I think we can achieve the same with JmsRecord. You can always use a DoFn at any part of your pipeline (after reading from kafka, redis, whatever), where you can create JmsRecord. If JmsRecord contains a property with destination, it can be populated dynamically JmsDestination. So basically, it means that we can add JmsDestinationType on the JmsRecord and use it in the sink. Something lie: pipeline .apply(...) // returns PCollection // JmsReccord can be created/populated with destination and destination type .apply(JmsIO.write().withConnectionFactory(myConnectionFactory)) I think it's less invasive. It's basically the same that you propose, but reusing JmsRecord and avoiding getDynamic() hook. Again, I'm not against with your proposal, but we can have something more concise. Just my €0.01 ;) Regards JB On Thu, Feb 24, 2022 at 3:42 PM BALLADA Vincent wrote: > > Hi Jean François > > > > Please to hear of the author of JmsIO . > > Many thanks for your suggestion. > > JmsRecord is used at read time, in most use cases we will use a mapper to > provide an object that will be used in several transform in the pipeline. > > The destination won’t necessary be included in the read JmsRecord, as for > instance in many pipelines we do data enrichment (form redis, database, etc…) > and the dynamic part of the topic would be extracted at any stage of the > pipeline. > > So for me it would be more flexible to not be tied with JmsRecord. > > > > Regards > > > > Vincent BALLADA > > > > De : Jean-Baptiste Onofré > Date : samedi, 19 février 2022 à 06:40 > À : dev@beam.apache.org > Objet : Re: [Proposal] => JMSIO dynamic topic publishing > > [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] > > Hi Vincent, > > It looks interesting. Another possible approach is to have some > implicit (instead of being explicit) but defining the destination on > the JmsRecord. If the JmsRecord contains the destination (that could > be "dynamic"), we use it, overriding the destination provided on the > IO configuration. > Thoughts ? > > Regards > JB > > NB: I'm the original author of JmsIO ;) > > On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent > wrote: > > > > Hi all > > > > > > > > Here is a proposal to implement the ability to publish on dynamic topics > > with JMSIO: > > > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing > > > > > > > > There is also a JIRA issue: > > > > https://issues.apache.org/jira/browse/BEAM-13608 > > > > > > > > Best regards > > > > > > > > Vincent BALLADA > > > > > > Confidential C > > > > -- Disclaimer > > Ce message ainsi que les eventuelles pieces jointes constituent une > > correspondance privee et confidentielle a l'attention exclusive du > > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du > > present message ou une personne susceptible de pouvoir le lui delivrer, il > > vous est signifie que toute divulgation, distribution ou copie de cette > > transmission est strictement interdite. Si vous avez recu ce message par > > erreur, nous vous remercions d'en informer l'expediteur par telephone ou de > > lui retourner le present message, puis d'effacer immediatement ce message > > de votre systeme. > > > > *** This e-mail and any attachments is a confidential correspondence > > intended only for use of the individual or entity named above. If you are > > not the intended recipient or the agent responsible for delivering the > > message to the intended recipient, you are hereby
Re: [Proposal] => JMSIO dynamic topic publishing
Hi Vincent, It's Jean-Baptiste, not Jean-François, but it works as well ;) I got your point, however, I think we can achieve the same with JmsRecord. You can always use a DoFn at any part of your pipeline (after reading from kafka, redis, whatever), where you can create JmsRecord. If JmsRecord contains a property with destination, it can be populated dynamically JmsDestination. So basically, it means that we can add JmsDestinationType on the JmsRecord and use it in the sink. Something lie: pipeline .apply(...) // returns PCollection // JmsReccord can be created/populated with destination and destination type .apply(JmsIO.write().withConnectionFactory(myConnectionFactory)) I think it's less invasive. It's basically the same that you propose, but reusing JmsRecord and avoiding getDynamic() hook. Again, I'm not against with your proposal, but we can have something more concise. Just my €0.01 ;) Regards JB On Thu, Feb 24, 2022 at 3:42 PM BALLADA Vincent wrote: > > Hi Jean François > > > > Please to hear of the author of JmsIO . > > Many thanks for your suggestion. > > JmsRecord is used at read time, in most use cases we will use a mapper to > provide an object that will be used in several transform in the pipeline. > > The destination won’t necessary be included in the read JmsRecord, as for > instance in many pipelines we do data enrichment (form redis, database, etc…) > and the dynamic part of the topic would be extracted at any stage of the > pipeline. > > So for me it would be more flexible to not be tied with JmsRecord. > > > > Regards > > > > Vincent BALLADA > > > > De : Jean-Baptiste Onofré > Date : samedi, 19 février 2022 à 06:40 > À : dev@beam.apache.org > Objet : Re: [Proposal] => JMSIO dynamic topic publishing > > [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] > > Hi Vincent, > > It looks interesting. Another possible approach is to have some > implicit (instead of being explicit) but defining the destination on > the JmsRecord. If the JmsRecord contains the destination (that could > be "dynamic"), we use it, overriding the destination provided on the > IO configuration. > Thoughts ? > > Regards > JB > > NB: I'm the original author of JmsIO ;) > > On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent > wrote: > > > > Hi all > > > > > > > > Here is a proposal to implement the ability to publish on dynamic topics > > with JMSIO: > > > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing > > > > > > > > There is also a JIRA issue: > > > > https://issues.apache.org/jira/browse/BEAM-13608 > > > > > > > > Best regards > > > > > > > > Vincent BALLADA > > > > > > Confidential C > > > > -- Disclaimer > > Ce message ainsi que les eventuelles pieces jointes constituent une > > correspondance privee et confidentielle a l'attention exclusive du > > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du > > present message ou une personne susceptible de pouvoir le lui delivrer, il > > vous est signifie que toute divulgation, distribution ou copie de cette > > transmission est strictement interdite. Si vous avez recu ce message par > > erreur, nous vous remercions d'en informer l'expediteur par telephone ou de > > lui retourner le present message, puis d'effacer immediatement ce message > > de votre systeme. > > > > *** This e-mail and any attachments is a confidential correspondence > > intended only for use of the individual or entity named above. If you are > > not the intended recipient or the agent responsible for delivering the > > message to the intended recipient, you are hereby notified that any > > disclosure, distribution or copying of this communication is strictly > > prohibited. If you have received this communication in error, please notify > > the sender by phone or by replying this message, and then delete this > > message from your system. > > > > > Confidential C > > -- Disclaimer > Ce message ainsi que les eventuelles pieces jointes constituent une > correspondance privee et confidentielle a l'attention exclusive du > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present > message ou une personne susceptible de pouvoir le lui delivrer, il vous est > signifie que toute divulgation, distribution ou copie de cette transmission > est strictement interdite. Si vous avez recu ce message par erreur, n
Re: [Proposal] => JMSIO dynamic topic publishing
Hi Jean François Please to hear of the author of JmsIO . Many thanks for your suggestion. JmsRecord is used at read time, in most use cases we will use a mapper to provide an object that will be used in several transform in the pipeline. The destination won’t necessary be included in the read JmsRecord, as for instance in many pipelines we do data enrichment (form redis, database, etc…) and the dynamic part of the topic would be extracted at any stage of the pipeline. So for me it would be more flexible to not be tied with JmsRecord. Regards Vincent BALLADA De : Jean-Baptiste Onofré Date : samedi, 19 février 2022 à 06:40 À : dev@beam.apache.org Objet : Re: [Proposal] => JMSIO dynamic topic publishing [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments] Hi Vincent, It looks interesting. Another possible approach is to have some implicit (instead of being explicit) but defining the destination on the JmsRecord. If the JmsRecord contains the destination (that could be "dynamic"), we use it, overriding the destination provided on the IO configuration. Thoughts ? Regards JB NB: I'm the original author of JmsIO ;) On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent wrote: > > Hi all > > > > Here is a proposal to implement the ability to publish on dynamic topics with > JMSIO: > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing > > > > There is also a JIRA issue: > > https://issues.apache.org/jira/browse/BEAM-13608 > > > > Best regards > > > > Vincent BALLADA > > > Confidential C > > -- Disclaimer > Ce message ainsi que les eventuelles pieces jointes constituent une > correspondance privee et confidentielle a l'attention exclusive du > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present > message ou une personne susceptible de pouvoir le lui delivrer, il vous est > signifie que toute divulgation, distribution ou copie de cette transmission > est strictement interdite. Si vous avez recu ce message par erreur, nous vous > remercions d'en informer l'expediteur par telephone ou de lui retourner le > present message, puis d'effacer immediatement ce message de votre systeme. > > *** This e-mail and any attachments is a confidential correspondence intended > only for use of the individual or entity named above. If you are not the > intended recipient or the agent responsible for delivering the message to the > intended recipient, you are hereby notified that any disclosure, distribution > or copying of this communication is strictly prohibited. If you have received > this communication in error, please notify the sender by phone or by replying > this message, and then delete this message from your system. Confidential C -- Disclaimer Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme. *** This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system.
Re: [Proposal] => JMSIO dynamic topic publishing
Hi Vincent, It looks interesting. Another possible approach is to have some implicit (instead of being explicit) but defining the destination on the JmsRecord. If the JmsRecord contains the destination (that could be "dynamic"), we use it, overriding the destination provided on the IO configuration. Thoughts ? Regards JB NB: I'm the original author of JmsIO ;) On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent wrote: > > Hi all > > > > Here is a proposal to implement the ability to publish on dynamic topics with > JMSIO: > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing > > > > There is also a JIRA issue: > > https://issues.apache.org/jira/browse/BEAM-13608 > > > > Best regards > > > > Vincent BALLADA > > > Confidential C > > -- Disclaimer > Ce message ainsi que les eventuelles pieces jointes constituent une > correspondance privee et confidentielle a l'attention exclusive du > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present > message ou une personne susceptible de pouvoir le lui delivrer, il vous est > signifie que toute divulgation, distribution ou copie de cette transmission > est strictement interdite. Si vous avez recu ce message par erreur, nous vous > remercions d'en informer l'expediteur par telephone ou de lui retourner le > present message, puis d'effacer immediatement ce message de votre systeme. > > *** This e-mail and any attachments is a confidential correspondence intended > only for use of the individual or entity named above. If you are not the > intended recipient or the agent responsible for delivering the message to the > intended recipient, you are hereby notified that any disclosure, distribution > or copying of this communication is strictly prohibited. If you have received > this communication in error, please notify the sender by phone or by replying > this message, and then delete this message from your system.
[Proposal] => JMSIO dynamic topic publishing
Hi all Here is a proposal to implement the ability to publish on dynamic topics with JMSIO: https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing There is also a JIRA issue: https://issues.apache.org/jira/browse/BEAM-13608 Best regards Vincent BALLADA Confidential C -- Disclaimer Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme. *** This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system.
