Hi I'm completly fine to contribute, let's start with Cloud SQL and Firestore. Get me right: I've already written a working writer and sink. So how to make them native so they will be able to be supported in the streamline pipeline? Just inheritance from the native_io?
My sinks inherit from iobase.Sink and my Writers from iobase.Writer. Regards, *Sébastien MORAND* Team Lead Solution Architect Technology & Operations / Digital Factory Veolia - Group Information Systems & Technology (IS&T) Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08 Bureau 0144C (Ouest) 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France *www.veolia.com <http://www.veolia.com>* <http://www.veolia.com> <https://www.facebook.com/veoliaenvironment/> <https://www.youtube.com/user/veoliaenvironnement> <https://www.linkedin.com/company/veolia-environnement> <https://twitter.com/veolia> On Wed, 27 Jun 2018 at 22:30, Ismaël Mejía <ieme...@gmail.com> wrote: > It seems like a nice opportunity also to contribute them to the project in > case you are able to work on them. > Don't hesitate to contact us or ask for help if needed. > > > On Wed, Jun 27, 2018 at 4:45 PM Ahmet Altay <al...@google.com> wrote: > >> Hi Sébastien, >> >> Currently there is no work in progress for including the write transforms >> for the locations you listed. You could develop your own version if >> interested. Please see WriteToBigquery transform [1] for reference. >> >> Ahmet >> >> [1] >> https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287 >> >> On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand < >> sebastien.mor...@veolia.com> wrote: >> >>> Hi, >>> >>> Thanks for your answer. >>> >>> Ok looking forward and ready to test alpha on this. Because we have >>> actually some use cases to send data to CloudSQL or Spanner or BigTable or >>> Firestore. As far as I read the documentation, there is no native support >>> for them and so we have already implemented a custom source support. >>> >>> So is there any work in progress for any of the 4 sinks above for native >>> support in the python SDK? >>> >>> Regards, >>> >>> *Sébastien MORAND* >>> Team Lead Solution Architect >>> Technology & Operations / Digital Factory >>> Veolia - Group Information Systems & Technology (IS&T) >>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08 >>> Bureau 0144C (Ouest) >>> 30, rue Madeleine-Vionnet >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>> - >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300 >>> Aubervilliers, France >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>> *www.veolia.com <http://www.veolia.com>* >>> <http://www.veolia.com> >>> <https://www.facebook.com/veoliaenvironment/> >>> <https://www.youtube.com/user/veoliaenvironnement> >>> <https://www.linkedin.com/company/veolia-environnement> >>> <https://twitter.com/veolia> >>> >>> >>> On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote: >>> >>>> +dev@beam.apache.org >>>> >>>> Python streaming custom source support will be available via >>>> SplittableDoFn. It is actively being worked on by a few contributors but to >>>> my knowledge there is no roadmap yet for having support for this for >>>> Dataflow. >>>> >>>> >>>> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand < >>>> sebastien.mor...@veolia.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> We need to setup streaming dataflow in python using developed source >>>>> (in the opposite of native source) for Cloud SQL integration and Firestore >>>>> integration. >>>>> >>>>> This is currently not supported as far as I understood the >>>>> documentation, can you confirm? Any roadmap on the topic? >>>>> >>>>> Thanks by advance, >>>>> Regards, >>>>> >>>>> *Sébastien MORAND* >>>>> Team Lead Solution Architect >>>>> Technology & Operations / Digital Factory >>>>> Veolia - Group Information Systems & Technology (IS&T) >>>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08 >>>>> Bureau 0144C (Ouest) >>>>> 30, rue Madeleine-Vionnet >>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>>>> - >>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300 >>>>> Aubervilliers, France >>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>>>> *www.veolia.com <http://www.veolia.com>* >>>>> <http://www.veolia.com> >>>>> <https://www.facebook.com/veoliaenvironment/> >>>>> <https://www.youtube.com/user/veoliaenvironnement> >>>>> <https://www.linkedin.com/company/veolia-environnement> >>>>> <https://twitter.com/veolia> >>>>> >>>>> >>>>> >>>>> -------------------------------------------------------------------------------------------- >>>>> This e-mail transmission (message and any attached files) may contain >>>>> information that is proprietary, privileged and/or confidential to Veolia >>>>> Environnement and/or its affiliates and is intended exclusively for the >>>>> person(s) to whom it is addressed. If you are not the intended recipient, >>>>> please notify the sender by return e-mail and delete all copies of this >>>>> e-mail, including all attachments. Unless expressly authorized, any use, >>>>> disclosure, publication, retransmission or dissemination of this e-mail >>>>> and/or of its attachments is strictly prohibited. >>>>> >>>>> Ce message electronique et ses fichiers attaches sont strictement >>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>>>> message par erreur, merci de le retourner a son emetteur et de le detruire >>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>>>> publication, la distribution, ou la reproduction non expressement >>>>> autorisees de ce message et de ses pieces attachees sont interdites. >>>>> >>>>> -------------------------------------------------------------------------------------------- >>>>> >>>> >>> >>> >>> -------------------------------------------------------------------------------------------- >>> This e-mail transmission (message and any attached files) may contain >>> information that is proprietary, privileged and/or confidential to Veolia >>> Environnement and/or its affiliates and is intended exclusively for the >>> person(s) to whom it is addressed. If you are not the intended recipient, >>> please notify the sender by return e-mail and delete all copies of this >>> e-mail, including all attachments. Unless expressly authorized, any use, >>> disclosure, publication, retransmission or dissemination of this e-mail >>> and/or of its attachments is strictly prohibited. >>> >>> Ce message electronique et ses fichiers attaches sont strictement >>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>> message par erreur, merci de le retourner a son emetteur et de le detruire >>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>> publication, la distribution, ou la reproduction non expressement >>> autorisees de ce message et de ses pieces attachees sont interdites. >>> >>> -------------------------------------------------------------------------------------------- >>> >> >> -- --------------------------------------------------------------------------------------------This e-mail transmission (message and any attached files) may contain information that is proprietary, privileged and/or confidential to Veolia Environnement and/or its affiliates and is intended exclusively for the person(s) to whom it is addressed. If you are not the intended recipient, please notify the sender by return e-mail and delete all copies of this e-mail, including all attachments. Unless expressly authorized, any use, disclosure, publication, retransmission or dissemination of this e-mail and/or of its attachments is strictly prohibited. Ce message electronique et ses fichiers attaches sont strictement confidentiels et peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc destines a l'usage de leurs seuls destinataires. Si vous avez recu ce message par erreur, merci de le retourner a son emetteur et de le detruire ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la publication, la distribution, ou la reproduction non expressement autorisees de ce message et de ses pieces attachees sont interdites. --------------------------------------------------------------------------------------------
