Re: [VOTE] Release 2.52.0, release candidate #5

[Jean-Baptiste Onofré]

+1 (binding)

Quickly tested Java SDK and checked the legal part (hash, signatures, headers).

Regards
JB

On Tue, Nov 14, 2023 at 12:06 AM Danny McCormick via dev
 wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #5 for the version 2.52.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found. Only PMC member votes will 
> count towards the final vote, but votes from all community members is 
> encouraged and helpful for finding regressions; you can either test your own 
> use cases or use cases from the validation sheet [10].
>
> The complete staging area is available for your review, which includes:
>
> GitHub Release notes [1]
> the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint D20316F712213422 [3]
> all artifacts to be deployed to the Maven Central Repository [4]
> source code tag "v2.52.0-RC5" [5]
> website pull request listing the release [6], the blog post [6], and 
> publishing the API reference manual [7]
> Python artifacts are deployed along with the source release to the 
> dist.apache.org [2] and PyPI[8].
> Go artifacts and documentation are available at pkg.go.dev [9]
> Validation sheet with a tab for 2.52.0 release to help with validation [10]
> Docker images published to Docker Hub [11]
> PR to run tests against release branch [12]
>
>
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our blog 
> post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Danny
>
> [1] https://github.com/apache/beam/milestone/16
> [2] https://dist.apache.org/repos/dist/dev/beam/2.52.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1363/
> [5] https://github.com/apache/beam/tree/v2.52.0-RC5
> [6] https://github.com/apache/beam/pull/29331
> [7] https://github.com/apache/beam-site/pull/655
> [8] https://pypi.org/project/apache-beam/2.52.0rc5/
> [9] https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.52.0-RC5/go/pkg/beam
> [10] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1387982510
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/29418

Re: [VOTE] Release 2.52.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Wed, Nov 8, 2023 at 12:24 AM Danny McCormick via dev
 wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #2 for the version 2.52.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found. Only PMC member votes will 
> count towards the final vote, but votes from all community members is 
> encouraged and helpful for finding regressions; you can either test your own 
> use cases or use cases from the validation sheet [10].
>
> The complete staging area is available for your review, which includes:
>
> GitHub Release notes [1]
> the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint D20316F712213422 [3]
> all artifacts to be deployed to the Maven Central Repository [4]
> source code tag "v2.52.0-RC1" [5]
> website pull request listing the release [6], the blog post [6], and 
> publishing the API reference manual [7]
> Python artifacts are deployed along with the source release to the 
> dist.apache.org [2] and PyPI[8].
> Go artifacts and documentation are available at pkg.go.dev [9]
> Validation sheet with a tab for 2.52.0 release to help with validation [10]
> Docker images published to Docker Hub [11]
> PR to run tests against release branch [12]
>
>
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our blog 
> post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Danny
>
> [1] https://github.com/apache/beam/milestone/16
> [2] https://dist.apache.org/repos/dist/dev/beam/2.52.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1360/
> [5] https://github.com/apache/beam/tree/v2.52.0-RC2
> [6] https://github.com/apache/beam/pull/29331
> [7] https://github.com/apache/beam-site/pull/652
> [8] https://pypi.org/project/apache-beam/2.52.0rc2/
> [9] https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.52.0-RC2/go/pkg/beam
> [10] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1387982510
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/29319

Re: [VOTE] Release 2.51.0, release candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Thanks !
Regards
JB

On Tue, Oct 3, 2023 at 7:58 PM Kenneth Knowles  wrote:
>
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 2.51.0, as 
> follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found. Only PMC member votes will 
> count towards the final vote, but votes from all community members is 
> encouraged and helpful for finding regressions; you can either test your own 
> use cases or use cases from the validation sheet [10].
>
> The complete staging area is available for your review, which includes:
>
> GitHub Release notes [1],
> the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint  [3],
> all artifacts to be deployed to the Maven Central Repository [4],
> source code tag "v1.2.3-RC3" [5],
> website pull request listing the release [6], the blog post [6], and 
> publishing the API reference manual [7].
> Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle JDK 
> JDK_VERSION.
> Python artifacts are deployed along with the source release to the 
> dist.apache.org [2] and PyPI[8].
> Go artifacts and documentation are available at pkg.go.dev [9]
> Validation sheet with a tab for 1.2.3 release to help with validation [10].
> Docker images published to Docker Hub [11].
> PR to run tests against release branch [12].
>
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our blog 
> post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Kenn
>
> [1] https://github.com/apache/beam/milestone/15
> [2] https://dist.apache.org/repos/dist/dev/beam/2.51.0
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1356/
> [5] https://github.com/apache/beam/tree/v2.51.0-RC1
> [6] https://github.com/apache/beam/pull/28800
> [7] https://github.com/apache/beam-site/pull/649
> [8] https://pypi.org/project/apache-beam/2.51.0rc1/
> [9] https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.51.0-RC1/go/pkg/beam
> [10] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=437054928
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/28663

Re: [PROPOSAL] Preparing for 2.51.0 Release

[Jean-Baptiste Onofré]

Awesome ! Thanks Kenn !

Regards
JB

On Thu, Sep 14, 2023 at 3:20 AM Kenneth Knowles  wrote:
>
> Hello Beam community!
>
> The next release (2.51.0) branch cut is scheduled for September 20, 2023, one 
> week from today, according to the release calendar [1].
>
> I'd like to volunteer to perform this release. My plan is to cut the branch 
> on that date, and cherrypick release-blocking fixes afterwards, if any.
>
> Please help me make sure the release goes smoothly by:
>
> - Making sure that any unresolved release blocking issues for 2.51.0 should 
> have their "Milestone" marked as "2.51.0 Release" as soon as possible.
>
> - Reviewing the current release blockers [2] and remove the Milestone if they 
> don't meet the criteria at [3]. There are currently 12 release blockers.
>
> Let me know if you have any comments/objections/questions.
>
> Thanks,
>
> Kenn
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/15
> [3] https://beam.apache.org/contribute/release-blocking/

Re: DRAFT - Apache Beam Board Report - June 2023

[Jean-Baptiste Onofré]

It looks good to me, thanks !

Regards
JB

On Fri, Jun 9, 2023 at 11:28 PM Kenneth Knowles  wrote:
>
> Hi all,
>
> The next Beam board report is due next Wednesday, June 14. Please help me to 
> draft it at https://s.apache.org/beam-draft-report-2023-06.
>
> Ideas:
>
>  - highlights from CHANGES.md
>  - interesting technical discussions
>  - integrations with other projects
>  - community events
>  - major user facing addition/deprecation
>  - stuff that will be presented at Beam Summit next week :-)
>
> Past reports are at https://whimsy.apache.org/board/minutes/Beam.html for 
> examples.
>
> I will edit the final version from everyone's suggestions.
>
> Thanks,
>
> Kenn
>
>

Re: 2.48.0 Release PMC Finalization

[Jean-Baptiste Onofré]

Hi

I can do that today if there's no other PMC available.

Regards
JB

On Thu, Jun 1, 2023 at 2:02 AM Ritesh Ghorse via dev
 wrote:
>
> Could a PMC member help with the PMC-only finalization steps for 2.48.0 [1]? 
> Specifically:
>
> - Deploy source release to dist.apache.org
> - Recordkeeping with ASF
>
> Once those steps are done all that's left is to promote the release [2].
>
> Thank you!
>
> [1] https://beam.apache.org/contribute/release-guide/#pmc-only-finalization
> [2] https://beam.apache.org/contribute/release-guide/#12-promote-the-release

Re: [VOTE] Release 2.47.0, release candidate #3

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Fri, May 5, 2023 at 4:52 AM Jack McCluskey via dev 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #3 for the version 2.47.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if no issues are found. *Non-PMC members are
> allowed and encouraged to vote. Please help validate the release for your
> use case!*
>
> The complete staging area is available for your review, which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2],
> which is signed with the key with fingerprint DF3CBA4F3F4199F4 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.47.0-RC3" [5],
> * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle 7.5.1 and OpenJDK/Oracle JDK
> 8.0.322.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI[8].
> * Go artifacts and documentation are available at pkg.go.dev [9]
> * Validation sheet with a tab for 2.47.0 release to help with validation
> [10].
> * Docker images published to Docker Hub [11].
> * PR to run tests against release branch [12].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> The GCR copies of the FnAPI containers are rolling out now, they should be
> out within the next 8 hours or so.
>
> For guidelines on how to try the release in your projects, check out our
> blog post at /blog/validate-beam-release/.
>
> Thanks,
>
> Jack McCluskey
>
> [1] https://github.com/apache/beam/milestone/10
> [2] https://dist.apache.org/repos/dist/dev/beam/2.47.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1322/
> [5] https://github.com/apache/beam/tree/v2.47.0-RC3
> [6] https://github.com/apache/beam/pull/26439
> [7] https://github.com/apache/beam-site/pull/644
> [8] https://pypi.org/project/apache-beam/2.47.0rc3/
> [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.47.0-RC3/go/pkg/beam
> [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=.
> ..
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/26152
>
> --
>
>
> Jack McCluskey
> SWE - DataPLS PLAT/ Dataflow ML
> RDU
> jrmcclus...@google.com
>
>
>

Re: [VOTE] Release 2.43.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Sun, Nov 13, 2022 at 3:52 PM Chamikara Jayalath via dev
 wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #2 for the version 2.43.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if
> no issues are found.
>
> The complete staging area is available for your review, which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint 
> 40C61FBE1761E5DB652A1A780CCD5EB2A718A56E [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.43.0-RC2" [5],
> * website pull request listing the release [6], the blog post [6], and 
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle 7.5.1 and openjdk version 
> 1.8.0_181-google-v7.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org [2] and PyPI[8].
> * Go artifacts and documentation are available at pkg.go.dev [9]
> * Validation sheet with a tab for 2.43.0 release to help with validation [10].
> * Docker images published to Docker Hub [11].
>
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our blog 
> post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Cham
>
> [1] https://github.com/apache/beam/milestone/5
> [2] https://dist.apache.org/repos/dist/dev/beam/2.43.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1288/
> [5] https://github.com/apache/beam/tree/v2.43.0-RC2
> [6] https://github.com/apache/beam/pull/24044
> [7] https://github.com/apache/beam-site/pull/636
> [8] https://pypi.org/project/apache-beam/2.43.0rc2/
> [9] https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.43.0-RC2/go/pkg/beam
> [10] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1310009119
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: FOSDEM 2023 is back as in person event

[Jean-Baptiste Onofré]

Hi,

Yes, I will be at FOSDEM 23, and I will help on the ASF booth, and we
should have a room about Apache Camel & Karaf.

Regards
JB

On Mon, Oct 17, 2022 at 9:05 PM Aizhamal Nurmamat kyzy
 wrote:
>
> Hi Beam community!
>
> FOSDEM 2023  is back as an in person event! I have 
> heard only great things about the event where thousands of developers get 
> together to talk all about open source!
>
> Is anyone from the Beam community planning to attend? The event takes place 
> in Brussels on February 4 & 5, 2023. I believe it is also free to attend but 
> don't quote me on this.
>
> As an open source project we can also have
> - a stand for free https://fosdem.org/2023/news/2022-09-26-stands-cfp/
> - a Devroom https://fosdem.org/2023/news/2022-09-29-call_for_devrooms/
>
> Anyone interested?
>

Re: [VOTE] Release 2.42.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Fri, Oct 14, 2022 at 7:22 PM Chamikara Jayalath via dev
 wrote:
>
> +1 (binding)
>
> Thanks,
> Cham
>
> On Fri, Oct 14, 2022 at 5:43 AM Alexey Romanenko  
> wrote:
>>
>> +1 (binding)
>>
>> Tested with  https://github.com/Talend/beam-samples/
>> (Java SDK v8 & v11, Spark 3 runner).
>>
>> ---
>> Alexey
>>
>> On 14 Oct 2022, at 05:17, Ahmet Altay via dev  wrote:
>>
>> +1 (binding)
>>
>> Tested python quickstart examples on the direct runner. Thank you!
>>
>> On Thu, Oct 13, 2022 at 5:35 PM Robert Bradshaw via dev 
>>  wrote:
>>>
>>> +1 (binding)
>>>
>>> Validated release artifacts and signatures. Tested a Python pipeline
>>> on a clean install.
>>>
>>> On Thu, Oct 13, 2022 at 1:22 PM Ritesh Ghorse via dev
>>>  wrote:
>>> >
>>> > +1 (non-binding)
>>> > Validated Go SDK Quickstart on Direct and Dataflow runner.
>>> >
>>> > Thanks,
>>> > Ritesh Ghorse
>>> >
>>> > On Thu, Oct 13, 2022 at 4:01 PM Pablo Estrada via dev 
>>> >  wrote:
>>> >>
>>> >> +1 (binding)
>>> >>
>>> >> I've validated local/unit tests for existing dataflow templates. They 
>>> >> look good!
>>> >> Best
>>> >> -P.
>>> >>
>>> >> On Thu, Oct 13, 2022 at 10:41 AM Ning Kang via dev  
>>> >> wrote:
>>> >>>
>>> >>> +1 Thank you, Robert!
>>> >>>
>>> >>> On Thu, Oct 13, 2022 at 12:47 AM Robert Burke  
>>> >>> wrote:
>>> 
>>>  Hi everyone,
>>>  Please review and vote on the release candidate #2 for the version 
>>>  2.42.0, as follows:
>>>  [ ] +1, Approve the release
>>>  [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>>  Reviewers are encouraged to test their own use cases with the release 
>>>  candidate, and vote +1 if no issues are found.
>>> 
>>>  The complete staging area is available for your review, which includes:
>>>  * GitHub Release notes [1],
>>>  * the official Apache source release to be deployed to dist.apache.org 
>>>  [2], which is signed with the key with fingerprint 
>>>  A52F5C83BAE26160120EC25F3D56ACFBFB2975E1 [3],
>>>  * all artifacts to be deployed to the Maven Central Repository [4],
>>>  * source code tag "v2.42.0-RC2" [5],
>>>  * website pull request listing the release [6], the blog post [6], and 
>>>  publishing the API reference manual [7].
>>>  * Java artifacts were built with Gradle 7.5.1 and AdoptOpen JDK 
>>>  1.8.0_292.
>>>  * Python artifacts are deployed along with the source release to the 
>>>  dist.apache.org [2] and PyPI [8]
>>>  * Go Package information and SDK RC [9]
>>>  * Validation sheet with a tab for 2.42.0 release to help with 
>>>  validation [10].
>>>  * Docker images published to Docker Hub [11]. (Soon)
>>> 
>>>  The vote will be open for at least 72 hours. It is adopted by majority 
>>>  approval, with at least 3 PMC affirmative votes.
>>> 
>>>  Updates from RC1 include a fix to SpannerIO backlog estimation [12] 
>>>  and a fix to the BigQueryIO interpretation of coders on an internal 
>>>  flatten [13]. Otherwise, previous validation should be unaffected.
>>> 
>>>  For guidelines on how to try the release in your projects, check out 
>>>  our blog post at https://beam.apache.org/blog/validate-beam-release/.
>>> 
>>>  Thanks,
>>>  Robert Burke
>>>  2.42.0 Release Manager
>>> 
>>>  [1] https://github.com/apache/beam/milestone/4
>>>  [2] https://dist.apache.org/repos/dist/dev/beam/2.42.0/
>>>  [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>  [4] 
>>>  https://repository.apache.org/content/repositories/orgapachebeam-1286/
>>>  [5] https://github.com/apache/beam/tree/v2.42.0-RC2
>>>  [6] https://github.com/apache/beam/pull/23406
>>>  [7] https://github.com/apache/beam-site/pull/634
>>>  [8] https://pypi.org/project/apache-beam/2.42.0rc2/
>>>  [9] 
>>>  https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.42.0-RC2/go/pkg/beam
>>>  [10] 
>>>  https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=265602293
>>>  [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>>  [12] https://github.com/apache/beam/issues/23494
>>>  [13] https://github.com/apache/beam/issues/23561
>>> 
>>
>>

Re: [JmsIO] => Pull Request to fix message acknowledgement issue

[Jean-Baptiste Onofré]

Hi Vincent,

thanks, I will take a look (as original JmsIO author ;)).

Regards
JB

On Mon, Aug 29, 2022 at 6:43 PM BALLADA Vincent
 wrote:
>
> Hi all,
>
>
>
> Here is a PR related to the following issue (Runner acknowledges messages on 
> closed session):
>
> https://github.com/apache/beam/issues/20814
>
>
>
> And here is a documentation explaining the fix:
>
> https://docs.google.com/document/d/19HiNPoJeIlzCFyWGdlw7WEFutceL2AmN_5Z0Vmi1s-I/edit?usp=sharing
>
>
>
> And finally the PR:
>
> https://github.com/apache/beam/pull/22932
>
>
>
> Regards
>
>
>
> Vincent BALLADA
>
>
>
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une 
> correspondance privee et confidentielle a l'attention exclusive du 
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present 
> message ou une personne susceptible de pouvoir le lui delivrer, il vous est 
> signifie que toute divulgation, distribution ou copie de cette transmission 
> est strictement interdite. Si vous avez recu ce message par erreur, nous vous 
> remercions d'en informer l'expediteur par telephone ou de lui retourner le 
> present message, puis d'effacer immediatement ce message de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence intended 
> only for use of the individual or entity named above. If you are not the 
> intended recipient or the agent responsible for delivering the message to the 
> intended recipient, you are hereby notified that any disclosure, distribution 
> or copying of this communication is strictly prohibited. If you have received 
> this communication in error, please notify the sender by phone or by replying 
> this message, and then delete this message from your system.

Re: [VOTE] Release 2.39.0, candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Smoke quick tests with the direct runner.

Regards
JB

On Tue, May 17, 2022 at 5:05 PM Ahmet Altay  wrote:
>
> +1 (binding). I validated python quick starts on Direct runner and on 
> Dataflow.
>
> Thank you Yichi!
>
> On Mon, May 16, 2022 at 11:58 PM Yichi Zhang  wrote:
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version 2.39.0, 
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release 
>> candidate, and vote +1 if no issues are found.
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2], 
>> which is signed with the key with fingerprint 
>> 0DB9FF04B5D967903A28427021DE16602971EB7C [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.39.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and 
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK 1.8.0_332.
>> * Python artifacts are deployed along with the source release to the 
>> dist.apache.org [2] and PyPI[8].
>> * Validation sheet with a tab for 2.39.0 release to help with validation [9].
>> * Docker images published to Docker Hub [10].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority 
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our 
>> blog post at https://beam.apache.org/blog/validate-beam-release/.
>>
>> Thanks,
>> Yichi Zhang
>>
>> [1] 
>> https://issues.apache.org/jira/secure/ConfigureReleaseNote.jspa?projectId=12319527=12351170
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.39.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1271/
>> [5] https://github.com/apache/beam/tree/v2.39.0-RC1
>> [6] https://github.com/apache/beam/pull/17690
>> [7] https://github.com/apache/beam-site/pull/627
>> [8] https://pypi.org/project/apache-beam/2.39.0rc1/
>> [9] 
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=584711835
>> [10] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: [PROPOSAL] Stop Spark2 support in Spark Runner

[Jean-Baptiste Onofré]

+1 for me to drop Spark 2.x support.

Users who want to still use Spark 2.x can be back on a previous Beam release.

Regards
JB

On Thu, Mar 31, 2022 at 5:51 PM Alexey Romanenko
 wrote:
>
> Hi everyone,
>
> For the moment, Beam Spark Runner supports two versions of Spark - 2.x and 
> 3.x.
>
> Taking into account the several things that:
> - almost all cloud providers already mostly moved to Spark 3.x as a main 
> supported version;
> - the latest Spark 2.x release (Spark 2.4.8, maintenance release) was done 
> almost a year ago;
> - Spark 3 is considered as a mainstream Spark version for development and bug 
> fixing;
> - better to avoid the burden of maintenance (there are some incompatibilities 
> between Spark 2 and 3) of two versions;
>
> I’d suggest to stop support Spark 2 for the Spark Runner in the one of the 
> next Beam releases.
>
> What are your thoughts on this? Are there any principal objections or reasons 
> for not doing this that I probably missed?
>
> —
> Alexey
>
>

Re: [VOTE] Release 2.37.0, release candidate #3

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Tue, Mar 1, 2022 at 3:29 AM Brian Hulette  wrote:

> Hi everyone,
> Please review and vote on release candidate #3 for version 2.37.0, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if no issues are found.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint 1D190C85 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.37.0-RC3" [5],
> * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle 7.3.2 and OpenJDK 8u232.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI[8].
> * Validation sheet with a tab for 2.37.0 release to help with validation
> [9].
> * Docker images published to Docker Hub [10].
>
> For guidelines on how to try the release in your projects, check out our
> blog post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Brian
>
> [1]
> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=projectId=12319527=12351168
> [2] https://dist.apache.org/repos/dist/dev/beam/2.37.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1254/
> [5] https://github.com/apache/beam/tree/v2.37.0-RC3
> [6] https://github.com/apache/beam/pull/16887
> [7] https://github.com/apache/beam-site/pull/623
> [8] https://pypi.org/project/apache-beam/2.37.0rc3/
> [9]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1090588300
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image
>

Re: [VOTE] Release 2.37.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

On Wed, Feb 23, 2022 at 6:37 PM Brian Hulette  wrote:
>
> Hi everyone,
> Please review and vote on release candidate #2 version 2.37.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2], 
> which is signed with the key with fingerprint 1D190C85 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.37.0-RC2" [5],
> * website pull request listing the release [6], the blog post [6], and 
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle 7.3.2 and OpenJDK 8u232.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org [2] and PyPI[8].
> * Validation sheet with a tab for 2.37.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
>
> For guidelines on how to try the release in your projects, check out our blog 
> post at https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Brian
>
> [1] 
> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=projectId=12319527=12351168
> [2] https://dist.apache.org/repos/dist/dev/beam/2.37.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1253/
> [5] https://github.com/apache/beam/tree/v2.37.0-RC2
> [6] https://github.com/apache/beam/pull/16887
> [7] https://github.com/apache/beam-site/pull/623
> [8] https://pypi.org/project/apache-beam/2.37.0rc2/
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1090588300
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: [Proposal] => JMSIO dynamic topic publishing

[Jean-Baptiste Onofré]

All JMS related properties can be null (@Nullable).

Good point about breaking change.

The part on which I'm not big fan in your proposal is this
getDynamic() method. Maybe we can mimic what I did in JdbcIO with a fn
we can inject to define the destination.

But, ok, I would be happy to review a PR (I can comment on the PR though).

Regards
JB

On Thu, Feb 24, 2022 at 5:16 PM BALLADA Vincent
 wrote:
>
> Hi Jean Baptiste 
>
>
>
> Thank you for your feedback, it is interesting.
>
>
>
> In your proposal, the write transform would take a PCollection< JmsRecord>.
>
> JmsRecord has a lot of property that are related to the read operation 
> (jmsRedelivered, correlationId, etc…)
>
> As a result I think it won’t be easy to create it with all relevant 
> properties at late stage of the pipeline, before the write operation.
>
> It would introduce also a breaking change as the current implementation takes 
> a Pcollection as input.
>
> Plus the new standard for IO implementation seems to be parameterized 
> read/write (see annotated document).
>
>
>
> Thanks,
>
>
>
> Regards
>
>
>
> Vincent BALLADA
>
>
>
> De : Jean-Baptiste Onofré 
> Date : jeudi, 24 février 2022 à 16:32
> À : dev@beam.apache.org 
> Objet : Re: [Proposal] => JMSIO dynamic topic publishing
>
> [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments]
>
> Hi Vincent,
>
> It's Jean-Baptiste, not Jean-François, but it works as well ;)
>
> I got your point, however, I think we can achieve the same with
> JmsRecord. You can always use a DoFn at any part of your pipeline
> (after reading from kafka, redis, whatever), where you can create
> JmsRecord. If JmsRecord contains a property with destination, it can
> be populated dynamically JmsDestination.
>
> So basically, it means that we can add JmsDestinationType on the
> JmsRecord and use it in the sink.
>
> Something lie:
>
> pipeline
>  .apply(...) // returns PCollection // JmsReccord can be
> created/populated with destination and destination type
> .apply(JmsIO.write().withConnectionFactory(myConnectionFactory))
>
> I think it's less invasive.
>
> It's basically the same that you propose, but reusing JmsRecord and
> avoiding getDynamic() hook.
>
> Again, I'm not against with your proposal, but we can have something
> more concise.
>
> Just my €0.01 ;)
>
> Regards
> JB
>
> On Thu, Feb 24, 2022 at 3:42 PM BALLADA Vincent
>  wrote:
> >
> > Hi Jean François
> >
> >
> >
> > Please to hear of the author of JmsIO .
> >
> > Many thanks for your suggestion.
> >
> > JmsRecord is used at read time, in most use cases we will use a mapper to 
> > provide an object that will be used in several transform in the pipeline.
> >
> > The destination won’t necessary be included in the read JmsRecord, as for 
> > instance in many pipelines we do data enrichment (form redis, database, 
> > etc…) and the dynamic part of the topic would be extracted at any stage of 
> > the pipeline.
> >
> > So for me it would be more flexible to not be tied with JmsRecord.
> >
> >
> >
> > Regards
> >
> >
> >
> > Vincent BALLADA
> >
> >
> >
> > De : Jean-Baptiste Onofré 
> > Date : samedi, 19 février 2022 à 06:40
> > À : dev@beam.apache.org 
> > Objet : Re: [Proposal] => JMSIO dynamic topic publishing
> >
> > [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments]
> >
> > Hi Vincent,
> >
> > It looks interesting. Another possible approach is to have some
> > implicit (instead of being explicit) but defining the destination on
> > the JmsRecord. If the JmsRecord contains the destination (that could
> > be "dynamic"), we use it, overriding the destination provided on the
> > IO configuration.
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > NB: I'm the original author of JmsIO ;)
> >
> > On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent
> >  wrote:
> > >
> > > Hi all
> > >
> > >
> > >
> > > Here is a proposal to implement the ability to publish on dynamic topics 
> > > with JMSIO:
> > >
> > > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing
> > >
> > >
> > >
> > > There is also a JIRA issue:
> > >
> > > https://issues.apache.org/jira/browse/BEAM-13608
> > >
> > >
> > >
> > > Best regards
> > >
> > >
> > >
&g

Re: [Proposal] => JMSIO dynamic topic publishing

[Jean-Baptiste Onofré]

Hi Vincent,

It's Jean-Baptiste, not Jean-François, but it works as well ;)

I got your point, however, I think we can achieve the same with
JmsRecord. You can always use a DoFn at any part of your pipeline
(after reading from kafka, redis, whatever), where you can create
JmsRecord. If JmsRecord contains a property with destination, it can
be populated dynamically JmsDestination.

So basically, it means that we can add JmsDestinationType on the
JmsRecord and use it in the sink.

Something lie:

pipeline
 .apply(...) // returns PCollection // JmsReccord can be
created/populated with destination and destination type
.apply(JmsIO.write().withConnectionFactory(myConnectionFactory))

I think it's less invasive.

It's basically the same that you propose, but reusing JmsRecord and
avoiding getDynamic() hook.

Again, I'm not against with your proposal, but we can have something
more concise.

Just my €0.01 ;)

Regards
JB

On Thu, Feb 24, 2022 at 3:42 PM BALLADA Vincent
 wrote:
>
> Hi Jean François
>
>
>
> Please to hear of the author of JmsIO .
>
> Many thanks for your suggestion.
>
> JmsRecord is used at read time, in most use cases we will use a mapper to 
> provide an object that will be used in several transform in the pipeline.
>
> The destination won’t necessary be included in the read JmsRecord, as for 
> instance in many pipelines we do data enrichment (form redis, database, etc…) 
> and the dynamic part of the topic would be extracted at any stage of the 
> pipeline.
>
> So for me it would be more flexible to not be tied with JmsRecord.
>
>
>
> Regards
>
>
>
> Vincent BALLADA
>
>
>
> De : Jean-Baptiste Onofré 
> Date : samedi, 19 février 2022 à 06:40
> À : dev@beam.apache.org 
> Objet : Re: [Proposal] => JMSIO dynamic topic publishing
>
> [EXTERNAL EMAIL : Be CAUTIOUS, especially with links and attachments]
>
> Hi Vincent,
>
> It looks interesting. Another possible approach is to have some
> implicit (instead of being explicit) but defining the destination on
> the JmsRecord. If the JmsRecord contains the destination (that could
> be "dynamic"), we use it, overriding the destination provided on the
> IO configuration.
> Thoughts ?
>
> Regards
> JB
>
> NB: I'm the original author of JmsIO ;)
>
> On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent
>  wrote:
> >
> > Hi all
> >
> >
> >
> > Here is a proposal to implement the ability to publish on dynamic topics 
> > with JMSIO:
> >
> > https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing
> >
> >
> >
> > There is also a JIRA issue:
> >
> > https://issues.apache.org/jira/browse/BEAM-13608
> >
> >
> >
> > Best regards
> >
> >
> >
> > Vincent BALLADA
> >
> >
> > Confidential C
> >
> > -- Disclaimer 
> > Ce message ainsi que les eventuelles pieces jointes constituent une 
> > correspondance privee et confidentielle a l'attention exclusive du 
> > destinataire designe ci-dessus. Si vous n'etes pas le destinataire du 
> > present message ou une personne susceptible de pouvoir le lui delivrer, il 
> > vous est signifie que toute divulgation, distribution ou copie de cette 
> > transmission est strictement interdite. Si vous avez recu ce message par 
> > erreur, nous vous remercions d'en informer l'expediteur par telephone ou de 
> > lui retourner le present message, puis d'effacer immediatement ce message 
> > de votre systeme.
> >
> > *** This e-mail and any attachments is a confidential correspondence 
> > intended only for use of the individual or entity named above. If you are 
> > not the intended recipient or the agent responsible for delivering the 
> > message to the intended recipient, you are hereby notified that any 
> > disclosure, distribution or copying of this communication is strictly 
> > prohibited. If you have received this communication in error, please notify 
> > the sender by phone or by replying this message, and then delete this 
> > message from your system.
>
> 
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une 
> correspondance privee et confidentielle a l'attention exclusive du 
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present 
> message ou une personne susceptible de pouvoir le lui delivrer, il vous est 
> signifie que toute divulgation, distribution ou copie de cette transmission 
> est strictement interdite. Si vous avez recu ce message par erreur, n

Re: [Proposal] => JMSIO dynamic topic publishing

[Jean-Baptiste Onofré]

Hi Vincent,

It looks interesting. Another possible approach is to have some
implicit (instead of being explicit) but defining the destination on
the JmsRecord. If the JmsRecord contains the destination (that could
be "dynamic"), we use it, overriding the destination provided on the
IO configuration.
Thoughts ?

Regards
JB

NB: I'm the original author of JmsIO ;)

On Fri, Feb 18, 2022 at 7:00 PM BALLADA Vincent
 wrote:
>
> Hi all
>
>
>
> Here is a proposal to implement the ability to publish on dynamic topics with 
> JMSIO:
>
> https://docs.google.com/document/d/1IY4_e5g1g71XvTLL4slHRyVfX7ByiwjD_de3WGsBQXg/edit?usp=sharing
>
>
>
> There is also a JIRA issue:
>
> https://issues.apache.org/jira/browse/BEAM-13608
>
>
>
> Best regards
>
>
>
> Vincent BALLADA
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une 
> correspondance privee et confidentielle a l'attention exclusive du 
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present 
> message ou une personne susceptible de pouvoir le lui delivrer, il vous est 
> signifie que toute divulgation, distribution ou copie de cette transmission 
> est strictement interdite. Si vous avez recu ce message par erreur, nous vous 
> remercions d'en informer l'expediteur par telephone ou de lui retourner le 
> present message, puis d'effacer immediatement ce message de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence intended 
> only for use of the individual or entity named above. If you are not the 
> intended recipient or the agent responsible for delivering the message to the 
> intended recipient, you are hereby notified that any disclosure, distribution 
> or copying of this communication is strictly prohibited. If you have received 
> this communication in error, please notify the sender by phone or by replying 
> this message, and then delete this message from your system.

Re: [DISCUSS] Migrate Jira to GitHub Issues?

[Jean-Baptiste Onofré]

Hi,

I don't have concerns: if Beam is OK with the issue single milestone use,
that's fine with me ;)

Thanks for the detailed document, it helps!

Regards
JB

On Tue, Feb 15, 2022 at 6:52 AM Aizhamal Nurmamat kyzy 
wrote:

> Very humbly, I think the benefits of moving to GitHub Issues outweigh the
> shortcomings.
>
> Jan, Kenn, Alexey, JB: adding you directly as you had some concerns.
> Please, let us know if they were addressed by the options that we described
> in the doc [1]?
>
> If noone objects, I can start working with some of you on Migration TODOs
> outlined in the doc I am referencing.
>
>
> [1]
> https://docs.google.com/document/d/1_n7gboVbSKPs-CVcHzADgg8qpNL9igiHqUPCmiOslf0/edit#bookmark=id.izn35w5gsjft
>
> On Thu, Feb 10, 2022 at 1:12 PM Danny McCormick 
> wrote:
>
>> I'm definitely +1 on moving to help make the bar for entry lower for new
>> contributors (like myself!)
>>
>> Thanks,
>> Danny
>>
>> On Thu, Feb 10, 2022 at 2:32 PM Aizhamal Nurmamat kyzy <
>> aizha...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> I think we've had a chance to discuss shortcomings and advantages. I
>>> think each person may have a different bias / preference. My bias is to
>>> move to Github, to have a more inclusive, approachable project despite the
>>> differences in workflow. So I'm +1 on moving.
>>>
>>> Could others share their bias? Don't think of this as a vote, but I'd
>>> like to get a sense of people's preferences, to see if there's a
>>> strong/slight feeling either way.
>>>
>>> Again, the sticky points are summarized here [1], feel free to add to
>>> the doc.
>>>
>>> [1]
>>> https://docs.google.com/document/d/1_n7gboVbSKPs-CVcHzADgg8qpNL9igiHqUPCmiOslf0/edit#
>>>
>>>
>>> On Mon, Jan 31, 2022 at 7:23 PM Aizhamal Nurmamat kyzy <
>>> aizha...@apache.org> wrote:
>>>
 Welcome to the Beam community, Danny!

 We would love your help if/when we end up migrating.

 Please add your comments to the doc I shared[1], in case we missed some
 cool GH features that could be helpful. Thanks!

 [1]
 https://docs.google.com/document/d/1_n7gboVbSKPs-CVcHzADgg8qpNL9igiHqUPCmiOslf0/edit#

 On Mon, Jan 31, 2022, 10:06 AM Danny McCormick <
 dannymccorm...@google.com> wrote:

> > Then (this is something you'd have to code) you could easily write
> or use an existing GithubAction or bot that will assign the labels based 
> on
> the initial selection done by the user at entry. We have not done it yet
> but we might.
>
> Hey, new contributor here - wanted to chime in with a shameless plug
> because I happen to have written an action that does pretty much exactly
> what you're describing[1] and could be extensible to the use case 
> discussed
> here - it should basically just require writing some config (example in
> action[2]). In general, automated management of labels based on the 
> initial
> issue description + content isn't too hard, it does get significantly
> trickier (but definitely still possible) if you try to automate labels
> based on responses or edits.
>
> Also, big +1 that the easy integration with Actions is a significant
> advantage of using issues since it helps keep your automations in one 
> place
> (or at least fewer places) and gives you a lot of tools out of the box 
> both
> from the community and from the Actions org. *Disclaimer:* I am
> definitely biased. Until 3 weeks ago I was working on the Actions team at
> GitHub.
>
> I'd be happy to help with some of the issue automation if we decide
> that would be helpful, whether that's reusing existing work or tailoring 
> it
> more exactly to the Beam use case.
>
> [1] https://github.com/damccorm/tag-ur-it
> [2] https://github.com/microsoft/azure-pipelines-tasks/issues/15839
>
> Thanks,
> Danny
>
> On Mon, Jan 31, 2022 at 12:49 PM Zachary Houfek 
> wrote:
>
>> > You can link PR to the issue by just mentioning #Issue in the
>> commit message. If you do not prefix it with "Closes:" "Fixes:" or 
>> similar
>> it will be just linked
>>
>> Ok, thanks for the clarification there.
>>
>> Regards,
>> Zach
>>
>> On Mon, Jan 31, 2022 at 12:43 PM Cristian Constantinescu <
>> zei...@gmail.com> wrote:
>>
>>> I've been semi-following this thread, apologies if this has been
>>> raised already.
>>>
>>> From a user point of view, in some corporate environments (that I've
>>> worked at), Github is blocked. That includes the issues part. The Apache
>>> Jira is not blocked and does at times provide value while investigating
>>> issues.
>>>
>>> Obviously, users stuck in those unfortunate circonstances can just
>>> use their personal device. Not advocating any direction on the matter, 
>>> just
>>> putting this out there.
>>>
>>> On Mon, Jan 31, 2022 at 12:21 PM Zachary 

Re: JdbcIO for writing to Dynamic Schemas in Postgres

[Jean-Baptiste Onofré]

Did you create a jira about that already ?I will do the improvement on JdbcIO. Regards JBThanksRegards JBLe dim. 31 mai 2020 ? 11:25, Willem Pienaar  a ?crit :Hi Reuven,To be clear, we already have this solved for BigQueryIO. I am hoping there is a similar solution for JdbcIO.Regards,WillemOn Sun, May 31, 2020, at 12:42 PM, Reuven Lax wrote:This should be possible using the Beam programmatic API. You can pass BigQueryIO a function that determines the BigQuery table based on the input element.On Sat, May 30, 2020 at 9:20 PM Willem Pienaar  wrote:Hi JB,  Apologies for resurrecting this thread, but I have a related question.  We've built a feature store Feast (https://github.com/feast-dev/feast) primarily on Beam. We have been very happy with our decision to use Beam thus far. Beam is mostly used as the ingestion layer that writes data into stores (BigQuery, Redis). I am currently implementing JdbcIO (for PostgreSQL) and it's working fine so far. I set up all the tables when the job is launched, and I write into different tables depending on the input elements.  However, a problem we are facing is that schema changes are happening very rapidly based on our users' activity. Every time the user changes a collection of features/fields, we have to launch a new Dataflow job in order to support the new database schema. This can take 3-4 minutes. Every time the jobs are in an updating state we have to block all user activity, which is quite disruptive.  What we want to do is dynamically configure the SQL insert statement based on the input elements. This would allow us to keep the same job running indefinitely, dramatically improving the user experience. We have found solutions for BigQueryIO and our other IO, but not yet for JdbcIO. As far as I can tell it isn't possible to modify the SQL insert statement to write to a new table or to the same table with new columns, without restarting the job.  Do you have any suggestions one how we can achieve the above? If it can't be done with the current implementation, would it be reasonable to contribute this functionality back to Beam?  Regards, Willem  On Tue, Mar 3, 2020, at 1:30 AM, Jean-Baptiste Onofre wrote: > Hi >  > You have the setPrepareStatement() method where you define the target tables. > However, it?s in the same database (datasource) per pipeline. >  > You can define several datasources and use a different datasource in  > each JdbcIO write. Meaning that you can divide in sub pipelines. >  > Regards > JB >  > > Le 29 f?vr. 2020 ? 17:52, Vasu Gupta  a ?crit : > >  > > Hey folks, > >  > > Can we use JdbcIO for writing data to multiple Schemas(For Postgres Database) dynamically using Apache beam Java Framework? Currently, I can't find any property that I could set to JdbcIO transform for providing schema or maybe I am missing something. > >  > > Thanks >  >

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288

[Jean-Baptiste Onofré]

+1 (binding)RegardsJBLe mar. 3 mars 2020 ? 19:31, Luke Cwik  a ?crit :Please review the release of the following artifacts that we vendor: * beam-vendor-grpc-1_26_0Hi everyone,Please review and vote on the release candidate #1 for the version 0.3, as follows:[ ] +1, Approve the release[ ] -1, Do not approve the release (please provide specific comments)The complete staging area is available for your review, which includes:* the official Apache source release to be deployed to dist.apache.org [1], which is signed with the key with fingerprint EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],* all artifacts to be deployed to the Maven Central Repository [3],* commit hash "7df0d6cda0b1e1d5d360ea68c9ee01ca35acc230" [4],The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.Thanks,Release Manager[1] https://dist.apache.org/repos/dist/dev/beam/vendor/[2] https://dist.apache.org/repos/dist/release/beam/KEYS[3] https://repository.apache.org/content/repositories/orgapachebeam-1096/[4] https://github.com/apache/beam/commit/7df0d6cda0b1e1d5d360ea68c9ee01ca35acc230

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8

[Jean-Baptiste Onofré]

+1Regards JBLe mar. 25 f?vr. 2020 ? 13:43, Isma?l Mej?a  a ?crit :Please review the release of the following artifacts that we vendor: * beam-vendor-bytebuddy-1_10_8Hi everyone,Please review and vote on the release candidate #1 for the version 0.1, as follows:[ ] +1, Approve the release[ ] -1, Do not approve the release (please provide specific comments)The complete staging area is available for your review, which includes:* the official Apache source release to be deployed to dist.apache.org [1], which is signed with the key with fingerprint 3415631729E15B33051ADB670A9DAF6713B86349 [2],* all artifacts to be deployed to the Maven Central Repository [3],* commit hash "0756edfba7c682b1e75f4691c0ecb38a1dc9d16a" [4],The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.Thanks,Release Manager[1] https://dist.apache.org/repos/dist/dev/beam/vendor/[2] https://dist.apache.org/repos/dist/release/beam/KEYS[3] https://repository.apache.org/content/repositories/orgapachebeam-1093/[4] https://github.com/apache/beam/commit/0756edfba7c682b1e75f4691c0ecb38a1dc9d16a

Re: [VOTE] Upgrade gradle to 6.2

[Jean-Baptiste Onofré]

Hi AlexI also have couple of contacts at Gradle. Let me know if needed. Regards JBLe mar. 25 f?vr. 2020 ? 08:20, Alex Van Boxel  a ?crit :OK, great. I know someone that works at gradle, so I can ping them when I have some problems.Any other know pitfalls I can expect, let me know :-) _/_/ Alex Van BoxelOn Tue, Feb 25, 2020 at 7:20 AM Jean-Baptiste Onofr?  wrote:+1It makes sense. Thanks. Regards JBLe lun. 24 f?vr. 2020 ? 22:37, Alex Van Boxel  a ?crit :Anyone objections that I upgrade gradle to 6.2. If ok this will be done over several commits where I will:Upgrade pluginsUpgrade gradle to 6.2See where we can use some of the new features _/_/ Alex Van Boxel

Re: [VOTE] Upgrade gradle to 6.2

[Jean-Baptiste Onofré]

+1It makes sense. Thanks. Regards JBLe lun. 24 f?vr. 2020 ? 22:37, Alex Van Boxel  a ?crit :Anyone objections that I upgrade gradle to 6.2. If ok this will be done over several commits where I will:Upgrade pluginsUpgrade gradle to 6.2See where we can use some of the new features _/_/ Alex Van Boxel

Re: [PROPOSAL] Vendored bytebuddy dependency release

[Jean-Baptiste Onofré]

+1RegardsJBLe sam. 22 f?vr. 2020 ? 15:02, Isma?l Mej?a  a ?crit :The version of bytebuddy Beam is vendoring (1.9.3) is more than 16 months old. Iwould like to propose that we upgrade it [1] to the most recent version (1.10.8)[2] so we can benefit of the latest improvements for Java 11 and upgraded ASM.If everyone agrees I would like to volunteer as the release manager for thisupgrade.[1] https://issues.apache.org/jira/browse/BEAM-9342[2] https://github.com/raphw/byte-buddy/blob/master/release-notes.md

Re: Labels on PR

[Jean-Baptiste Onofré]

ThanksIt sounds good to me. Regards JBLe lun. 10 f?vr. 2020 ? 08:35, Alex Van Boxel  a ?crit :I've started putting labels on PR's. I've done the first page for now (as I'm afraid putting them on older once could affect the stale bot. I hope this is ok.For now I'm only focussing on language and I'm going to see if I can write a GitLab action for it. I hope this is useful. Other kind of suggestions for labels, that can be automated, are welcome. _/_/ Alex Van Boxel

Re: Compile error on Java 11 when running :examples:java:test

[Jean-Baptiste Onofré]

HiNo jdk 11 is not yet fully supported.I?ve started to work on it but it?s not yet ready.RegardsJBLe ven. 7 f?vr. 2020 ? 20:20, David Cavazos  a ?crit :Hi Beamers,I'm trying to run the tests for the Java examples using Java 11 and there is a compilation error due to an incompatible version.I'm using the latest version of master.If I downgrade to Java 8, it works. But isn't Java 11 supported?Thanks!

Re: Release 2.19.0, release candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Checked quickly on beam-samples.

Thanks,
Regards
JB

On 30/01/2020 06:02, Boyuan Zhang wrote:
> Hi everyone,
> Please review and vote on the release candidate #1 for the version
> 2.19.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org/> [2], which is signed with the key with
> fingerprint D6BF C115 EAA6 9828 5A89  3165 99DF 385D A877 C596[3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.19.0-RC1" [5],
> * website pull request listing the release [6], publishing the API
> reference manual [7], and the blog post [8].
> * Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle
> JDK JDK_VERSION.
> * Python artifacts are deployed along with the source release to
> the dist.apache.org <http://dist.apache.org/> [2].
> * Validation sheet with a tab for 2.18.0 release to help with validation
> [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Release Manager
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12346582
> [2] https://dist.apache.org/repos/dist/dev/beam/2.19.0/
> <https://dist.apache.org/repos/dist/dev/beam/2.19.0/>
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> <https://dist.apache.org/repos/dist/release/beam/KEYS>
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1091/
> [5] https://github.com/apache/beam/tree/v2.19.0-RC1
> <https://github.com/apache/beam/tree/v2.19.0-RC1>
> [6] https://github.com/apache/beam/pull/10713
> [7] https://github.com/apache/beam-site/pull/597
> [8] https://github.com/apache/beam/pull/10722
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1148160512
> [10] https://hub.docker.com/u/apachebeam

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: New contributor

[Jean-Baptiste Onofré]

Welcome aboard.

Regards
JB

On 28/01/2020 14:14, Ludovic Boutros wrote:
> Thanks Ismaël !
> 
> 
> 
> Le mar. 28 janv. 2020 à 12:08, Ismaël Mejía  <mailto:ieme...@gmail.com>> a écrit :
> 
> You were added to JIRA. Enjoy!
> 
> On Tue, Jan 28, 2020 at 12:04 PM Ludovic Boutros  <mailto:boutr...@gmail.com>> wrote:
> 
> Dear all,
> 
> please, could you add me as a contributor in Jira ? My user name
> is lboutros.
> 
> Thank you,
> 
> Ludovic.
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.18.0, release candidate #1

[Jean-Baptiste Onofré]

>>>>>>
> >>> >>>>>>
> >>> >>>>>> I believe we can drop MAVEN_VERSION now that it is no
> longer used. I do not think it is needed to add a Gradle version
> either because the version itself is part of the repo anyway.
> >>> >>>>>>
> >>> >>>>>> I do not know if java, python etc. versions are
> helpful. Maybe others can comment. I would prefer to reduce the
> load on the release manager and drop this if this is not
> particularly important.
> >>> >>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On Mon, Jan 13, 2020 at 7:37 PM Valentyn Tymofieiev
> mailto:valen...@google.com>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>> There are some issues in this message, part of the
> message is still a template (1.2.3, TODO, MAVEN_VERSION).
> >>> >>>>>>>> Before I noticed these issues, I ran a few Batch
> and Streaming Python 3.7 pipelines using Direct and Dataflow
> runners, and they all succeeded.
> >>> >>>>>>>>
> >>> >>>>>>>> On Mon, Jan 13, 2020 at 4:09 PM Udi Meiri
> mailto:eh...@google.com>> wrote:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Hi everyone,
> >>> >>>>>>>>> Please review and vote on the release candidate #3
> for the version 1.2.3, as follows:
> >>> >>>>>>>>> [ ] +1, Approve the release
> >>> >>>>>>>>> [ ] -1, Do not approve the release (please provide
> specific comments)
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> The complete staging area is available for your
> review, which includes:
> >>> >>>>>>>>> * JIRA release notes [1],
> >>> >>>>>>>>> * the official Apache source release to be
> deployed to dist.apache.org <http://dist.apache.org> [2], which
> is signed with the key with fingerprint 8961 F3EF 8E79 6688
> 4067  87CF 587B 049C 36DA AFE6 [3],
> >>> >>>>>>>>> * all artifacts to be deployed to the Maven
> Central Repository [4],
> >>> >>>>>>>>> * source code tag "v1.2.3-RC3" [5],
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> Tag is "v2.18.0-RC1". This is correct in the
> referenced link.
> >>> >>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> * website pull request listing the release [6],
> publishing the API reference manual [7], and the blog post [8].
> >>> >>>>>>>>> * Java artifacts were built with Maven
> MAVEN_VERSION and OpenJDK/Oracle JDK JDK_VERSION.
> >>> >>>>>>>>> TODO: do these versions matter, and are they
> stamped into the artifacts?
> >>> >>>>>>>>> * Python artifacts are deployed along with the
> source release to the dist.apache.org <http://dist.apache.org> [2].
> >>> >>>>>>>>> * Validation sheet with a tab for 2.18.0 release
> to help with validation [9].
> >>> >>>>>>>>> * Docker images published to Docker Hub [10].
> >>> >>>>>>>>>
> >>> >>>>>>>>> The vote will be open for at least 72 hours. It is
> adopted by majority approval, with at least 3 PMC affirmative votes.
> >>> >>>>>>>>>
> >>> >>>>>>>>> Thanks,
> >>> >>>>>>>>> Release Manager
> >>> >>>>>>>>>
> >>> >>>>>>>>> [1]
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346383=12319527
> >>> >>>>>>>>> [2]
> https://dist.apache.org/repos/dist/dev/beam/2.18.0/
> >>> >>>>>>>>> [3]
> https://dist.apache.org/repos/dist/release/beam/KEYS
> >>> >>>>>>>>> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1090/
> >>> >>>>>>>>> [5] https://github.com/apache/beam/tree/v2.18.0-RC1
> >>> >>>>>>>>> [6] https://github.com/apache/beam/pull/10574
> >>> >>>>>>>>> [7] https://github.com/apache/beam-site/pull/595
> >>> >>>>>>>>> [8] https://github.com/apache/beam/pull/10575
> >>> >>>>>>>>> [9]
> 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1178617819
> >>> >>>>>>>>> [10] https://hub.docker.com/u/apachebeam
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Contributor permissions

[Jean-Baptiste Onofré]

Hi,

Do you have already Jira in target ?

Regards
JB

On 21/12/2019 13:43, Shashanka Balakuntala wrote:
> Hello all,
> 
> My name is Shashanka Balakuntala Srinivasa,I would like to contribute to
> this project so can I be added as contributor in JIRA.My username is
> "balaShashanka"
> 
> Thanks in advance.
> 
> _Regards_
>   Shashanka Balakuntala Srinivasa
>       

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: RabbitMqIO issues and open PRs

[Jean-Baptiste Onofré]

Hi,

I just provided feedback in the PRs.

Let me know if you want to chat about some initial implementation (as
I'm the original author of the IO, I remember some discussion in the
past ;) ).

Regards
JB

On 31/10/2019 21:38, Daniel Robert wrote:
> I'm pretty new to the Beam ecosystem, so apologies if this is not the
> right forum for this.
> 
> My team has been learning and starting to use Beam for the past few
> months and have run into myriad problems with the RabbitIO connector for
> java, aspects of which seem perhaps fundamentally broken or incorrect in
> the released implementation. I've tracked our significant issues down
> and opened tickets and PRs for them. I'm not certain what the typical
> response time is, but given the severity of the issues (as I perceive
> them), I'd like to escalate some of these PRs and try to get some fixes
> into the next Beam release.
> 
> I originally opened BEAM-8390 (https://github.com/apache/beam/pull/9782)
> as it was impossible to set the 'useCorrelationId' property (implying
> this functionality was also untested). Since then, I've found and PR'd
> the following, which are awaiting feedback/approval:
> 
> 1. Watermarks not advancing
> 
> Ticket/PR: BEAM-8347 - https://github.com/apache/beam/pull/9820
> 
> Impact: under low message volumes, the watermark never advances and
> windows can never 'on time' fire.
> 
> Notes: The RabbitMq UnboundedSource uses 'oldest known time' as a
> watermark when other similar sources (and documentation on watermarking)
> state for monotonically increasing timestamps (the case with a queue) it
> should be the most recent time. I have a few open questions about
> testing and implementation details in the PR but it should work as-is.
> 
> 2. Exchanges are always declared, which fail if a pre-existing exchange
> differs
> 
> Ticket/PR: BEAM-8513 - https://github.com/apache/beam/pull/9937
> 
> Impact: It is impossible to utilize an existing, durable exchange.
> 
> Notes: I'm hooking Beam up to an existing topic exchange (an 'event
> bus') that is 'durable'. RabbitMqIO current implementation will always
> attempt to declare the exchange, and does so as non-durable, which
> causes rabbit to fail the declaration. (Interestingly qpid does not fail
> in this scenario.) The PR allows the caller to disable declaring the
> exchange, similar to `withQueueDeclare` for declaring a queue.
> 
> This PR also calls out a lot of the documentation that seems misleading;
> implying that one either interacts with queues *or* exchanges when that
> is not how AMQP fundamentally operates. The implementation of the
> RabbitMqIO connector before this PR seems like it probably works with
> the default exchange and maybe a fanout exchange, but not a topic exchange.
> 
> 3. Library versions
> 
> Tickets/PR: BEAM-7434, BEAM-5895, and BEAM-5894 -
> https://github.com/apache/beam/pull/9900
> 
> Impact: The rabbitmq amqp client for java released the 5.x line in
> September of 2017. Some automated tickets were open to upgrade, plus a
> manual ticket to drop the use of the deprecated QueueConsumer API.
> 
> Notes: The upgrade was relatively simple, but I implemented it using a
> pull-based API rather than push-based (Consumer) which may warrant some
> discussion. I'm used to discussing this type of thing over PRs but I'm
> happy to do whatever the community prefers.
> 
> ---
> 
> Numbers 1 and 2 above are 'dealbreaker' issues for my team. They
> effectively make rabbitmq unusable as an unbounded source, forcing
> developers to fork and modify the code. Number 3 is much less
> significant and I've put it here more for 'good, clean living' than an
> urgent issue.
> 
> Aside from the open issues, given the low response rate so far, I'd be
> more than happy to take on a more active role in maintaining/reviewing
> the rabbitmq io for java. For now, however, is this list the best way to
> 'bump' these open issues and move forward? Further, is the general
> approach before opening a PR to ask some preliminary questions in this
> email list?
> 
> Thank you,
> -Daniel Robert
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: RabbitMqIO issues and open PRs

[Jean-Baptiste Onofré]

Hi Daniel,

I will review and work on this IO.

Sorry for the delay, I wasn't super active on Beam recently, but now back ;)

Regards
JB

On 31/10/2019 21:38, Daniel Robert wrote:
> I'm pretty new to the Beam ecosystem, so apologies if this is not the
> right forum for this.
> 
> My team has been learning and starting to use Beam for the past few
> months and have run into myriad problems with the RabbitIO connector for
> java, aspects of which seem perhaps fundamentally broken or incorrect in
> the released implementation. I've tracked our significant issues down
> and opened tickets and PRs for them. I'm not certain what the typical
> response time is, but given the severity of the issues (as I perceive
> them), I'd like to escalate some of these PRs and try to get some fixes
> into the next Beam release.
> 
> I originally opened BEAM-8390 (https://github.com/apache/beam/pull/9782)
> as it was impossible to set the 'useCorrelationId' property (implying
> this functionality was also untested). Since then, I've found and PR'd
> the following, which are awaiting feedback/approval:
> 
> 1. Watermarks not advancing
> 
> Ticket/PR: BEAM-8347 - https://github.com/apache/beam/pull/9820
> 
> Impact: under low message volumes, the watermark never advances and
> windows can never 'on time' fire.
> 
> Notes: The RabbitMq UnboundedSource uses 'oldest known time' as a
> watermark when other similar sources (and documentation on watermarking)
> state for monotonically increasing timestamps (the case with a queue) it
> should be the most recent time. I have a few open questions about
> testing and implementation details in the PR but it should work as-is.
> 
> 2. Exchanges are always declared, which fail if a pre-existing exchange
> differs
> 
> Ticket/PR: BEAM-8513 - https://github.com/apache/beam/pull/9937
> 
> Impact: It is impossible to utilize an existing, durable exchange.
> 
> Notes: I'm hooking Beam up to an existing topic exchange (an 'event
> bus') that is 'durable'. RabbitMqIO current implementation will always
> attempt to declare the exchange, and does so as non-durable, which
> causes rabbit to fail the declaration. (Interestingly qpid does not fail
> in this scenario.) The PR allows the caller to disable declaring the
> exchange, similar to `withQueueDeclare` for declaring a queue.
> 
> This PR also calls out a lot of the documentation that seems misleading;
> implying that one either interacts with queues *or* exchanges when that
> is not how AMQP fundamentally operates. The implementation of the
> RabbitMqIO connector before this PR seems like it probably works with
> the default exchange and maybe a fanout exchange, but not a topic exchange.
> 
> 3. Library versions
> 
> Tickets/PR: BEAM-7434, BEAM-5895, and BEAM-5894 -
> https://github.com/apache/beam/pull/9900
> 
> Impact: The rabbitmq amqp client for java released the 5.x line in
> September of 2017. Some automated tickets were open to upgrade, plus a
> manual ticket to drop the use of the deprecated QueueConsumer API.
> 
> Notes: The upgrade was relatively simple, but I implemented it using a
> pull-based API rather than push-based (Consumer) which may warrant some
> discussion. I'm used to discussing this type of thing over PRs but I'm
> happy to do whatever the community prefers.
> 
> ---
> 
> Numbers 1 and 2 above are 'dealbreaker' issues for my team. They
> effectively make rabbitmq unusable as an unbounded source, forcing
> developers to fork and modify the code. Number 3 is much less
> significant and I've put it here more for 'good, clean living' than an
> urgent issue.
> 
> Aside from the open issues, given the low response rate so far, I'd be
> more than happy to take on a more active role in maintaining/reviewing
> the rabbitmq io for java. For now, however, is this list the best way to
> 'bump' these open issues and move forward? Further, is the general
> approach before opening a PR to ask some preliminary questions in this
> email list?
> 
> Thank you,
> -Daniel Robert
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [spark structured streaming runner] merge to master?

[Jean-Baptiste Onofré]

I agree, it would be the easiest way and allow users to switch easily as 
well using a single artifact.


Regards
JB

On 29/10/2019 23:54, Kenneth Knowles wrote:
Is it just as easy to have two jars and build an uber jar with both 
included? Then the runner can still be toggled with a flag.


Kenn

On Tue, Oct 29, 2019 at 9:38 AM Alexey Romanenko 
mailto:aromanenko@gmail.com>> wrote:


Hmm, I don’t think that jar size should play a big role comparing
to the whole size of shaded jar of users job. Even more, I think
it will be quite confusing for users to choose which jar to use if
we will have 3 different ones for similar purposes. Though, let’s
see what others think.


On 29 Oct 2019, at 15:32, Etienne Chauchot mailto:echauc...@apache.org>> wrote:

Hi Alexey,

Thanks for your opinion !

Comments inline

Etienne

On 28/10/2019 17:34, Alexey Romanenko wrote:

Let me share some of my thoughts on this.


    - shall we filter out the package name from the release?


Until new runner is not ready to be used in production (or, at
least, be used for beta testing but users should be clearly
warned about that in this case), I believe we need to filter out
its classes from published jar to avoid a confusion.

Yes that is what I think also


    - should we release 2 jars: one for the old and one for
the new ?

    - should we release 3 jars: one for the new, one for the
new and one for both ?


Once new runner will be released, then I think we need to
provide only one single jar and allow user to switch between
different Spark runners with CLI option.


I would vote for 3 jars: one for new, one for old, and one for
both. Indeed, in some cases, users are looking very closely at
the size of jars. This solution meets all use cases


    - should we create a special entry to the capability matrix ?


Sure, since it has its own uniq characteristics and
implementation, but again, only once new runner will be
"officially released".

+1




On 28 Oct 2019, at 10:27, Etienne Chauchot
mailto:echauc...@apache.org>> wrote:

Hi guys,

Any opinions on the point2 communication to users ?

Etienne

On 24/10/2019 15:44, Etienne Chauchot wrote:


Hi guys,

I'm glad to announce that the PR for the merge to master of
the new runner based on Spark Structured Streaming framework
is submitted:

https://github.com/apache/beam/pull/9866


1. Regarding the status of the runner:

-the runner passes 93% of the validates runner tests in batch
mode.

-Streaming mode is barely started (waiting for the
multi-aggregations support in spark Structured Streaming
framework from the Spark community)

-Runner can execute Nexmark

-Some things are not wired up yet

  -Beam Schemas not wired with Spark Schemas

  -Optional features of the model not implemented: state api,
timer api, splittable doFn api, …


2. Regarding the communication to users:

- for reasons explained by Ismael: the runner is in the same
module as the "older" one. But it is in a different
sub-package and both runners share the same build.

- How should we communicate to users:

    - shall we filter out the package name from the release?

    - should we release 2 jars: one for the old and one for
the new ?

    - should we release 3 jars: one for the new, one for the
new and one for both ?

    - should we create a special entry to the capability matrix ?

WDYT ?

Best

Etienne


On 23/10/2019 19:11, Mikhail Gryzykhin wrote:

+1 to merge.

It is worth keeping things in master with explicitly marked
status. It will make effort more visible to users and easier
to get feedback upon.

--Mikhail

On Wed, Oct 23, 2019 at 8:36 AM Etienne Chauchot
mailto:echauc...@apache.org>> wrote:

Hi guys,

The new spark runner now supports beam coders and passes
93% of the batch validates runner tests (+4%). I think it
is time to merge it to master. I will submit a PR in the
coming days.

next steps: support schemas and thus better leverage
catalyst optimizer (among other things optims based on
data), port perfs optims that were done in the current
runner.

Best

Etienne

On 11/10/2019 22:48, Pablo Estrada wrote:

+1 for merging : )

On Fri, Oct 11, 2019 at 12:43 PM Robert Bradshaw
mailto:rober...@google.com>> wrote:

Sounds like a good plan to me.

On Fri, Oct 11, 2019 at 6:20 AM Etienne Chauchot
mailto:echauc...@apache.org>>
wrote:

Comments inline

On 10/10/2019 23:44, Ismaël Mejía wrote:

+1

The earlier we get to master the better to encourage not only 
code

Re: JdbcIO read needs to fit in memory

[Jean-Baptiste Onofré]

HiJdbcIO is basically a DoFn. So it could load all on a single executor (there's no obvious way to split).It's what you mean ?RegardsJBLe 24 oct. 2019 15:26, Jozef Vilcek  a écrit :Hi,I am in a need to read a big-ish data set via JdbcIO. This forced me to bump up memory for my executor (right now using SparkRunner). It seems that JdbcIO has a requirement to fit all data in memory as it is using DoFn to unfold query to list of elements.BoundedSource would not face the need to fit result in memory, but JdbcIO is using DoFn. Also, in recent discussion [1] it was suggested that BoudnedSource should not be used as it is obsolete.Does anyone faced this issue? What would be the best way to solve it? If DoFn should be kept, then I can only think of splitting the query to ranges and try to find most fitting number of rows to read at once.I appreciate any thoughts. [1] https://lists.apache.org/list.html?dev@beam.apache.org:lte=1M:Reading%20from%20RDB%2C%20ParDo%20or%20BoundedSource

Re: Contributor permission for Beam Jira tickets

[Jean-Baptiste Onofré]

Hi Israel,

Welcome aboard !

I just added you in contributors on Jira.

Regards
JB

On 22/10/2019 21:43, Israel Herraiz wrote:
> Hi,
> 
> My name is Israel, and I work in the Professional Services team at Google.
> 
> I have created a JIRA issue and submitted the corresponding pull request
> (see https://issues.apache.org/jira/browse/BEAM-8458), and I would like
> to be able to assign the issue to myself.
> 
> That's my second pull request sent to Apache Beam.
> 
> My JIRA username is iht
> (see https://issues.apache.org/jira/secure/ViewProfile.jspa?name=iht).
> 
> Please could anyone give me that permission in JIRA?
> 
> Thanks in advance.
> 
> Israel
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [spark structured streaming runner] merge to master?

[Jean-Baptiste Onofré]

+1

As the runner seems almost "equivalent" to the one we have, it makes sense.

Question is: do we keep the "old" spark runner for a while or not (or
just keep on previous version/tag on git) ?

Regards
JB

On 10/10/2019 09:39, Etienne Chauchot wrote:
> Hi guys,
> 
> You probably know that there has been for several months an work
> developing a new Spark runner based on Spark Structured Streaming
> framework. This work is located in a feature branch here:
> https://github.com/apache/beam/tree/spark-runner_structured-streaming
> 
> To attract more contributors and get some user feedback, we think it is
> time to merge it to master. Before doing so, some steps need to be
> achieved:
> 
> - finish the work on spark Encoders (that allow to call Beam coders)
> because, right now, the runner is in an unstable state (some transforms
> use the new way of doing ser/de and some use the old one, making a
> pipeline incoherent toward serialization)
> 
> - clean history: The history contains commits from November 2018, so
> there is a good amount of work, thus a consequent number of commits.
> They were already squashed but not from September 2019
> 
> Regarding status:
> 
> - the runner passes 89% of the validates runner tests in batch mode. We
> hope to pass more with the new Encoders
> 
> - Streaming mode is barely started (waiting for the multi-aggregations
> support in spark SS framework from the Spark community)
> 
> - Runner can execute Nexmark
> 
> - Some things are not wired up yet
> 
>     - Beam Schemas not wired with Spark Schemas
> 
>     - Optional features of the model not implemented:  state api, timer
> api, splittable doFn api, …
> 
> WDYT, can we merge it to master once the 2 steps are done ?
> 
> Best
> 
> Etienne
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.16.0, release candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Quickly tested on couple of examples.

Regards
JB

On 01/10/2019 18:18, Mark Liu wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #1 for the version
> 2.16.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint C110B1C82074883A4241D977599D6305FF3ABB32 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag ""v2.16.0-RC1" [5],
> * website pull request listing the release [6], publishing the API
> reference manual [7], and the blog post [8].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
> * Validation sheet with a tab for 2.16.0 release to help with validation
> [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Mark Liu, Release Manager
> 
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345494
> [2] https://dist.apache.org/repos/dist/dev/beam/2.16.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1085/
> [5] https://github.com/apache/beam/tree/v2.16.0-RC1
> [6] https://github.com/apache/beam/pull/9667
> [7] https://github.com/apache/beam-site/pull/593
> [8] https://github.com/apache/beam/pull/9671
> [9]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=890914284
> [10] https://hub.docker.com/u/apachebeam

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra flaky on Jenkins?

[Jean-Baptiste Onofré]

Hi Etienne,

let me take a look, I'm not sure.

Regards
JB

On 19/09/2019 16:42, Etienne Chauchot wrote:
> Hi all,
> I just created a PR (1) that tries to fix the flakiness of
> CassandraIOTest (underlying
> ticket https://jira.apache.org/jira/browse/BEAM-8025 that was assigned
> to me). We will see with the test repetitions if it is no more flaky.
> 
> JB, I don't know if my PR will also fix the ticket
> https://issues.apache.org/jira/browse/BEAM-7355 assigned to you, or if
> the tickets are the same/related. I hope it does.
> 
> 
> [1]
> <https://github.com/apache/beam/pull/9614>https://github.com/apache/beam/pull/9614
> 
> Best,
> Etienne
> Le mercredi 04 septembre 2019 à 16:27 +0200, Jean-Baptiste Onofré a écrit :
>> Thanks David,
>>
>> it makes sense, it gives me time to investigate and fix.
>>
>> Regards
>> JB
>>
>> On 04/09/2019 15:01, David Morávek wrote:
>> Hi, temporarily disabling the test
>> <https://github.com/apache/beam/pull/9470>, until BEAM-8025
>> <https://jira.apache.org/jira/browse/BEAM-8025> is resolved (marking it
>> as blocker for 2.16), so we can unblock ongoing pull requests.
>>
>> Best,
>> D.
>>
>> On Tue, Sep 3, 2019 at 3:57 PM Jean-Baptiste Onofré > <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
>>
>> Hi Max,
>>
>> yup, I'm starting the investigation.
>>
>> I keep you posted.
>>
>> Regards
>> JB
>>
>> On 03/09/2019 15:34, Maximilian Michels wrote:
>> > The newest incarnation of this is here:
>> > https://jira.apache.org/jira/browse/BEAM-8025
>> >
>> > Would be good if you could take a look JB.
>> >
>> > Thanks,
>> > Max
>> >
>> > On 03.09.19 15:32, David Morávek wrote:
>> >> yes, that looks similar. example:
>> >>
>> >> https://github.com/apache/beam/pull/9464
>> >>
>> >> D.
>> >>
>> >> On 3 Sep 2019, at 15:18, Jean-Baptiste Onofré > <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>
>> >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net> 
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>> wrote:
>>     >>
>> >>> Thanks David,
>> >>>
>> >>> the build is running on my machine to see if I can reproduce
>> locally.
>> >>>
>> >>> It sounds like https://issues.apache.org/jira/browse/BEAM-7355
>> right ?
>> >>>
>> >>> Regards
>> >>> JB
>> >>>
>> >>> On 03/09/2019 15:11, David Morávek wrote:
>> >>>> I’m running into these failures too
>> >>>>
>> >>>> D.
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>>> On 3 Sep 2019, at 14:34, Jean-Baptiste Onofré > <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>
>> >>>>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net> 
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> Let me take a look. Do you always have this issue on Jenkins or
>> >>>>> randomly ?
>> >>>>>
>> >>>>> Regards
>> >>>>> JB
>> >>>>>
>> >>>>>> On 03/09/2019 14:19, Alex Van Boxel wrote:
>> >>>>>> Hi, is it only me that are bumping on the flaky Cassandra on
>> >>>>>> Jenkins? I
>> >>>>>> like to get my PR approved but I can't get past the Cassandra
>> >>>>>> error...
>>     >>>>>>
>> >>>>>> * org.apache.beam.sdk.io
>> <http://org.apache.beam.sdk.io>.cassandra.CassandraIOTest.classMethod
>> >>>>>>
>>       
>> <https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1300/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> _/
>> >>>>>> _/ Alex Van Boxel
>> >>>>>
>> >>>>> -- 
>> >>>>> Jean-Baptiste Onofré
>> >>>>> jbono...@apache.org <mailto:jbono...@apache.org> 
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org> 
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>>
>> >>>>> http://blog.nanthrax.net
>> >>>>> Talend - http://www.talend.com
>> >>>
>> >>> -- 
>> >>> Jean-Baptiste Onofré
>> >>> jbono...@apache.org <mailto:jbono...@apache.org> 
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org> 
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>>
>> >>> http://blog.nanthrax.net
>> >>> Talend - http://www.talend.com
>>
>> -- 
>> Jean-Baptiste Onofré
>> jbono...@apache.org <mailto:jbono...@apache.org> 
>> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>
>>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: MQTT to Python SDK

[Jean-Baptiste Onofré]

Regarding Java SDK, you have MqttIO available.

Regards
JB

On 16/09/2019 21:07, Lucas Magalhães wrote:
> Thanks Altay.. Do you know where I could find more about cross language
> transforms? Documentation and examples as well.
> 
> thanks again
> 
> On Mon, Sep 16, 2019 at 4:00 PM Ahmet Altay  <mailto:al...@google.com>> wrote:
> 
> A framework for python sdk to use a native unbounded connector does
> not exist yet. You might be able to use the same connector from Java
> using cross language transforms.
> 
> /cc +Chamikara Jayalath <mailto:chamik...@google.com>  
> 
> On Mon, Sep 16, 2019 at 11:00 AM Lucas Magalhães
>  <mailto:lucas.magalh...@paralelocs.com.br>> wrote:
> 
> Hello dears!
> 
> I'm starding a new project here and the mainly source is a MQTT.
> 
> I could´n find any documentantion about to How to develeop a
> unbounded connector.
> 
> Could anyone send me some instructions or guide line?
> 
> Thanks a lot
> 
> -- 
> Lucas Magalhães,
> CTO
> 
> Paralelo CS - Consultoria e Serviços
> Tel: +55 (11) 3090-5557 
> Cel: +55 (11) 99420-4667 
> lucas.magalh...@paralelocs.com.br
> <mailto:lucas.magalh...@paralelocs.com.br>
> 
> <http://www.inteligenciaemnegocios.com.br>www.paralelocs.com.br
> <http://www.paralelocs.com.br>
> 
> 
> 
> -- 
> Lucas Magalhães,
> CTO
> 
> Paralelo CS - Consultoria e Serviços
> Tel: +55 (11) 3090-5557
> Cel: +55 (11) 99420-4667
> lucas.magalh...@paralelocs.com.br <mailto:lucas.magalh...@paralelocs.com.br>
> 
> <http://www.inteligenciaemnegocios.com.br>www.paralelocs.com.br
> <http://www.paralelocs.com.br>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Jira - Contributor

[Jean-Baptiste Onofré]

Done:

you are in the contributor list and the Jira is assigned to you.

Regards
JB

On 05/09/2019 10:31, Matthew Darwin wrote:
> Hi,
> 
> Could I please be added as a contributor on Jira so I can assign
> https://issues.apache.org/jira/browse/BEAM-8153 to me?
> 
> Kind regards
> 
> Matthew

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra flaky on Jenkins?

[Jean-Baptiste Onofré]

Thanks David,

it makes sense, it gives me time to investigate and fix.

Regards
JB

On 04/09/2019 15:01, David Morávek wrote:
> Hi, temporarily disabling the test
> <https://github.com/apache/beam/pull/9470>, until BEAM-8025
> <https://jira.apache.org/jira/browse/BEAM-8025> is resolved (marking it
> as blocker for 2.16), so we can unblock ongoing pull requests.
> 
> Best,
> D.
> 
> On Tue, Sep 3, 2019 at 3:57 PM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi Max,
> 
> yup, I'm starting the investigation.
> 
> I keep you posted.
> 
> Regards
> JB
> 
> On 03/09/2019 15:34, Maximilian Michels wrote:
> > The newest incarnation of this is here:
> > https://jira.apache.org/jira/browse/BEAM-8025
> >
> > Would be good if you could take a look JB.
> >
> > Thanks,
> > Max
> >
> > On 03.09.19 15:32, David Morávek wrote:
> >> yes, that looks similar. example:
> >>
> >> https://github.com/apache/beam/pull/9464
> >>
> >> D.
> >>
> >> On 3 Sep 2019, at 15:18, Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>
> >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
> >>
> >>> Thanks David,
> >>>
> >>> the build is running on my machine to see if I can reproduce
> locally.
> >>>
> >>> It sounds like https://issues.apache.org/jira/browse/BEAM-7355
> right ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 03/09/2019 15:11, David Morávek wrote:
> >>>> I’m running into these failures too
> >>>>
> >>>> D.
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 3 Sep 2019, at 14:34, Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>
> >>>>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> Let me take a look. Do you always have this issue on Jenkins or
> >>>>> randomly ?
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 03/09/2019 14:19, Alex Van Boxel wrote:
> >>>>>> Hi, is it only me that are bumping on the flaky Cassandra on
> >>>>>> Jenkins? I
> >>>>>> like to get my PR approved but I can't get past the Cassandra
> >>>>>> error...
> >>>>>>
> >>>>>> * org.apache.beam.sdk.io
> <http://org.apache.beam.sdk.io>.cassandra.CassandraIOTest.classMethod
> >>>>>>
>   
> <https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1300/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _/
> >>>>>> _/ Alex Van Boxel
> >>>>>
> >>>>> -- 
> >>>>> Jean-Baptiste Onofré
> >>>>> jbono...@apache.org <mailto:jbono...@apache.org>
> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
> >>>>> http://blog.nanthrax.net
> >>>>> Talend - http://www.talend.com
> >>>
> >>> -- 
> >>> Jean-Baptiste Onofré
> >>> jbono...@apache.org <mailto:jbono...@apache.org>
> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra flaky on Jenkins?

[Jean-Baptiste Onofré]

Hi Max,

yup, I'm starting the investigation.

I keep you posted.

Regards
JB

On 03/09/2019 15:34, Maximilian Michels wrote:
> The newest incarnation of this is here:
> https://jira.apache.org/jira/browse/BEAM-8025
> 
> Would be good if you could take a look JB.
> 
> Thanks,
> Max
> 
> On 03.09.19 15:32, David Morávek wrote:
>> yes, that looks similar. example:
>>
>> https://github.com/apache/beam/pull/9464
>>
>> D.
>>
>> On 3 Sep 2019, at 15:18, Jean-Baptiste Onofré > <mailto:j...@nanthrax.net>> wrote:
>>
>>> Thanks David,
>>>
>>> the build is running on my machine to see if I can reproduce locally.
>>>
>>> It sounds like https://issues.apache.org/jira/browse/BEAM-7355 right ?
>>>
>>> Regards
>>> JB
>>>
>>> On 03/09/2019 15:11, David Morávek wrote:
>>>> I’m running into these failures too
>>>>
>>>> D.
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 3 Sep 2019, at 14:34, Jean-Baptiste Onofré >>>> <mailto:j...@nanthrax.net>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Let me take a look. Do you always have this issue on Jenkins or
>>>>> randomly ?
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> On 03/09/2019 14:19, Alex Van Boxel wrote:
>>>>>> Hi, is it only me that are bumping on the flaky Cassandra on
>>>>>> Jenkins? I
>>>>>> like to get my PR approved but I can't get past the Cassandra
>>>>>> error...
>>>>>>
>>>>>> * org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod
>>>>>>   
>>>>>> <https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1300/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _/
>>>>>> _/ Alex Van Boxel
>>>>>
>>>>> -- 
>>>>> Jean-Baptiste Onofré
>>>>> jbono...@apache.org <mailto:jbono...@apache.org>
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>
>>> -- 
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org <mailto:jbono...@apache.org>
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra flaky on Jenkins?

[Jean-Baptiste Onofré]

Thanks David,

the build is running on my machine to see if I can reproduce locally.

It sounds like https://issues.apache.org/jira/browse/BEAM-7355 right ?

Regards
JB

On 03/09/2019 15:11, David Morávek wrote:
> I’m running into these failures too
> 
> D.
> 
> Sent from my iPhone
> 
>> On 3 Sep 2019, at 14:34, Jean-Baptiste Onofré  wrote:
>>
>> Hi,
>>
>> Let me take a look. Do you always have this issue on Jenkins or randomly ?
>>
>> Regards
>> JB
>>
>>> On 03/09/2019 14:19, Alex Van Boxel wrote:
>>> Hi, is it only me that are bumping on the flaky Cassandra on Jenkins? I
>>> like to get my PR approved but I can't get past the Cassandra error...
>>>
>>>  * org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod
>>>
>>> <https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1300/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/>
>>>
>>>
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>
>> -- 
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra flaky on Jenkins?

[Jean-Baptiste Onofré]

Hi,

Let me take a look. Do you always have this issue on Jenkins or randomly ?

Regards
JB

On 03/09/2019 14:19, Alex Van Boxel wrote:
> Hi, is it only me that are bumping on the flaky Cassandra on Jenkins? I
> like to get my PR approved but I can't get past the Cassandra error...
> 
>   * org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod
> 
> <https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1300/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/>
> 
> 
> 
>  _/
> _/ Alex Van Boxel

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Query about JdbcIO.readRows()

[Jean-Baptiste Onofré]

Agree. I will fix that.

Regards
JB

Le 2 août 2019 à 17:15, à 17:15, Vishwas Bm  a écrit:
>Hi Kishor,
>
>+ dev (dev@beam.apache.org)
>
>This looks like a bug.  The attribute statementPreparator is nullable
>It should have been handled in the same way as in the expand method of
>Read class.
>
>
>*Thanks & Regards,*
>
>*Vishwas *
>
>
>On Fri, Aug 2, 2019 at 2:48 PM Kishor Joshi  wrote:
>
>> Hi,
>>
>> I am using the just released 2.14 version for JdbcIO with the newly
>added
>> "readRows" functionality.
>>
>> I want to read table data with a query without parameters (select *
>from
>> table_name).
>> As per my understanding, this should not require
>"StatementPreperator".
>> However, if I use the newly added "readRows" function, I get an
>exception
>> that seems to force me to use the "StatementPreperator".
>> Stacktrace below.
>>
>> java.lang.IllegalArgumentException: statementPreparator can not be
>null
>> at
>>
>org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>> at
>>
>org.apache.beam.sdk.io.jdbc.JdbcIO$Read.withStatementPreparator(JdbcIO.java:600)
>> at
>> org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.expand(JdbcIO.java:499)
>> at
>> org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.expand(JdbcIO.java:410)
>> at
>org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>> at
>org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>> at
>>
>com.nokia.csf.dfle.transforms.DfleRdbmsSource.expand(DfleRdbmsSource.java:34)
>> at
>>
>com.nokia.csf.dfle.transforms.DfleRdbmsSource.expand(DfleRdbmsSource.java:10)
>> at
>org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>> at
>org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>> at
>> com.nokia.csf.dfle.dsl.DFLEBeamMain.dagWireUp(DFLEBeamMain.java:49)
>> at
>com.nokia.csf.dfle.dsl.DFLEBeamMain.main(DFLEBeamMain.java:120)
>>
>>
>>
>> The test added in JdbcIOTest.java for this functionality only tests
>for
>> queries with parameters.
>> Is this new function supported only in the above case and not for
>normal
>> "withQuery" (without parameters) ?
>>
>>
>> Thanks & regards,
>> Kishor
>>

Re: WebSocket/Https connector for Apache Beam (Java)?

[Jean-Baptiste Onofré]

Hi,

I have a websocket and REST IO about that (I proposed the IO while ago).

As you are not the first one to ask for such IO, I will revive the IO.

Regards
JB

On 02/07/2019 13:34, I-Feng Lin wrote:
> Hello all,
> 
> I have an Apache Beam pipeline in Java where I would like to read data
> that comes from a WebSocket and write data to the server through Https.
> 
> I have been looking for some connectors but so far my search was
> unsuccessful. I know that it is possible to create custom connectors but
> I want to check if there is anything already exists.
> 
> Thanks in advance,
> 
> Ifeng
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Preparing for Beam 2.14.0 release

[Jean-Baptiste Onofré]

+1

Regards
JB

Le 6 juin 2019 à 19:02, à 19:02, Ankur Goenka  a écrit:
>+1
>
>On Thu, Jun 6, 2019, 9:13 AM Ahmet Altay  wrote:
>
>> +1, thank you for keeping the cadence.
>>
>> On Thu, Jun 6, 2019 at 9:04 AM Anton Kedin  wrote:
>>
>>> Hello Beam community!
>>>
>>> Beam 2.14 release branch cut date is June 19 according to the
>release
>>> calendar [1]. I would like to volunteer myself to do this release.
>The plan
>>> is to cut the branch on that date, and cherrypick fixes if needed.
>>>
>>> If you have release blocking issues for 2.14 please mark their "Fix
>>> Version" as 2.14.0 [2]. Please use 2.15.0 release in JIRA in case
>you
>>> would like to move any non-blocking issues to that version.
>>>
>>> And if we're doing a 2.7.1 release it should probably happen
>>> independently and in parallel if we want to maintain the release
>cadence.
>>>
>>> Thoughts, comments, objections?
>>>
>>> Thanks,
>>> Anton
>>>
>>> [1]
>>>
>https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>> [2]
>>>
>https://issues.apache.org/jira/browse/BEAM-7478?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.14.0
>>>
>>

Re: [VOTE] Release 2.13.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Quickly tested on beam-samples.

Regards
JB

On 31/05/2019 04:52, Ankur Goenka wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #2 for the version
> 2.13.0, as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint 6356C1A9F089B0FA3DE8753688934A6699985948 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.13.0-RC2" [5],
> * website pull request listing the release [6] and publishing the API
> reference manual [7].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
> * Validation sheet with a tab for 2.13.0 release to help with validation
> [8].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Ankur
> 
> [1]
> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345166
> [2] https://dist.apache.org/repos/dist/dev/beam/2.13.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1070/
> [5] https://github.com/apache/beam/tree/v2.13.0-RC2
> [6] https://github.com/apache/beam/pull/8645
> [7] https://github.com/apache/beam-site/pull/589
> [8]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1031196952

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra and hadoop test broken on master and in previous releases

[Jean-Baptiste Onofré]

Hi,

It works fine on my machine. I'm using this JDK:

java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

Regarding your Gradle scan, the JVM is crashing.

Do you have a chance to try another JDK ?

I will try with OpenJDK 1.8.0_181.

Regards
JB

On 11/05/2019 01:34, Ankur Goenka wrote:
> Hi,
> 
> Cassandra and Hadoop tests for targets :beam-sdks-java-io-cassandra:test
> :beam-sdks-java-io-hadoop-format:test are failing at master and in
> 2.12.0 release with jvm crash. 
> 
> Gradle Scan: https://gradle.com/s/rhseoqeouup6e
> 
> Any help on the debugging failure will be useful.
> 
> Thanks,
> Ankur

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Cassandra and hadoop test broken on master and in previous releases

[Jean-Baptiste Onofré]

Hi,

let me try to reproduce on my box.

Regards
JB

On 11/05/2019 01:34, Ankur Goenka wrote:
> Hi,
> 
> Cassandra and Hadoop tests for targets :beam-sdks-java-io-cassandra:test
> :beam-sdks-java-io-hadoop-format:test are failing at master and in
> 2.12.0 release with jvm crash. 
> 
> Gradle Scan: https://gradle.com/s/rhseoqeouup6e
> 
> Any help on the debugging failure will be useful.
> 
> Thanks,
> Ankur

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: request for beam minor release

[Jean-Baptiste Onofré]

I second Max here. If you are just looking for this specific commit, you
can take a next release that will include it.

Regards
JB

On 08/05/2019 16:27, Maximilian Michels wrote:
> Hi Richard,
> 
> Would it be an option to use the upcoming 2.13.0 release? The commit
> will be part of that release.
> 
> Thanks,
> Max
> 
> On 08.05.19 15:43, Jean-Baptiste Onofré wrote:
>> Hi,
>>
>> Any release are tagging. We create a branch based on a master commit.
>>
>> Are you requesting 2.10.1 maintenance release ?
>>
>> Regards
>> JB
>>
>> On 08/05/2019 15:10, Moorhead,Richard wrote:
>>> Is there a process for tagging a commit in master for a minor release?
>>>
>>> I am trying to get this
>>> <https://github.com/apache/beam/pull/8503/commits/ffa5632bca8c7264993702c39c6ca013a9f6ecdb>
>>>  commit
>>>
>>> released into 2.10.1
>>>  
>>> CONFIDENTIALITY NOTICE This message and any included attachments are
>>> from Cerner Corporation and are intended only for the addressee. The
>>> information contained in this message is confidential and may constitute
>>> inside or non-public information under international, federal, or state
>>> securities laws. Unauthorized forwarding, printing, copying,
>>> distribution, or use of such information is strictly prohibited and may
>>> be unlawful. If you are not the addressee, please promptly delete this
>>> message and notify the sender of the delivery error by e-mail or you may
>>> call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1)
>>> (816)221-1024.
>>>
>>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: request for beam minor release

[Jean-Baptiste Onofré]

Hi,

Any release are tagging. We create a branch based on a master commit.

Are you requesting 2.10.1 maintenance release ?

Regards
JB

On 08/05/2019 15:10, Moorhead,Richard wrote:
> Is there a process for tagging a commit in master for a minor release?
> 
> I am trying to get this
> <https://github.com/apache/beam/pull/8503/commits/ffa5632bca8c7264993702c39c6ca013a9f6ecdb>
>  commit
> released into 2.10.1
>  
> 
> CONFIDENTIALITY NOTICE This message and any included attachments are
> from Cerner Corporation and are intended only for the addressee. The
> information contained in this message is confidential and may constitute
> inside or non-public information under international, federal, or state
> securities laws. Unauthorized forwarding, printing, copying,
> distribution, or use of such information is strictly prohibited and may
> be unlawful. If you are not the addressee, please promptly delete this
> message and notify the sender of the delivery error by e-mail or you may
> call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1)
> (816)221-1024.
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: :beam-sdks-java-io-hadoop-input-format:test is extremely flaky

[Jean-Baptiste Onofré]

Agree, +1

Regards
JB

On 29/04/2019 15:30, Ismaël Mejía wrote:
> +1 to remove it on this release, this is a maintenance pain for no real 
> reason.
> 
> On Mon, Apr 29, 2019 at 3:06 PM Alexey Romanenko
>  wrote:
>>
>> Despite the fact that after fixing an issue with ports allocation (thanks to 
>> Etienne!) for embedded Cassandra cluster (it’s used in hadoop-input-format 
>> and this was the main cause of flakiness) it's got much better, I’m 100% pro 
>> to remove this module since it’s already been deprecated for several last 
>> releases.
>>
>> PS: Just an observation when I was digging into PreCommit jobs results - 
>> “org.apache.beam.runners.dataflow.worker.fn.BeamFnControlServiceTest.testClientConnecting”
>>  fails quite often in the last time. Anyone works on this?
>>
>>> On 29 Apr 2019, at 14:19, Maximilian Michels  wrote:
>>>
>>> I don't know what going on with it but I agree it's annoying.
>>>
>>> Came across https://jira.apache.org/jira/browse/BEAM-6247, maybe it is time 
>>> to remove this module for the next release?
>>>
>>> -Max
>>>
>>> On 26.04.19 20:10, Reuven Lax wrote:
>>>> I find I usually have to rerun Presubmit multiple times to get a green 
>>>> run, and this test is one of the biggest culprits (though it's not the 
>>>> only culprit). Does anyone know what's going on with it?
>>>> Reuven
>>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Add new JIRA component for Python IO?

[Jean-Baptiste Onofré]

It sounds good. +1

Regards
JB

On 25/04/2019 21:43, Pablo Estrada wrote:
> Hello all,
> there are only two JIRA components for python: `sdk-py-core`, and
> `sdk-py-harness`. Naturally, sdk-py-core is the component with the most
> bugs (>1000).
> 
> I believe we really need a component to route issues for Python IO
> (`io-python-all`?). Maybe even something more granular.
> 
> Thoughts?
> -P.
> 
> 

Re: Hello from Hannah Jiang

[Jean-Baptiste Onofré]

Welcome aboard !

Regards
JB

On 25/04/2019 19:55, Griselda Cuevas wrote:
> Welcome Hannah! - Very excited to see you in the Beam community :) 
> 
> On Tue, 23 Apr 2019 at 12:59, Hannah Jiang  > wrote:
> 
> Hi everyone
> 
> I joined Google recently and would work on Python portability part.
> I am happy to be part of the community. Looking forward to working
> with all of you together.
> 
> I have a minor request, can admin please give me access to JIRA?
> 
> Thanks,
> Hannah
> 
> 

Re: [PROPOSAL] Prepare for LTS bugfix release 2.7.1

[Jean-Baptiste Onofré]

+1 it sounds good to me.

Thanks !

Regards
JB

On 26/04/2019 02:42, Kenneth Knowles wrote:
> Hi all,
> 
> Since the release of 2.7.0 we have identified some serious bugs:
> 
>  - There are 8 (non-dupe) issues* tagged with Fix Version 2.7.1
>  - 2 are rated "Blocker" (aka P0) but I think the others may be underrated
>  - If you know of a critical bug that is not on that list, please file
> an LTS backport ticket for it
> 
> If a user is on an old version and wants to move to the LTS, there are
> some real blockers. I propose that we perform a 2.7.1 release starting now.
> 
> I volunteer to manage the release. What do you think?
> 
> Kenn
> 
> *Some are "resolved" but this is not accurate as the LTS 2.7.1 branch is
> not created yet. I suggest filing a ticket to track just the LTS
> backport when you hit a bug that merits it.
> 

Re: CassandraIO breakage

[Jean-Baptiste Onofré]

It builds fine on my machine.

Let me check on Jenkins.

Regards
JB

On 17/04/2019 21:48, Reuven Lax wrote:
> Did something break with CassandraIO? It no longer seems to compile.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: CassandraIO breakage

[Jean-Baptiste Onofré]

Let me check if it works on my machine.

Regards
JB

On 17/04/2019 21:48, Reuven Lax wrote:
> Did something break with CassandraIO? It no longer seems to compile.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Wait on JdbcIO write completion

[Jean-Baptiste Onofré]

I second Alexey (and thanks Alexey ;)).

I also started similar improvements in other IOs (PRs will come soon).

Regards
JB

On 17/04/2019 17:31, Alexey Romanenko wrote:
> Hi Jonathan,
> 
> I just wanted to let you know that this feature [1] was implemented and,
> finally, merged into master. So, it should be included into next Beam
> 2.13 release.
> 
> In few words, it was added new method called “/Write.withResults()/”
> which returns /WriteVoid/ transform that provides “/PCollection/”
> as an output and can be used together with "/Wait.on()"/. So, the simple
> example of writing into two different databases can look like this:
> 
> /PCollection firstWriteResults = data.apply(JdbcIO.write()
>     .withDataSourceConfiguration(CONF_DB_1).withResults());
> data.apply(Wait.on(firstWriteResults))
>     .apply(JdbcIO.write().withDataSourceConfiguration(CONF_DB_2));/
> 
> [1] https://issues.apache.org/jira/browse/BEAM-6732
> 
>> On 22 Feb 2019, at 16:52, Alexey Romanenko > <mailto:aromanenko@gmail.com>> wrote:
>>
>> I have created new Jira issue for this feature:
>> https://issues.apache.org/jira/browse/BEAM-6732
>>
>> Jonathan, feel free to assign it to yourself if you want to
>> contribute, it is always welcomed =)
>>
>>> On 21 Feb 2019, at 10:23, Jonathan Perron
>>> mailto:jonathan.per...@lumapps.com>> wrote:
>>>
>>> Thank you Eugene for your answer.
>>>
>>> According to your explanation, I think I will go with your 3rd
>>> solution, as this seems the most robust and friendly way to act.
>>>
>>> Jonathan
>>>
>>> On 21/02/2019 02:22, Eugene Kirpichov wrote:
>>>> Hi Jonathan,
>>>>
>>>> Wait.on() requires a PCollection - it is not possible to change it
>>>> to wait on PDone because all PDone's in the pipeline are the same so
>>>> it's not clear what exactly you'd be waiting on.
>>>>
>>>> To use the Wait transform with JdbcIO.write(), you would need to
>>>> change 
>>>> https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L761-L762
>>>>  to
>>>> simply "return input.apply(ParDo.of(...))" and propagate that into
>>>> the type signature. Then you'd get a waitable PCollection.
>>>>
>>>> This is a very simple, but backwards-incompatible change. Up to the
>>>> Beam community whether/when people would want to make it.
>>>>
>>>> It's also possible to make a slightly larger but compatible change,
>>>> where JdbcIO.write() would stay as is, but you could write e.g.
>>>> "JdbcIO.write().withResults()" which would be a new transform that
>>>> *does* return results and is waitable. A similar approach is taken
>>>> in TextIO.write().withOutputFilenames().
>>>>
>>>> On Wed, Feb 20, 2019 at 4:58 AM Jonathan Perron
>>>> mailto:jonathan.per...@lumapps.com>>
>>>> wrote:
>>>>
>>>> Hello folks,
>>>>
>>>> I am meeting a special case where I need to wait for a
>>>> JdbcIO.write()
>>>> operation to be complete to start a second one.
>>>>
>>>> In the details, I have a PCollection> which
>>>> is used
>>>> to fill two different SQL statement. It is used in a first
>>>> JdbcIO.write() operation to store anonymized user in a table
>>>> (userId
>>>> with an associated userUuid generated with UUID.randomUUID()).
>>>> These two
>>>> parameters have a unique constraint, meaning that a userId
>>>> cannot have
>>>> multiple userUuid. Unfortunately, on several runs of my
>>>> pipeline, the
>>>> UUID will be different, meaning that I need to query this table
>>>> at some
>>>> point, or to use what I describe in the following.
>>>>
>>>> I am planning to fill a second table with this userUuid with a
>>>> couple of
>>>>     others information such as the time of first visit. To limit I/O
>>>> and as
>>>> I got a lot of information in my PCollection, I want to use it
>>>> once more
>>>> with a different SQL statement, where the userUuid is read from the
>>>> first table using a SELECT statement. This cannot work if the first
>>>> JdbcIO.write() operation is not complete.
>>>>
>>>> I saw that the Java SDK proposes a Wait.on() PTransform, but it is
>>>> unfortunately only compatible with PCollection, and not a PDone
>>>> such as
>>>> the one output from the JdbcIO operation. Could my issue be
>>>> solved by
>>>> expanding the Wait.On() or should I go with an other solution ?
>>>> If so,
>>>> how could I implement it ?
>>>>
>>>> Many thanks for your input !
>>>>
>>>> Jonathan
>>>>
>>
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: New contributor to Beam

[Jean-Baptiste Onofré]

Welcome !

Regards
JB

On 17/04/2019 16:05, Cyrus Maden wrote:
> Hi all!
> 
> My name's Cyrus and I'd like to start contributing to Beam. I'm a
> technical writer so I'm particularly looking forward to contributing to
> the Beam docs. Could someone add me as a contributor on JIRA so I can
> create and assign tickets?
> 
> My JIRA name is: *cyrusmaden*
> *
> *
> Excited to be a part of this community and to work with ya'll!
> 
> Best,
> Cyrus

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.12.0, release candidate #4

[Jean-Baptiste Onofré]

+1 (binding)

Quickly checked with beam-samples.

Regards
JB

On 16/04/2019 00:50, Andrew Pilloud wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #4 for the version
> 2.12.0, as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint 9E7CEC0661EFD610B632C610AE8FE17F9F8AE3D4 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.12.0-RC4" [5],
> * website pull request listing the release [6], publishing the API
> reference manual [7], and the blog post [8].
> * Java artifacts were built with Gradle/5.2.1 and OpenJDK/Oracle JDK
> 1.8.0_181.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
> * Validation sheet with a tab for 2.12.0 release to help with validation
> [9].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Andrew
> 
> 1] 
> https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344944
> [2] https://dist.apache.org/repos/dist/dev/beam/2.12.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1068/
> [5] https://github.com/apache/beam/tree/v2.12.0-RC4 
> [6] https://github.com/apache/beam/pull/8215
> [7] https://github.com/apache/beam-site/pull/588
> [8] https://github.com/apache/beam/pull/8314
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1007316984

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [review?] WordCount in Kotlin

[Jean-Baptiste Onofré]

Thanks for the update Pablo.

I will try to take a look during the week end.

Regards
JB

On 04/04/2019 23:16, Pablo Estrada wrote:
> Hello all,
> as community member has been very kind to contribute a Kotlin
> translation of the WordCount pipeline[1]. The documentation, tests, and
> gradle structure for it is very good, so I am happy to merge, but since
> this code will become our first Kotlin "documentation"/entrypoint, I
> wanted to be cautious.
> So if anyone wants to take a look to review the change, please do. I
> will merge this in a couple days.
> Thanks!
> -P.
> 
> [1] https://github.com/apache/beam/pull/8034

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

[Jean-Baptiste Onofré]

 configuring the MavenPublication."
> 
> 
> During the gradle migration this wasn't that easy. The
> new maven publish plugin improved a lot since then.
>  
> 
> Using the default at the time also broke the
> artifact names for intra project dependencies
> that we generate[1]. Finally, we also ran into
> an issue because we had more then one Gradle
> project with the same directory name even though
> they were under a different parent folder (I
> think it was "core") and that was leading to
> some strange build time behavior.
> 
> 
> Weird. But I think the Jira should still stand as a
> move towards simplifying our build and making it
> more discoverable for new contributors.
> 
>  
> Agree on the JIRA makes sense, just calling out that
> there were other issues that this naming had caused in
> the past which should be checked before we call this done.
> 
> 
> Totally agree. It will be quite a large task with a lot of
> boilerplate that might not be separable from technical
> blockers that come up as you go through the boilerplate.
> 
> Kenn
>  
>  
> 
> Kenn 
>  
> 
> We didn't migrate to a flat project structure
> where each project is a folder underneath the
> root project because of the existing Maven build
> rules that were being maintained in parallel and
> I'm not sure if people would want to have a flat
> project structure either.
> 
> 1: 
> https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1055
> 
> On Mon, Apr 1, 2019 at 9:49 AM Michael Luckey
>  <mailto:adude3...@gmail.com>> wrote:
> 
> Hi,
> 
> although I did not yet manage to get deeper
> involved into actual development, I think
> this ability would be a useful addition.
> 
> But I would also like to point out, that
> this is kind of implicit, as soon we
> get 
> https://issues.apache.org/jira/browse/BEAM-4046
> included.
> 
> For instance, we would change the current
> setup from
> 
> include "beam-sdks-java-core"
> project(":beam-sdks-java-core").dir = 
> file("sdks/java/core")
> 
> to something like
> 
> include(":sdks:java:core")
> include(":sdks:java:extensions:sql")
> include(":sdks:python")
> 
> 
> With this in place a plain
> 
> $ ./gradlew -p sdks/java build
> 
> 
> would exactly do what you want. And, of
> course, this will also work for
> 'sdks/java/io', 'runners/' etc. Hope, you
> get the point.
> 
> Currently, we deviate from gradle default
> convention and therefore have to implement
> some quirks to restore default behaviour.
> And I somehow dislike the structure
>     introduced by parent/child folders, which
> will be destroyed by our current project
> definitions.
> 
> But, to be honest, although I have some
> clear understanding on how to proceed here -
> especially regarding the requirement to keep
> the change backwards compatible - we might
> decide not to switch. Because deepe

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

[Jean-Baptiste Onofré]

Hi,

I mean that I did change on a local branch to be able to do:

./gradlew :beam-sdks-java:build

and/or

./gradlew -p sdks/java build

Regards
JB

On 01/04/2019 19:47, Michael Luckey wrote:
> Hmm... now you lost me :(
> 
> Currently I am not able to do a
> 
> $./gradlew -p sdks/java build
> It fails with error
> 
> Project directory '/Users/michel/GitHub/adude3141/beam/sdks/java' is not
> part of the build defined by settings file
> 
> 
> on my machine, which - again - should be expected.
> 
> Regarding the display, it would look like this if we would be able to switch
> 
> \--- Project ':sdks'
> 
>      +--- Project ':sdks:java'
> 
>      |    +--- Project ':sdks:java:core'- Apache Beam :: SDKs :: Java ::
> Core
> 
>      |    \--- Project ':sdks:java:extensions'
> 
>      |         \--- Project ':sdks:java:extensions:sql'- Apache Beam ::
> SDKs :: Java :: Extensions :: SQL
> 
>      \--- Project ':sdks:python'
> 
> 
> 
> On Mon, Apr 1, 2019 at 7:36 PM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> By the way, another reason is to have this clearly displayed in
> ./gradlew projects ;)
> 
> On 01/04/2019 18:49, Michael Luckey wrote:
> > Hi,
> >
> > although I did not yet manage to get deeper involved into actual
> > development, I think this ability would be a useful addition.
> >
> > But I would also like to point out, that this is kind of implicit, as
> > soon we get https://issues.apache.org/jira/browse/BEAM-4046 included.
> >
> > For instance, we would change the current setup from
> >
> > include "beam-sdks-java-core"
> > project(":beam-sdks-java-core").dir = file("sdks/java/core")
> >
> > to something like
> >
> > include(":sdks:java:core")
> > include(":sdks:java:extensions:sql")
> > include(":sdks:python")
> >
> >
> > With this in place a plain
> >
> > $ ./gradlew -p sdks/java build
> >
> >
> > would exactly do what you want. And, of course, this will also
> work for
> > 'sdks/java/io', 'runners/' etc. Hope, you get the point.
> >
> > Currently, we deviate from gradle default convention and therefore
> have
> > to implement some quirks to restore default behaviour. And I somehow
> > dislike the structure introduced by parent/child folders, which
> will be
> > destroyed by our current project definitions.
> >
> > But, to be honest, although I have some clear understanding on how to
> > proceed here - especially regarding the requirement to keep the change
> > backwards compatible - we might decide not to switch. Because deeper
> > investigation might reveal issues, which I am currently not aware of.
> >
> > Best,
> >
> > michel
> >
> > On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré
> mailto:j...@nanthrax.net>
> > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
> >
> >     Hi guys,
> >
> >     I would like to introduce a Gradle "meta" project for the build:
> >     beam-sdks-java.
> >
> >     The idea is to simply build all Java SDK related resources (core,
> >     IO, ...).
> >
> >     The purpose is also to be aligned with the other SDKs which
> provide
>     >     beam-sdks-go and beam-sdks-python.
> >
> >     Thoughts ?
> >
> >     Regards
> >     JB
> >     --
> >     Jean-Baptiste Onofré
> >     jbono...@apache.org <mailto:jbono...@apache.org>
> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
> >     http://blog.nanthrax.net
> >     Talend - http://www.talend.com
> >
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: kafka 0.9 support

[Jean-Baptiste Onofré]

+1 to remove 0.9 support.

I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 ;)

Regards
JB

On 01/04/2019 19:36, David Morávek wrote:
> Hello,
> 
> is there still a reason to keep Kafka 0.9 support? This unfortunately
> adds lot of complexity to KafkaIO implementation.
> 
> Kafka 0.9 was released on Nov 2015.
> 
> My first shot on removing Kafka 0.9 support would remove second
> consumer, which is used for fetching offsets.
> 
> WDYT? Is this support worth keeping?
> 
> https://github.com/apache/beam/pull/8186
> 
> D.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

[Jean-Baptiste Onofré]

By the way, another reason is to have this clearly displayed in
./gradlew projects ;)

On 01/04/2019 18:49, Michael Luckey wrote:
> Hi,
> 
> although I did not yet manage to get deeper involved into actual
> development, I think this ability would be a useful addition.
> 
> But I would also like to point out, that this is kind of implicit, as
> soon we get https://issues.apache.org/jira/browse/BEAM-4046 included.
> 
> For instance, we would change the current setup from
> 
> include "beam-sdks-java-core"
> project(":beam-sdks-java-core").dir = file("sdks/java/core")
> 
> to something like
> 
> include(":sdks:java:core")
> include(":sdks:java:extensions:sql")
> include(":sdks:python")
> 
> 
> With this in place a plain
> 
> $ ./gradlew -p sdks/java build
> 
> 
> would exactly do what you want. And, of course, this will also work for
> 'sdks/java/io', 'runners/' etc. Hope, you get the point.
> 
> Currently, we deviate from gradle default convention and therefore have
> to implement some quirks to restore default behaviour. And I somehow
> dislike the structure introduced by parent/child folders, which will be
> destroyed by our current project definitions.
> 
> But, to be honest, although I have some clear understanding on how to
> proceed here - especially regarding the requirement to keep the change
> backwards compatible - we might decide not to switch. Because deeper
> investigation might reveal issues, which I am currently not aware of.
> 
> Best,
> 
> michel
> 
> On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi guys,
> 
> I would like to introduce a Gradle "meta" project for the build:
> beam-sdks-java.
> 
> The idea is to simply build all Java SDK related resources (core,
> IO, ...).
> 
> The purpose is also to be aligned with the other SDKs which provide
> beam-sdks-go and beam-sdks-python.
> 
> Thoughts ?
> 
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

[Jean-Baptiste Onofré]

Hi Michael,

Yes, I know the -p option and it's currently what I'm using.

However the proposal is also in order to have some more "consistent"
with other modules.

Regards
JB

On 01/04/2019 18:49, Michael Luckey wrote:
> Hi,
> 
> although I did not yet manage to get deeper involved into actual
> development, I think this ability would be a useful addition.
> 
> But I would also like to point out, that this is kind of implicit, as
> soon we get https://issues.apache.org/jira/browse/BEAM-4046 included.
> 
> For instance, we would change the current setup from
> 
> include "beam-sdks-java-core"
> project(":beam-sdks-java-core").dir = file("sdks/java/core")
> 
> to something like
> 
> include(":sdks:java:core")
> include(":sdks:java:extensions:sql")
> include(":sdks:python")
> 
> 
> With this in place a plain
> 
> $ ./gradlew -p sdks/java build
> 
> 
> would exactly do what you want. And, of course, this will also work for
> 'sdks/java/io', 'runners/' etc. Hope, you get the point.
> 
> Currently, we deviate from gradle default convention and therefore have
> to implement some quirks to restore default behaviour. And I somehow
> dislike the structure introduced by parent/child folders, which will be
> destroyed by our current project definitions.
> 
> But, to be honest, although I have some clear understanding on how to
> proceed here - especially regarding the requirement to keep the change
> backwards compatible - we might decide not to switch. Because deeper
> investigation might reveal issues, which I am currently not aware of.
> 
> Best,
> 
> michel
> 
> On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi guys,
> 
> I would like to introduce a Gradle "meta" project for the build:
> beam-sdks-java.
> 
> The idea is to simply build all Java SDK related resources (core,
> IO, ...).
> 
> The purpose is also to be aligned with the other SDKs which provide
>     beam-sdks-go and beam-sdks-python.
> 
> Thoughts ?
> 
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[PROPOSAL] Introduce beam-sdks-java gradle project

[Jean-Baptiste Onofré]

Hi guys,

I would like to introduce a Gradle "meta" project for the build:
beam-sdks-java.

The idea is to simply build all Java SDK related resources (core, IO, ...).

The purpose is also to be aligned with the other SDKs which provide
beam-sdks-go and beam-sdks-python.

Thoughts ?

Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [spark runner dataset POC] workCount works !

[Jean-Baptiste Onofré]

Congrats and huge thanks !

(I'm glad to be one of the little "launcher" to this effort ;) )

Regards
JB

On 21/03/2019 15:47, Ismaël Mejía wrote:

This is excellent news. Congrats Etienne, Alexey and the others
involved for the great work!

On Thu, Mar 21, 2019 at 3:10 PM Etienne Chauchot  wrote:


Hi guys,

We are glad to announce that the spark runner POC that was re-written from 
scratch using the structured-streaming framework and the dataset API can now 
run WordCount !

It is still embryonic. For now it only runs in batch mode and there is no fancy 
stuff like state, timer, SDF, metrics, ... but it is still a major step forward 
!

Streaming support work has just started.

You can find the branch here: 
https://github.com/apache/beam/tree/spark-runner_structured-streaming

Enjoy,

Etienne

Re: [VOTE] Release 2.11.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Tested with beam-samples.

Regards
JB

On 26/02/2019 10:40, Ahmet Altay wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #2 for the version
> 2.11.0, as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint 64B84A5AD91F9C20F5E9D9A7D62E71416096FA00 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.11.0-RC2" [5],
> * website pull request listing the release [6] and publishing the API
> reference manual [7].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
> * Validation sheet with a tab for 2.11.0 release to help with validation
> [8].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Ahmet
> 
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344775
> [2] https://dist.apache.org/repos/dist/dev/beam/2.11.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1064/
> [5] https://github.com/apache/beam/tree/v2.11.0-RC2
> [6] https://github.com/apache/beam/pull/7924
> [7] https://github.com/apache/beam-site/pull/587
> [8]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=542393513
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Issue building master

[Jean-Baptiste Onofré]

Hi Michael,

Thanks for the update, I'm updating my box right now.

Regards
JB

On 25/02/2019 13:03, Michael Luckey wrote:
> Hi JB,
> 
> great your are back.
> 
> Last time I encountered those issues where
> 
> 1. python virtualEnv failling on missing python3.5. Note: Build
> currently asks explicitly for python3.5, so u have to make sure, that's
> on your path. But of course you might have hit another issue, difficult
> to diagnose without further insights.
> 
> 2. While testing with  "-PisRelase" enabled I also hit that issue. The
> release path seems to install that locally which causes rat failing on
> next build. Not sure, whether u had that flag set, though. I 'resolved'
> that issue for me manually deleting that created folder before issuing a
> new Gradle build. I am unsure, whether they should be produced with
> license included or if the need to be excluded from rat checks, as Kenn
> suggested.
> 
> hth,
> michel
> 
> 
> On Mon, Feb 25, 2019 at 7:05 AM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi Kenn,
> 
> I started with a git clean -fdx, so it's maybe the build which created
> those files.
> 
> Regards
> JB
> 
> On 25/02/2019 00:16, Kenneth Knowles wrote:
> > To #2 it looks like it has done an install relative to the Python SDK
> > directory. The files are not set up to be ignored by git/RAT.
> >
> > Kenn
> >
> > On Sun, Feb 24, 2019 at 2:02 PM Reuven Lax  <mailto:re...@google.com>
> > <mailto:re...@google.com <mailto:re...@google.com>>> wrote:
> >
>     >
>     >     Hi JB. Good to hear from you!
> >
> >     What version of Python do you have installed?
> >
> >     Reuven
> >
> >     On Sun, Feb 24, 2019 at 11:28 AM Jean-Baptiste Onofré
> >     mailto:j...@nanthrax.net>
> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
> >
> >         Hi guys,
> >
> >         As said couple of weeks ago, I'm happy to be back on the
> >         project. I'm
> >         resuming several PRs and improvements, including work on Spark
> >         runner
> >         with "my" guys (Etienne and Alexey especially).
> >
> >         I have couple of issues while building master, both on the
> >         Python SDK:
> >
> >         1. The first issue is that virtualenv doesn't work on my
> box. It
> >         fails with:
> >
> >         Process 'command 'virtualenv'' finished with non-zero exit
> value 3
> >
> >         2. It's maybe a consequence of 1, but I also see RAT
> failures on
> >         Python SDK:
> >
> >         Unapproved/unknown license:
> >         sdks/python/apache-beam-2.12.0.dev0/PKG-INFO
> >         Unapproved/unknown license:
> >         sdks/python/apache-beam-2.12.0.dev0/setup.cfg
> >
> >         Do you have the same issue ? In the mean time, I'm checking on
> >         Jenkins
> >         and the environment.
> >
> >         Regards
>     >         JB
> >         --
> >         Jean-Baptiste Onofré
> >         jbono...@apache.org <mailto:jbono...@apache.org>
> <mailto:jbono...@apache.org <mailto:jbono...@apache.org>>
> >         http://blog.nanthrax.net
> >         Talend - http://www.talend.com
> >
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Issue building master

[Jean-Baptiste Onofré]

Hi Kenn,

I started with a git clean -fdx, so it's maybe the build which created
those files.

Regards
JB

On 25/02/2019 00:16, Kenneth Knowles wrote:
> To #2 it looks like it has done an install relative to the Python SDK
> directory. The files are not set up to be ignored by git/RAT.
> 
> Kenn
> 
> On Sun, Feb 24, 2019 at 2:02 PM Reuven Lax  <mailto:re...@google.com>> wrote:
> 
> 
> Hi JB. Good to hear from you!
> 
> What version of Python do you have installed?
> 
> Reuven
> 
> On Sun, Feb 24, 2019 at 11:28 AM Jean-Baptiste Onofré
> mailto:j...@nanthrax.net>> wrote:
> 
> Hi guys,
> 
> As said couple of weeks ago, I'm happy to be back on the
> project. I'm
> resuming several PRs and improvements, including work on Spark
> runner
> with "my" guys (Etienne and Alexey especially).
> 
> I have couple of issues while building master, both on the
> Python SDK:
> 
> 1. The first issue is that virtualenv doesn't work on my box. It
> fails with:
> 
> Process 'command 'virtualenv'' finished with non-zero exit value 3
> 
> 2. It's maybe a consequence of 1, but I also see RAT failures on
> Python SDK:
> 
> Unapproved/unknown license:
> sdks/python/apache-beam-2.12.0.dev0/PKG-INFO
> Unapproved/unknown license:
> sdks/python/apache-beam-2.12.0.dev0/setup.cfg
> 
> Do you have the same issue ? In the mean time, I'm checking on
> Jenkins
> and the environment.
> 
> Regards
> JB
> -- 
>     Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.11.0, release candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Java SDK tested with beam samples.

Regards
JB

On 22/02/2019 09:50, Ahmet Altay wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #1 for the version
> 2.11.0, as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint 64B84A5AD91F9C20F5E9D9A7D62E71416096FA00 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.11.0-RC1" [5],
> * website pull request listing the release [6] and publishing the API
> reference manual [7].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
> * Validation sheet with a tab for 2.11.0 release to help with validation
> [8].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Ahmet
> 
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344775
> [2] https://dist.apache.org/repos/dist/dev/beam/2.11.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1061/
> [5] https://github.com/apache/beam/tree/v2.11.0-RC1
> [6] https://github.com/apache/beam/pull/7924
> [7] https://github.com/apache/beam-site/pull/587
> [8] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=542393513
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Issue building master

[Jean-Baptiste Onofré]

Hi Reuven,

I tried with Python 2.7.x. Let me try with Python 3.

Regards
JB

On 24/02/2019 23:02, Reuven Lax wrote:
> 
> Hi JB. Good to hear from you!
> 
> What version of Python do you have installed?
> 
> Reuven
> 
> On Sun, Feb 24, 2019 at 11:28 AM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi guys,
> 
> As said couple of weeks ago, I'm happy to be back on the project. I'm
> resuming several PRs and improvements, including work on Spark runner
> with "my" guys (Etienne and Alexey especially).
> 
> I have couple of issues while building master, both on the Python SDK:
> 
> 1. The first issue is that virtualenv doesn't work on my box. It
> fails with:
> 
> Process 'command 'virtualenv'' finished with non-zero exit value 3
> 
> 2. It's maybe a consequence of 1, but I also see RAT failures on
> Python SDK:
> 
> Unapproved/unknown license: sdks/python/apache-beam-2.12.0.dev0/PKG-INFO
> Unapproved/unknown license:
> sdks/python/apache-beam-2.12.0.dev0/setup.cfg
> 
> Do you have the same issue ? In the mean time, I'm checking on Jenkins
> and the environment.
> 
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Issue building master

[Jean-Baptiste Onofré]

Hi guys,

As said couple of weeks ago, I'm happy to be back on the project. I'm
resuming several PRs and improvements, including work on Spark runner
with "my" guys (Etienne and Alexey especially).

I have couple of issues while building master, both on the Python SDK:

1. The first issue is that virtualenv doesn't work on my box. It fails with:

Process 'command 'virtualenv'' finished with non-zero exit value 3

2. It's maybe a consequence of 1, but I also see RAT failures on Python SDK:

Unapproved/unknown license: sdks/python/apache-beam-2.12.0.dev0/PKG-INFO
Unapproved/unknown license: sdks/python/apache-beam-2.12.0.dev0/setup.cfg

Do you have the same issue ? In the mean time, I'm checking on Jenkins
and the environment.

Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [DISCUSS] Should File based IOs implement readAll() or just readFiles()

[Jean-Baptiste Onofré]

+1

Thanks for that Ismaël.

Regards
JB

On 06/02/2019 11:24, Ismaël Mejía wrote:
> Since it seems we have consensus on deprecating both transforms I created
> 
> BEAM-6605 Deprecate TextIO.readAll() and TextIO.ReadAll transform
> BEAM-6606 Deprecate AvroIO.readAll() and AvroIO.ReadAll transform
> 
> Thanks everyone.
> 
> On Fri, Feb 1, 2019 at 7:03 PM Chamikara Jayalath  
> wrote:
>>
>> Python SDK doesn't have FileIO yet so let's keep ReadAllFromFoo transforms 
>> currently available for various file types around till we have that.
>>
>> Thanks,
>> Cham
>>
>> On Fri, Feb 1, 2019 at 7:41 AM Jean-Baptiste Onofré  
>> wrote:
>>>
>>> Hi,
>>>
>>> readFiles() should be used IMHO. We should remove readAll() to avoid
>>> confusion.
>>>
>>> Regards
>>> JB
>>>
>>> On 30/01/2019 17:25, Ismaël Mejía wrote:
>>>> Hello,
>>>>
>>>> A ‘recent’ pattern of use in Beam is to have in file based IOs a
>>>> `readAll()` implementation that basically matches a `PCollection` of
>>>> file patterns and reads them, e.g. `TextIO`, `AvroIO`. `ReadAll` is
>>>> implemented by a expand function that matches files with FileIO and
>>>> then reads them using a format specific `ReadFiles` transform e.g.
>>>> TextIO.ReadFiles, AvroIO.ReadFiles. So in the end `ReadAll` in the
>>>> Java implementation is just an user friendly API to hide FileIO.match
>>>> + ReadFiles.
>>>>
>>>> Most recent IOs do NOT implement ReadAll to encourage the more
>>>> composable approach of File + ReadFiles, e.g. XmlIO and ParquetIO.
>>>>
>>>> Implementing ReadAll as a wrapper is relatively easy and is definitely
>>>> user friendly, but it has an  issue, it may be error-prone and it adds
>>>> more code to maintain (mostly ‘repeated’ code). However `readAll` is a
>>>> more abstract pattern that applies not only to File based IOs so it
>>>> makes sense for example in other transforms that map a `Pcollection`
>>>> of read requests and is the basis for SDF composable style APIs like
>>>> the recent `HBaseIO.readAll()`.
>>>>
>>>> So the question is should we:
>>>>
>>>> [1] Implement `readAll` in all file based IOs to be user friendly and
>>>> assume the (minor) maintenance cost
>>>>
>>>> or
>>>>
>>>> [2] Deprecate `readAll` from file based IOs and encourage users to use
>>>> FileIO + `readFiles` (less maintenance and encourage composition).
>>>>
>>>> I just checked quickly in the python code base but I did not find if
>>>> the File match + ReadFiles pattern applies, but it would be nice to
>>>> see what the python guys think on this too.
>>>>
>>>> This discussion comes from a recent slack conversation with Łukasz
>>>> Gajowy, and we wanted to settle into one approach to make the IO
>>>> signatures consistent, so any opinions/preferences?
>>>>
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.10.0, release candidate #2

[Jean-Baptiste Onofré]

+1 (binding)

Quickly tested on beam-samples.

Regards
JB

On 05/02/2019 23:57, Kenneth Knowles wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #2 for the
> version 2.10.0, as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org/> [2], which is signed with the key with
> fingerprint 6ED551A8AE02461C [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.10.0-RC1" [5],
> * website pull request listing the release [6] and publishing the API
> reference manual [7].
> * Python artifacts are deployed along with the source release to
> the dist.apache.org <http://dist.apache.org/> [2].
> * Validation sheet with a tab for 2.10.0 release to help with validation
> [7].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Kenn
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12344540
> [2] https://dist.apache.org/repos/dist/dev/beam/2.10.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> <https://dist.apache.org/repos/dist/release/beam/KEYS>
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1057/
> [5] https://github.com/apache/beam/tree/v2.10.0-RC2
> [6] https://github.com/apache/beam/pull/7651/files
> [7] https://github.com/apache/beam-site/pull/586
> [8] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2053422529

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [DISCUSS] Should File based IOs implement readAll() or just readFiles()

[Jean-Baptiste Onofré]

Hi,

readFiles() should be used IMHO. We should remove readAll() to avoid
confusion.

Regards
JB

On 30/01/2019 17:25, Ismaël Mejía wrote:
> Hello,
> 
> A ‘recent’ pattern of use in Beam is to have in file based IOs a
> `readAll()` implementation that basically matches a `PCollection` of
> file patterns and reads them, e.g. `TextIO`, `AvroIO`. `ReadAll` is
> implemented by a expand function that matches files with FileIO and
> then reads them using a format specific `ReadFiles` transform e.g.
> TextIO.ReadFiles, AvroIO.ReadFiles. So in the end `ReadAll` in the
> Java implementation is just an user friendly API to hide FileIO.match
> + ReadFiles.
> 
> Most recent IOs do NOT implement ReadAll to encourage the more
> composable approach of File + ReadFiles, e.g. XmlIO and ParquetIO.
> 
> Implementing ReadAll as a wrapper is relatively easy and is definitely
> user friendly, but it has an  issue, it may be error-prone and it adds
> more code to maintain (mostly ‘repeated’ code). However `readAll` is a
> more abstract pattern that applies not only to File based IOs so it
> makes sense for example in other transforms that map a `Pcollection`
> of read requests and is the basis for SDF composable style APIs like
> the recent `HBaseIO.readAll()`.
> 
> So the question is should we:
> 
> [1] Implement `readAll` in all file based IOs to be user friendly and
> assume the (minor) maintenance cost
> 
> or
> 
> [2] Deprecate `readAll` from file based IOs and encourage users to use
> FileIO + `readFiles` (less maintenance and encourage composition).
> 
> I just checked quickly in the python code base but I did not find if
> the File match + ReadFiles pattern applies, but it would be nice to
> see what the python guys think on this too.
> 
> This discussion comes from a recent slack conversation with Łukasz
> Gajowy, and we wanted to settle into one approach to make the IO
> signatures consistent, so any opinions/preferences?
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: BEAM-6324 / #7340: "I've pretty much given up on the PR being merged. I use my own fork for my projects"

[Jean-Baptiste Onofré]

I agree, the problem is not on the guideline but more on the behavior.

Regards
JB

On 26/01/2019 16:37, Thomas Weise wrote:
> In this case the guidelines are not the issue, rather there may be a
> shortage of review attention. Considering Beam is one of the top
> projects by activity/commits, we may want to check why we have this
> problem. Is it perhaps that committers are too focused on their own PRs
> rather than review of non-committer contributions? Perhaps we should
> improve identifying and prioritizing PRs that come from new contributors
> or occasional contributors since that is needed to grow the community?
> The overall goal should probably be to balance the few full time, high
> frequency contributors/committers with the much larger potential pool of
> occasional contributors (some of which contribute in their spare time).
> 
> Back to the guidelines: I currently don't see as much of a problem with
> the guidelines themselves, but with how they are applied.. I think that
> above mentioned high frequency committers should stick very close to the
> guidelines to make contributing to Beam more feasible to the community
> at large. I have seen few others comment on this as well, but recently
> it has been painful to rely on the master branch. Frequent build
> breakages or regressions make me want to sync less frequently, because I
> never know what surprise a pull might bring (time sink to resolve issues).
> 
> Thomas
> 
> 
> On Fri, Jan 25, 2019 at 10:34 PM Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Agree.
> 
> Unfortunately, I'm not so surprise about this feedback, even if no
> review at all is pretty rare. We improved a lot about the PR review, but
> it's still too long IMHO. As I said several times, IMHO, if a PR is
> basically good and the build pass, it should be merged.
> We do a lot of discussion in PR, the build is not easy (sorry, but
> gradle just sucks at least on my box), we should give more chance to
> people to really participate (sometime, we just do things straight
> forward).
> 
> I'm taking my hat as someone who started to lose motivation on Beam. I'm
> forcing myself to come back more active on the project, but it's hard:
> lot of messages/discussion, feeling that some parts can't be changed,
> that we are struggling and wasting time on non relevant discussion or
> already define by Apache policies, etc.
> 
> So, at some degree, I understand such contributor feedback.
> 
> Regards
> JB
> 
> On 25/01/2019 20:03, Kenneth Knowles wrote:
> > Yea, that guide oscillates in size because of the tension between (1)
> > being short enough that someone actually reads it and can get a
> > birds-eye view of what they have to do and (2) providing all the info
> > needed in case they don't already know what to do. It is already
> pretty
> > short, and most of the technical details are off on the wiki.
> >
> > Kenn
> >
> > On Fri, Jan 25, 2019 at 10:21 AM Rui Wang  <mailto:ruw...@google.com>
> > <mailto:ruw...@google.com <mailto:ruw...@google.com>>> wrote:
> >
> >     We have code contribution guidelines [1] and it says useful
> tips to
> >     make PR reviewed and merged. But I guess it hides in Beam
> website so
> >     new contributors are likely to ignore it. In order to make the
> >     guidance easy to find and read for new contributors, we
> probably can
> >
> >     a. Move number 5 item from [1] to a separate section and name it
> >     "Tips to get your PR reviewed and merged"
> >     b. Put the link to the Github pull request template, so when a
> >     contributor creates the first PR, the contributor could see
> the link
> >     (or even paste text from contribution guide). It will be a good
> >     chance that new contributors read what's in pull request template.
> >
> >
> >     -Rui
> >
> >     [1] https://beam.apache.org/contribute/#make-your-change
> >
> >     On Fri, Jan 25, 2019 at 9:24 AM Alexey Romanenko
> >     mailto:aromanenko@gmail.com>
> <mailto:aromanenko@gmail.com <mailto:aromanenko@gmail.com>>>
> wrote:
> >
> >         For sure, it’s a pity that this PR has not been addressed
> for a
> >         long time (I guess, we probably have other ones like this)
> but,
> >         as I can see from this PR history, review has not been
> requested
> >        

Re: [DISCUSSION] UTests and embedded backends

[Jean-Baptiste Onofré]

Hi,

it makes sense to use embedded backend when:

1. it's possible to easily embed the backend
2. when the backend is "predictable".

If it's easy to embed and the backend behavior is predictable, then it
makes sense.
In other cases, we can fallback to mock.

Regards
JB

On 21/01/2019 10:07, Etienne Chauchot wrote:
> Hi guys,
> 
> Lately I have been fixing various Elasticsearch flakiness issues in the
> UTests by: introducing timeouts, countdown latches, force refresh,
> embedded cluster size decrease ...
> 
> These flakiness issues are due to the embedded Elasticsearch not coping
> well with the jenkins overload. Still, IMHO I believe that having
> embedded backend for UTests are a lot better than mocks. Even if they
> are less tolerant to load, I prefer having UTests 100% representative of
> real backend and add countermeasures to protect against jenkins overload.
> 
> WDYT ?
> 
> Etienne
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Connection leaks with PostgreSQL instance

[Jean-Baptiste Onofré]

Hi,

I don't think we have connection leak in normal behavior.

The actual SQL statement is executed in @FinishBundle, where the
connection is closed.

The processElement adds record to process.

Does it mean that an Exception occurs in the batch addition ?

Regards
JB

On 17/01/2019 12:41, Alexey Romanenko wrote:
> Kenn,
> 
> I’m not sure that we have a connection leak in JdbcIO since new
> connection is being obtained from an instance of /javax.sql.DataSource/
> (created in @Setup) and which
> is /org.apache.commons.dbcp2.BasicDataSource/ by default.
> /BasicDataSource/ uses connection pool and closes all idle connections
> in "close()”. 
> 
> In its turn, JdbcIO calls/DataSource.close()/ in @Teardown, so all idle
> connections should be closed and released there in case of fails.
> Though, potentially some connections, that has been delegated to client
> before and were not not properly returned to pool, could be leaked…
> Anyway, I think it could be a good idea to call "/connection.close()/”
> (return to connection pool) explicitly in case of any exception happened
> during bundle processing.
> 
> Probably JB may provide more details as original author of JdbcIO.
> 
>> On 14 Jan 2019, at 21:37, Kenneth Knowles > <mailto:k...@apache.org>> wrote:
>>
>> Hi Jonathan,
>>
>> JdbcIO.write() just invokes this
>> DoFn: 
>> https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L765
>>
>> It establishes a connection in @StartBundle and then in @FinishBundle
>> it commits a batch and closes the connection. If an error happens
>> in @StartBundle or @ProcessElement there will be a retry with a fresh
>> instance of the DoFn, which will establish a new connection. It looks
>> like neither @StartBundle nor @ProcessElement closes the connection,
>> so I'm guessing that the old connection sticks around because the
>> worker process was not terminated. So the Beam runner and Dataflow
>> service are working as intended and this is an issue with JdbcIO,
>> unless I've made a mistake in my reading or analysis.
>>
>> Would you mind reporting these details
>> to https://issues.apache.org/jira/projects/BEAM/ ?
>>
>> Kenn
>>
>> On Mon, Jan 14, 2019 at 12:51 AM Jonathan Perron
>> mailto:jonathan.per...@lumapps.com>> wrote:
>>
>> Hello !
>>
>> My question is maybe mainly GCP-oriented, so I apologize if it is
>> not fully related to the Beam community.
>>
>> We have a streaming pipeline running on Dataflow which writes data
>> to a PostgreSQL instance hosted on Cloud SQL. This database is
>> suffering from connection leak spikes on a regular basis:
>>
>> 
>>
>> The connections are kept alive until the pipeline is canceled/drained:
>>
>> 
>>
>> We are writing to the database with:
>>
>> - individual DoFn where we open/close the connection using the
>> standard JDBC try/catch (SQLException ex)/finally statements;
>>
>> - a Pipeline.apply(JdbcIO.write()) operations.
>>
>> I observed that these spikes happens most of the time after I/O
>> errors with the database. Has anyone observed the same situation ?
>>
>> I have several questions/observations, please correct me if I am
>> wrong (I am not from the java environment, so some can seem pretty
>> trivial) :
>>
>> - Does the apply method handles SQLException or I/O errors ?
>>
>> - Would the use of a connection pool prevents such behaviours ? If
>> so, how would one implement it to allow all workers to use it ?
>> Could it be implemented with JDBC Connection pooling ?
>>
>> I am worrying about the serialization if one would pass a
>> Connection item as an argument of a DoFn.
>>
>> Thank you in advance for your comments and reactions.
>>
>> Best regards,
>>
>> Jonathan
>>
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Enforce javadoc comments in public methods?

[Jean-Baptiste Onofré]

Hi,

for the presence of a comment on public method, it's a good idea. Now,
about the number of lines, not sure it's a good idea. I'm thinking about
the getter/setter which are public. Most of the time, the comment is
pretty simple (and useless ;)).

Regards
JB

On 07/01/2019 04:35, Ruoyun Huang wrote:
> Hi, everyone,
> 
> 
> We were wondering whether it is a good idea to make checkstyle
> enforce public method comments. Our current behavior of JavaDoc check is:
> 
>  1.
> 
> Missing Class javadoc comment is reported as error.
> 
>  2.
> 
> Method comment missing is explicitly allowed. see [1].  It is not
> even shown as warning.
> 
>  3.
> 
> The actual javadoc target gives warning when certain tags are
> missing in javadoc, but not if the whole comment is missing.
> 
> 
>    How about we enforce method comments for **1) public method and 2)
> method that is longer than N lines**. (N=~30 seems a good number,
> leading to ~50 violations in current repository). I can find out the
> corresponding contributors to fill in the missing comments, before we
> turning the check fully on.
> 
> 
>    One caveat though is that we might want skip this check on test code,
> but I am not sure yet if our current setup can easily handle separated
> rules for main code versus test code.
> 
> 
> Is this a good idea?  Thoughts and suggestions?  
> 
> 
> [1]
> https://github.com/apache/beam/blame/5ceffb246c0c38ad68dd208e951a1f39c90ef85c/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml#L111
> 
> 
> Cheers, 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Prepare Beam 2.10.0 release

[Jean-Baptiste Onofré]

It sounds good to me.

Regards
JB

On 02/01/2019 16:16, Kenneth Knowles wrote:
> Hi All,
> 
> According to the release calendar [1] branch cut date for
> Beam 2.10.0 release is today, 2019 January 2. I'd like to volunteer to
> manage this release. Does anyone have any reason we should not release
> on schedule?
> 
> Otherwise, if you know of release-blocking bugs, please mark their "Fix
> Version" as 2.10.0 and they will show up in the burndown [2]. If you own
> a bug currently in the burndown, please double-check if it is truly
> release-blocking.
> 
> I've gone ahead and cut a release-2.10.0 branch from the current master,
> since it is green. If it turns out to be an inauspicious starting point,
> we can always reset it.
> 
> Kenn
> 
> [1] 
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
> [2] https://issues.apache.org/jira/projects/BEAM/versions/12344540

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.9.0, release candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

Le 13 déc. 2018 à 20:11, à 20:11, Reuven Lax  a écrit:
>+1 (binding)
>
>On Thu, Dec 13, 2018 at 8:39 AM Kenneth Knowles 
>wrote:
>
>> +1 (binding)
>>
>> A new feature request
>(https://issues.apache.org/jira/browse/BEAM-6212)
>> had been filed against 2.9.0 release (
>> https://issues.apache.org/jira/projects/BEAM/versions/12344258). I
>moved
>> it to 2.10.0.
>>
>> I additionally built [some targets in] the source release. The
>website
>> build does not work since it apparently depends on having a git repo
>> defined. We should fix that but no reason to block the release.
>>
>> Kenn
>>
>> On Wed, Dec 12, 2018 at 4:54 PM Andrew Pilloud 
>> wrote:
>>
>>> +1
>>>
>>> Turns out we broke DOUBLE on purpose. Updated the demo to use
>DECIMAL and
>>> it doesn't hard fail. This is a docs bug.
>>>
>>> On Wed, Dec 12, 2018 at 3:55 PM Scott Wegner 
>wrote:
>>>
 +1

 I verified the Java examples succeed on DirectRunner.

 On Wed, Dec 12, 2018 at 3:30 PM Chamikara Jayalath
>
 wrote:

> Thanks Andrew. Please make this a blocker and -1 the thread if you
> think we need a new RC.
>
> - Cham
>
> On Wed, Dec 12, 2018 at 3:27 PM Andrew Pilloud
>
> wrote:
>
>> I was just running the Beam SQL demo. I found one query fails
>with
>> "the keyCoder of a GroupByKey must be deterministic" and another
>just
>> hangs. I opened an issue:
>> https://issues.apache.org/jira/browse/BEAM-6224 Not sure if this
>> calls for canceling the release or just a release note (SQL is
>still
>> experimental). I'm continuing to track down the root cause, but
>am tied up
>> with the Beam Meetup in SFO today.
>>
>> Andrew
>>
>> On Tue, Dec 11, 2018 at 3:32 PM Ruoyun Huang 
>> wrote:
>>
>>> +1,  Looking forward to the release!
>>>
>>> On Tue, Dec 11, 2018 at 11:09 AM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 Hi All,

 I ran Beam RC verification script [1] and updated the
>validation
 spreadsheet [2]. I think the current release candidate looks
>good.

 So +1 for the release.

 Thanks,
 Cham

 [1]

>https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh
 [2]

>https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2053422529

 On Fri, Dec 7, 2018 at 7:19 AM Ismaël Mejía 
 wrote:

> Looking at the dates on the Spark runner git log there was a
>PR
> merged to change Spark translation from classes to URNs. I
>cannot see how
> this can impact performance. Looking at the other queries in
>the
> dashboards, there seems to be a great variability in the
>executions of the
> Spark runner to the point of feeling we don't have guarantees
>anymore. I
> wonder if this was because of other loads shared in the
>server(s), or
> because our sample is too small for the standard deviation.
>
> I would proceed with the release, the real question is if we
>can
> somehow constraint the execution of this tests to have a more
>consistent
> output.
>
>
> On Fri, Dec 7, 2018 at 4:10 PM Etienne Chauchot <
> echauc...@apache.org> wrote:
>
>> Hi all,
>> Regarding query7 in spark:
>> - there doesn't seem to be a functional regression: query
>passes
>> and output size is still the same
>>
>> - Also the performance degradation seems to be only on spark,
>the
>> other runners do not seem to suffer from it.
>>
>> - performance degradation seems to be constant from 11/12 so
>we
>> can eliminate temporary load on the jenkins server that would
>generate
>> delays in Max transform.
>>
>> => query7 uses Max transform, fanout and side inputs, has one
>of
>> these parts recently (11/12/18) changed in spark?
>>
>> Etienne
>>
>> Le jeudi 06 décembre 2018 à 21:32 -0800, Chamikara Jayalath a
>> écrit :
>>
>> Udi or anybody else who is familiar about Nexmark,  please -1
>the
>> vote thread if you think this particular performance
>regression for
>> Spark/Direct runners is a blocker. Otherwise I think we can
>continue the
>> vote.
>>
>> Thanks,
>> Cham
>>
>> On Thu, Dec 6, 2018 at 6:19 PM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>> Are either of these regressions due to known issues ? If not
>> should they be considered release blockers ?
>>
>> Thanks,
>> Cham
>>
>> On Thu, Dec 6, 2018 at 6:11 PM Udi Meiri 
>wrote:
>>
>> For DirectRunner there are regressions in 

Re: contributor in the Beam

[Jean-Baptiste Onofré]

Can you please fix the conflict in the PR ?

Thanks
Regards
JB

On 03/12/2018 08:52, Chaim Turkel wrote:
> it looks like there was a failure that is not due to the code, how can
> i continue the process?
> https://github.com/apache/beam/pull/7162
> 
> On Thu, Nov 29, 2018 at 9:15 PM Chaim Turkel  wrote:
>>
>> hi,
>>   i added another pr for the case of a self signed certificate ssl on
>> the mongodb server
>>
>> https://github.com/apache/beam/pull/7162
>> On Wed, Nov 28, 2018 at 5:16 PM Jean-Baptiste Onofré  
>> wrote:
>>>
>>> Hi,
>>>
>>> I already upgraded locally. Let me push the PR.
>>>
>>> Regards
>>> JB
>>>
>>> On 28/11/2018 16:02, Chaim Turkel wrote:
>>>> is there any reason that the mongo client version is still on 3.2.2?
>>>> can you upgrade it to 3.9.0?
>>>> chaim
>>>> On Tue, Nov 27, 2018 at 4:48 PM Jean-Baptiste Onofré  
>>>> wrote:
>>>>>
>>>>> Hi Chaim,
>>>>>
>>>>> The best is to create a Jira describing the new features you want to
>>>>> add. Then, you can create a PR related to this Jira.
>>>>>
>>>>> As I'm the original MongoDbIO author, I would be more than happy to help
>>>>> you and review the PR.
>>>>>
>>>>> Thanks !
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 27/11/2018 15:37, Chaim Turkel wrote:
>>>>>> Hi,
>>>>>>   I have added a few features to the MongoDbIO and would like to add
>>>>>> them to the project.
>>>>>> I have read https://beam.apache.org/contribute/
>>>>>> I have added a jira user, what do i need to do next?
>>>>>>
>>>>>> chaim
>>>>>>
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbono...@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [DISCUSS] Structuring Java based DSLs

[Jean-Baptiste Onofré]

Hi,

it sounds good to me.

Regards
JB

On 30/11/2018 15:29, Jan Lukavský wrote:
> Hi community,
> 
> I'm part of Euphoria DSL team, and on behalf of this team, I'd like to
> discuss possible development of Java based DSLs currently present in
> Beam. In my knowledge, there are currently two DSLs based on Java SDK -
> Euphoria and SQL. These DSLs currently share only the SDK itself,
> although there might be room to share some more effort. We already know
> that both Euphoria and SQL have need for retractions, but there are
> probably many more features that these two could share.
> 
> So, I'd like to open a discussion on what it would cost and what it
> would possibly bring, if instead of the current structure
> 
>   Java SDK
> 
>     |  SQL
> 
>     |  Euphoria
> 
> these DSLs would be structured as
> 
>   Java SDK ---> Euphoria ---> SQL
> 
> I'm absolutely sure that this would be a great investment and a huge
> change, but I'd like to gather some opinions and general feelings of the
> community about this. Some points to start the discussion from my side
> would be, that structuring DSLs like this has internal logical
> consistency, because each API layer further narrows completeness, but
> brings simpler API for simpler tasks, while adding additional high-level
> view of the data processing pipeline and thus enabling more
> optimizations. On Euphoria side, these are various implementations joins
> (most effective implementation depends on data), pipeline sampling and
> more. Some (or maybe most) of these optimizations would have to be
> implemented in both DSLs, so implementing them once is beneficial.
> Another benefit is that this would bring Euphoria "closer" to Beam core
> development (which would be good, it is part of the project anyway,
> right? :)) and help better drive features, that although currently
> needed mostly by SQL, might be needed by other Java users anyway.
> 
> Thanks for discussion and looking forward to any opinions.
> 
>   Jan
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Stand at FOSDEM 2019

[Jean-Baptiste Onofré]

Hi Max,

it sounds good to me.

Regards
JB

On 29/11/2018 12:06, Maximilian Michels wrote:
> Hi,
> 
> For everyone who might be attending FOSDEM19: What do you think about
> taking a slot for Beam at the Apache stand?
> 
> A slot is 2-3 hours. It is a great way to spread the word about Beam. We
> wouldn't have to prepare much, just bring some merch.
> 
> There is still plenty of space:
> https://cwiki.apache.org/confluence/display/COMDEV/FOSDEM+2019
> 
> Cheers,
> Max
> 
> PS: FOSDEM is an open-source conference in Brussels, Feb 2-3, 2019

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: contributor in the Beam

[Jean-Baptiste Onofré]

Hi,

I already upgraded locally. Let me push the PR.

Regards
JB

On 28/11/2018 16:02, Chaim Turkel wrote:
> is there any reason that the mongo client version is still on 3.2.2?
> can you upgrade it to 3.9.0?
> chaim
> On Tue, Nov 27, 2018 at 4:48 PM Jean-Baptiste Onofré  
> wrote:
>>
>> Hi Chaim,
>>
>> The best is to create a Jira describing the new features you want to
>> add. Then, you can create a PR related to this Jira.
>>
>> As I'm the original MongoDbIO author, I would be more than happy to help
>> you and review the PR.
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 27/11/2018 15:37, Chaim Turkel wrote:
>>> Hi,
>>>   I have added a few features to the MongoDbIO and would like to add
>>> them to the project.
>>> I have read https://beam.apache.org/contribute/
>>> I have added a jira user, what do i need to do next?
>>>
>>> chaim
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Build fail on Python SDK

[Jean-Baptiste Onofré]

Thanks,

let me run a new build with -scan option.

I keep you posted.

Regards
JB

On 27/11/2018 18:46, Ahmet Altay wrote:
> I have not seen this one before. Could you share the link to gradle
> build scan, perhaps that will have more information?
> 
> On Tue, Nov 27, 2018 at 6:42 AM, Jean-Baptiste Onofré  <mailto:j...@nanthrax.net>> wrote:
> 
> Hi guys
> 
> I have some tests failures on Python SDK (19 failures exactly).
> 
> They all look the same, for instance:
> 
> FAIL: testIndexing
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest)
> --
> Traceback (most recent call last):
>   File
> 
> "/home/jbonofre/Workspace/beam/sdks/python/apache_beam/typehints/trivial_inference_test.py",
> line 41, in testIndexing
>     self.assertReturnType(int, lambda x: x[0], [typehints.Tuple[int,
> str]])
>   File
> 
> "/home/jbonofre/Workspace/beam/sdks/python/apache_beam/typehints/trivial_inference_test.py",
> line 35, in assertReturnType
>     self.assertEquals(expected, trivial_inference.infer_return_type(f,
> inputs))
> AssertionError:  != Any
> 
> 
> Does someone else have the same issue ?
> 
> Thanks,
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
>     jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: MongoDbIO

[Jean-Baptiste Onofré]

Hi Chaim,

do you mean that MongoDbIO could provided a PCollection
containing the payload + id + time ?

Regards
JB

On 27/11/2018 12:47, Chaim Turkel wrote:
> Hi,
>   I would like to write a sync to validate that i have all records
> from mongo in my bigquery.
> to do this i would like to bring the fields id,time from mongo to
> biguqery, and only on the missing docuements to read the full
> document,
> I did not see a way to bring a paritial document?
> 
> chaim
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: contributor in the Beam

[Jean-Baptiste Onofré]

Hi Chaim,

The best is to create a Jira describing the new features you want to
add. Then, you can create a PR related to this Jira.

As I'm the original MongoDbIO author, I would be more than happy to help
you and review the PR.

Thanks !
Regards
JB

On 27/11/2018 15:37, Chaim Turkel wrote:
> Hi,
>   I have added a few features to the MongoDbIO and would like to add
> them to the project.
> I have read https://beam.apache.org/contribute/
> I have added a jira user, what do i need to do next?
> 
> chaim
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Build fail on Python SDK

[Jean-Baptiste Onofré]

Hi guys

I have some tests failures on Python SDK (19 failures exactly).

They all look the same, for instance:

FAIL: testIndexing
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest)
--
Traceback (most recent call last):
  File
"/home/jbonofre/Workspace/beam/sdks/python/apache_beam/typehints/trivial_inference_test.py",
line 41, in testIndexing
self.assertReturnType(int, lambda x: x[0], [typehints.Tuple[int, str]])
  File
"/home/jbonofre/Workspace/beam/sdks/python/apache_beam/typehints/trivial_inference_test.py",
line 35, in assertReturnType
self.assertEquals(expected, trivial_inference.infer_return_type(f,
inputs))
AssertionError:  != Any


Does someone else have the same issue ?

Thanks,
Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Reading CSV from google cloud storage to Data Flow

[Jean-Baptiste Onofré]

Hi Unais,

What SDK do you plan to use ? Java or Python ?

Regarding Java, I would use directly TextIO.

Regards
JB

On 25/11/2018 13:09, Unais T wrote:
> Hey guys,
> 
> One doubt 
> 
> I want to read a csv file from google cloud storage to Data Flow
> which is best method?
> 
> 1.   Read csv and sync to BQ and then use BigQuerySource method
> 2.   Read from cloud storage directly to Data Flow (Is there any source
> method for csv from cloud storage to CSV - like `ReadFromText` )
> 
> Whats the best way to read csv from cloud storage to Data Flow?

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

JB's back

[Jean-Baptiste Onofré]

Hi guys,

Sorry to have been quiet recently.

After some rushy things and having been sick last week, I'm back on Beam.

With Alexey, Etienne and Ismaël, we are working on the Spark runner and
I'm also resuming several works I was holding on the IOs.

Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Bay Area Apache Beam Kickoff!

[Jean-Baptiste Onofré]

Nice !!

Unfortunately I won't be able to be there. But good luck and I'm sure it
will be a great meetup !

Regards
JB

On 20/11/2018 02:36, Austin Bennett wrote:
> We have our first meetup scheduled for December 12th in San Francisco.  
> 
> Andrew Pilloud, a software engineer at Google and Beam committer, will
> demo the latest feature in Beam SQL: a standalone SQL shell. The talk
> cover why SQL is a good fit for streaming data processing, the technical
> details of the Beam SQL engine, and a peek into our future plans.
> 
> Kenn Knowles, a founding PMC Member and incoming PMC Chair for the
> Apache Beam project, as well as computer scientist and engineer at
> Google will share about all things Beam. Where it is, where its been,
> where its going.
> 
> More info:
>  https://www.meetup.com/San-Francisco-Apache-Beam/events/256348972/
> 
> For those in/around town (or that can be) come join in the fun!  
> 
> 
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] Prepare Beam 2.9.0 release

[Jean-Baptiste Onofré]

Hi Cham,

it sounds good to me.

I'm resuming some works on IOs but nothing blocker.

Regards
JB

On 21/11/2018 03:59, Chamikara Jayalath wrote:
> Hi All,
> 
> Looks like there are three blockers in the burndown list but they are
> actively being worked on.
> 
> If there's no objection I'll create the release branch tomorrow morning.
> We can cherry-pick fixes to the blockers before building the first RC
> hopefully on Monday.
> 
> Thanks,
> Cham
> 
> 
> On Sat, Nov 17, 2018 at 10:38 AM Kenneth Knowles  <mailto:k...@apache.org>> wrote:
> 
> 
> 
> On Fri, Nov 16, 2018, 18:25 Thomas Weise  <mailto:t...@apache.org> wrote:
> 
> 
> 
> On Thu, Nov 15, 2018 at 10:53 PM Charles Chen  <mailto:c...@google.com>> wrote:
> 
> +1
> 
> Note that we need to temporarily revert
> https://github.com/apache/beam/pull/6683 before the release
> branch cut per the discussion
> at 
> https://lists.apache.org/thread.html/78fe33dc41b04886f5355d66d50359265bfa2985580bb70f79c53545@%3Cdev.beam.apache.org%3E
> 
> 
> Since it is for the release I would prefer this to be done in
> the release branch, and not in master.
> 
> 
> +1 do it (and all remaining burndown) on the release branch.
> Tracking Jira targeting 2.9.0?
> 
> Kenn
> 
> 
> 
> 
> 
> 
> 
>  
> 
> On Thu, Nov 15, 2018 at 9:18 PM Tim
>  <mailto:timrobertson...@gmail.com>> wrote:
> 
> Thanks Cham
> +1
> 
> On 16 Nov 2018, at 05:30, Thomas Weise  <mailto:t...@apache.org>> wrote:
> 
>> +1
>>
>>
>> On Thu, Nov 15, 2018 at 4:34 PM Ahmet Altay
>> mailto:al...@google.com>> wrote:
>>
>> +1 Thank you.
>>
>> On Thu, Nov 15, 2018 at 4:22 PM, Kenneth Knowles
>> mailto:k...@apache.org>> wrote:
>>
>> SGTM. Thanks for keeping track of the schedule.
>>
>> Kenn
>>
>> On Thu, Nov 15, 2018 at 1:59 PM Chamikara
>> Jayalath > <mailto:chamik...@google.com>> wrote:
>>
>> Hi All,
>>
>> According to the release calendar [1]
>> branch cut date for Beam 2.9.0 release is
>> 11/21/2018. Since previous release branch
>> was cut close to the respective calendar
>> date I'd like to propose cutting release
>> branch for 2.9.0 on 11/21/2018. Next week
>> is Thanksgiving holiday in US and possibly
>> some folks will be out so we can try to
>> produce RC1 on Monday after (11/26/2018).
>> We can attend to current blocker JIRAs [2]
>> in the meantime. 
>>
>>     I'd like to volunteer to perform this release.
>>
>> WDYT ?
>>
>> Thanks,
>> Cham
>>
>> [1] 
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
>> [2] https://s.apache.org/beam-2.9.0-burndown
>>
>>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Mark 2.7.0 branch as a long term support (LTS) branch

[Jean-Baptiste Onofré]

+0

Regards
JB

On 09/11/2018 02:47, Ahmet Altay wrote:
> Hi all,
> 
> Please review the following statement:
> 
> "2.7.0 branch will be marked as the long-term-support (LTS) release
> branch. This branch will be supported for a window of 6 months starting
> from the day it is marked as an LTS branch. Beam community will decide
> on which issues will be backported and when patch releases on the branch
> will be made on a case by case basis.
> 
> [ ] +1, Approve
> [ ] -1, Do not approve (please provide specific comments)
> 
> Context:
> - Discussion on the dev@ list [1].
> - Beam's release policies [2].
> 
> Thank you,
> Ahmet
> 
> [1] 
> https://lists.apache.org/thread.html/f604b2d688ad467ddccd4cf56b664b7309dae78f1bd8849e4bb9aae2@%3Cdev.beam.apache.org%3E
> [2] https://beam.apache.org/community/policies/
> 
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [Euphoria] Looking for a reviewer.

[Jean-Baptiste Onofré]

Hi,

Yes, I'm on it. Sorry, I'm just back from 2 weeks of vacation ;)

Regards
JB

On 07/11/2018 17:07, Scott Wegner wrote:
> Václav Plajt had previously reached out [1] looking for reviewers for
> Euphoria, and a few individuals volunteered. JB, Max, are you still
> able to help out with Euphoria reviews?
>
> [1] 
> https://lists.apache.org/thread.html/af987d857b6eb583bd74a28ad7bd3ddb532d31ed5e66239e19ec5965@%3Cdev.beam.apache.org%3E
>
> On Tue, Nov 6, 2018 at 10:26 AM David Morávek  <mailto:david.mora...@gmail.com>> wrote:
>
> Hello,
>
> I'm looking for a reviewer for [BEAM-5790]
> <https://github.com/apache/beam/pull/6750> and also for any
> upcoming Euphoria PR which I submit.
>
> It has been already reviewed internally, but it should be also
> reviewed by a committer who did not author the code.
>
> It would be also great if other committer becomes familiar with
> the code base (at least through reviews).
>
> Is anyone willing to help?
>
> Thanks,
> D.
>
>
>
> -- 
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
> <https://tinyurl.com/swegner-feedback>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Stackoverflow Questions

[Jean-Baptiste Onofré]

That's "classic" in the Apache projects. And yes, most of the time, we
periodically send or ask the dev to check the questions on other
channels like stackoverflow.

It makes sense to send a reminder or a list of open questions on the
user mailing list (users can help each other too).

Regards
JB

On 05/11/2018 18:25, Anton Kedin wrote:
> Hi dev@,
> 
> I was looking at stackoverflow questions tagged with `apache-beam` [1]
> and wanted to ask your opinion. It feels like it's easier for some users
> to ask questions on stackoverflow than on user@. Overall frequency
> between the two channels seems comparable but a lot of stackoverflow
> questions are not answered while questions on user@ get some attention
> most of the time. Would it make sense to increase dev@ visibility into
> stackoverflow, e.g. by sending periodic digest or some other way?
> 
> [1] https://stackoverflow.com/questions/tagged/apache-beam
> 
> Regards,
> Anton

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com