Re: TextIO. Writing late files

> This is still confusing to me - why would the messages be dropped as late in > this case? Since you previously mentioned that the bug is due to the pane info missing, I just pointed out that the WriteFiles logic is expected to drop the pane info. @Jose Would it make sense to file a JIRA and

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations! On Tue, May 19, 2020 at 7:31 AM Aizhamal Nurmamat kyzy wrote: > Congratulations, Robin! Thank you for your contributions! > > On Mon, May 18, 2020, 7:18 PM Boyuan Zhang wrote: > >> Congrats~~ >> >> On Mon, May 18, 2020 at 7:17 PM Reza Rokni wrote: >> >>> Congratulations! >>>

Discussion on Project Idea for Session Of docs 2020

Hello Aizhamal, I am working on Big Data technologies and has hands-on experience on Flink, Spark, Kafka and also did POC where I created Docker image of Fink job and ran it on K8S cluster on the local machine. Attaching my POC project: https://github.com/sanghisha145/flink_on_k8s I really find

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats! On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov wrote: > Congratulations! > > On Tue, May 19, 2020 at 7:31 AM Aizhamal Nurmamat kyzy < > aizha...@apache.org> wrote: > >> Congratulations, Robin! Thank you for your contributions! >> >> On Mon, May 18, 2020, 7:18 PM Boyuan Zhang wrote: >>

Re: [ANNOUNCE] New committer: Robin Qiu

Woohoo! Congratulations Robin! On Tue, May 19, 2020 at 8:02 AM Tyson Hamilton wrote: > Congratulations! > > On Tue, May 19, 2020 at 6:10 AM Omar Ismail wrote: > >> Congrats! >> >> On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov wrote: >> >>> Congratulations! >>> >>> On Tue, May 19, 2020 at 7:31

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats Robin! On 5/19/20 5:01 PM, Tyson Hamilton wrote: Congratulations! On Tue, May 19, 2020 at 6:10 AM Omar Ismail > wrote: Congrats! On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov mailto:g...@spotify.com>> wrote: Congratulations!

Re: Running NexMark Tests

Looks like an accidental change to me. Running with either version, 1.9 or 1.10 works, but this should be changed back to using the latest version. Do you mind creating a PR? Thanks, Max On 19.05.20 13:02, Sruthi Sree Kumar wrote: > On the documentation, the version of Flink runner is changed

Re: Running NexMark Tests

On the documentation, the version of Flink runner is changed to 1.9 which was 1.10(latest) before https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9 . Is this an accidental change or is there any particular reason for this

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats! On Tue, May 19, 2020 at 5:33 PM Jan Lukavský wrote: > Congrats Robin! > On 5/19/20 5:01 PM, Tyson Hamilton wrote: > > Congratulations! > > On Tue, May 19, 2020 at 6:10 AM Omar Ismail wrote: > >> Congrats! >> >> On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov wrote: >> >>>

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats Robin! On Tue, May 19, 2020 at 8:56 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Congrats! > > On Tue, May 19, 2020 at 5:33 PM Jan Lukavský wrote: > >> Congrats Robin! >> On 5/19/20 5:01 PM, Tyson Hamilton wrote: >> >> Congratulations! >> >> On Tue, May 19, 2020 at 6:10

Google Season of Document

Dear Aizhamal, I am Hossam Elsafty a graduated from the Faculty of Engineering Computer and System Department. As one of my academic courses, I have finished Technical Reports Writing. Regarding my previous experience, I have done two internships as a Software Engineer including one in Valeo

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations! On Tue, May 19, 2020 at 6:10 AM Omar Ismail wrote: > Congrats! > > On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov wrote: > >> Congratulations! >> >> On Tue, May 19, 2020 at 7:31 AM Aizhamal Nurmamat kyzy < >> aizha...@apache.org> wrote: >> >>> Congratulations, Robin! Thank you

Re: Running NexMark Tests

PR for the update: https://github.com/apache/beam/pull/11751 Regards, Sruthi On Tue, May 19, 2020 at 3:51 PM Maximilian Michels wrote: > Looks like an accidental change to me. Running with either version, 1.9 > or 1.10 works, but this should be changed back to using the latest version. > > Do

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations Robin! On Tue, May 19, 2020, 10:15 Valentyn Tymofieiev wrote: > Congratulations, Robin! > > On Tue, May 19, 2020 at 9:10 AM Yichi Zhang wrote: > >> Congrats Robin! >> >> On Tue, May 19, 2020 at 8:56 AM Kamil Wasilewski < >> kamil.wasilew...@polidea.com> wrote: >> >>> Congrats!

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations, Robin! On Tue, May 19, 2020 at 9:10 AM Yichi Zhang wrote: > Congrats Robin! > > On Tue, May 19, 2020 at 8:56 AM Kamil Wasilewski < > kamil.wasilew...@polidea.com> wrote: > >> Congrats! >> >> On Tue, May 19, 2020 at 5:33 PM Jan Lukavský wrote: >> >>> Congrats Robin! >>> On

Re: [ANNOUNCE] New committer: Robin Qiu

yoohoo : ) On Tue, May 19, 2020 at 11:03 AM Yifan Zou wrote: > Congratulations, Robin! > > On Tue, May 19, 2020 at 10:53 AM Udi Meiri wrote: > >> Congratulations Robin! >> >> On Tue, May 19, 2020, 10:15 Valentyn Tymofieiev >> wrote: >> >>> Congratulations, Robin! >>> >>> On Tue, May 19, 2020

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations, Robin! On Tue, May 19, 2020 at 10:53 AM Udi Meiri wrote: > Congratulations Robin! > > On Tue, May 19, 2020, 10:15 Valentyn Tymofieiev > wrote: > >> Congratulations, Robin! >> >> On Tue, May 19, 2020 at 9:10 AM Yichi Zhang wrote: >> >>> Congrats Robin! >>> >>> On Tue, May 19,

Re: Removing DATETIME in Java Schemas

On Fri, May 15, 2020 at 10:33 PM Reuven Lax wrote: > > > On Fri, May 15, 2020 at 8:10 PM Kenneth Knowles wrote: > >> >> >> On Fri, May 15, 2020 at 5:25 PM Brian Hulette >> wrote: >> >>> After thinking about this more I've softened on it some, but I'm still a >>> little wary. I like Kenn's

Re: Try Beam Katas Today

Sure! I ran through all of the tests locally on my branch (as tests) and then performed a check against all of the known tasks (via Course Creator > Check All Tasks) and 35/36 tasks passed successfully with the only one that didn't being a Built-in IO one that doesn't currently have any

Re: BEAM-9958: Code Review Wanted for PR 11674

Ahmet, Thank you for the review and merging. On Fri, May 15, 2020 at 2:06 PM Tomo Suzuki wrote: > Hi Luke and Beam committers, > > Would you check this PR to use Linkage Checker's exclusion file? > https://github.com/apache/beam/pull/11674 > This script used to use "diff" command to identify

Re: Try Beam Katas Today

This is really cool Rion! I believe it's possible to start trying out the katas from your branch? If so, I can give them a try, and use that as a review... Henry, any other ideas? On Tue, May 19, 2020 at 12:04 PM Rion Williams wrote: > Hi all, > > I was recently added as a contributor and

Re: Try Beam Katas Today

Hi all, I was recently added as a contributor and created a JIRA ticket related to the existing Katas (https://issues.apache.org/jira/browse/BEAM-10027), specifically creating one that targets Kotlin specific as there are quite a few existing examples out there for Kotlin, so I thought a Kata

Re: [VOTE] Release 2.21.0, release candidate #1

Thanks for bringing that up Steve. I'll leave it to others to vote on whether that necessitates an RC #2. On Tue, May 19, 2020 at 5:22 PM Steve Niemitz wrote: > https://issues.apache.org/jira/browse/BEAM-10015 was marked as 2.21 but > isn't in the RC1 tag. It's marked as P1, and seems like the

Re: Publishing release notes to the Github releases page

Including the release notes in the tag seems like a good idea. However, I don't think there is an obvious way to automate the process, since Beam's primary release tool is shell scripts :) and Github tag summaries are a Github feature, not a standard git feature. Would it be sufficient to manually

Re: Publishing release notes to the Github releases page

> It seems that Github treats the summary as raw text, so you can't really feed it any complex formatting like Markdown. The Helm project you linked seems to be using markdown formatting. On Tue, May 19, 2020 at 3:52 PM Julien Phalip wrote: > I actually tried that - you can see a quick test

Re: [VOTE] Release 2.21.0, release candidate #1

+1 (binding) Verified: * Signatures and file hashes * Java quickstarts on local cluster runners and Dataflow. On Tue, May 19, 2020 at 1:51 PM Kyle Weaver wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 2.21.0, > as follows: > [ ] +1, Approve the

Publishing release notes to the Github releases page

Hi, I'm working with customers who would like to be automatically notified when new Beam releases come out. They'd also like to see the release notes so they know what changes were made. I know that these announcements are already sent to the user@ and dev@ mailing lists. However, they're not

Event Calendar?

Hi All, As we have events more often that are more accessible (digital), wondering whether others see a value of adding a calendar to the website? Perhaps related, is it worth updating https://beam.apache.org/community/in-person/ <- to something that isn't 'in-person' since doing things

Re: Publishing release notes to the Github releases page

I actually tried that - you can see a quick test here: https://github.com/jphalip/beam/releases/tag/v9.9.9 It seems that Github treats the summary as raw text, so you can't really feed it any complex formatting like Markdown. That said, that might be good enough if the summary just includes some

Re: More metadata in Coder Proto

I see. The problem is that you are trying to know certain properties of the coder to use in a downstream transform which enforces that it is deterministic like GroupByKey. In all the scenarios so far that I have seen we have required both SDKs to understand the coder, how are you having a cross

[VOTE] Release 2.21.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.21.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [ANNOUNCE] New committer: Robin Qiu

Nice! Congrats! -Rui On Tue, May 19, 2020 at 11:13 AM Pablo Estrada wrote: > yoohoo : ) > > On Tue, May 19, 2020 at 11:03 AM Yifan Zou wrote: > >> Congratulations, Robin! >> >> On Tue, May 19, 2020 at 10:53 AM Udi Meiri wrote: >> >>> Congratulations Robin! >>> >>> On Tue, May 19, 2020,

More metadata in Coder Proto

Hi all, Should there be more metadata in the Coder Proto? For example, adding an "is_deterministic" boolean field. This will allow for a language-agnostic way for SDKs to infer properties about a coder received from the expansion service. My motivation for this is that I recently ran into a

Re: Publishing release notes to the Github releases page

I don't think it would be that complicated to integrate into shell script based release tooling (maybe I'll have a different opinion in a few weeks after 2.22 is out?). Can't we just make a request to the releases API with curl? On Tue, May 19, 2020 at 3:39 PM Kyle Weaver wrote: > Including the

Re: BEAM-9958: Code Review Wanted for PR 11674

Thanks Ahmet. On Tue, May 19, 2020 at 1:26 PM Tomo Suzuki wrote: > Ahmet, > > Thank you for the review and merging. > > On Fri, May 15, 2020 at 2:06 PM Tomo Suzuki wrote: > >> Hi Luke and Beam committers, >> >> Would you check this PR to use Linkage Checker's exclusion file? >>

Re: Publishing release notes to the Github releases page

Yes, Markdown is possible when using the Releases API. There I was referring to the default behavior, where Github displays the tag summary as raw text if a formal Github release entry wasn't created for the tag. To create a formal Github release entry (

Re: [VOTE] Release 2.21.0, release candidate #1

https://issues.apache.org/jira/browse/BEAM-10015 was marked as 2.21 but isn't in the RC1 tag. It's marked as P1, and seems like the implication is that without the fix, pipelines can produce incorrect data. Is this a blocker? On Tue, May 19, 2020 at 4:51 PM Kyle Weaver wrote: > Hi everyone, >

Re: Publishing release notes to the Github releases page

For context, currently, we just create and push the tag using plain git: https://github.com/apache/beam/blob/master/website/www/site/content/en/contribute/release-guide.md#git-tag > I don't think it would be that complicated to integrate into shell script based release tooling (maybe I'll have a

Re: [VOTE] Release 2.21.0, release candidate #1

Rahul, do you believe that the release is severely broken without PR/11609 enough to require another release candidate or would waiting till 2.22 (which is due to be cut tomorrow)? On Tue, May 19, 2020 at 8:13 PM rahul patwari wrote: > Hi, > > Can the PR:

Re: Try Beam Katas Today

Thanks Rion for adding the Kotlin version. This is great to show other people that Beam can be done in Kotlin too! I can help to review your work. Please help to incorporate the Java Katas latest changes from master. There are recent changes to the task description file format from html to md.

Re: Publishing release notes to the Github releases page

That sounds good, I'll see what I can do and maybe write up a script to automate the release notes publication. On Tue, May 19, 2020 at 4:35 PM Kyle Weaver wrote: > Sorry, I should have worded that better. What I meant was that Brian and I > should focus on fixing the existing release process,

Re: Publishing release notes to the Github releases page

I'd be happy to help with code review (tag @TheNeuralBit on github), and/or kick the tires if you have something ready in time for 2.22. Brian On Tue, May 19, 2020 at 4:38 PM Julien Phalip wrote: > That sounds good, I'll see what I can do and maybe write up a script to > automate the release

Re: [VOTE] Release 2.21.0, release candidate #1

I confirmed that licenses/notices/source code are added to Java and Python docker images as expected. On Tue, May 19, 2020 at 2:36 PM Kyle Weaver wrote: > Thanks for bringing that up Steve. I'll leave it to others to vote on > whether that necessitates an RC #2. > > On Tue, May 19, 2020 at

Re: [VOTE] Release 2.21.0, release candidate #1

+1, I validated python 2 and 3 quickstarts. On Tue, May 19, 2020 at 4:57 PM Hannah Jiang wrote: > I confirmed that licenses/notices/source code are added to Java and Python > docker images as expected. > > > On Tue, May 19, 2020 at 2:36 PM Kyle Weaver wrote: > >> Thanks for bringing that up

Re: [VOTE] Release 2.21.0, release candidate #1

Hi, Can the PR: https://github.com/apache/beam/pull/11609 be cherry-picked for 2.21.0 release? If not, the fix version has to be changed for BEAM-9887 . Regards, Rahul On Wed, May 20, 2020 at 6:05 AM Ahmet Altay wrote: > +1, I validated python

Re: [VOTE] Release 2.21.0, release candidate #1

Hi Luke, The release is not severely broken without PR #11609. The PR ensures that, while building a Row with Logical Type, the input value provided is proper. If we take FixedBytes logical type with length 10, for example, the proper input value will be a byte array of length 10. But, without

Re: Publishing release notes to the Github releases page

Sure, I can try to help :) Can you share some pointers on the things that need fixing? On Tue, May 19, 2020 at 4:17 PM Kyle Weaver wrote: > For context, currently, we just create and push the tag using plain git: >

Re: More metadata in Coder Proto

Yes I'm unclear on how a PCollection with ExternalCoder made it into a downstream transform that enforces is_deterministic. My understanding of ExternalCoder (admittedly just based on a quick look at commit history) is that it's a shim added so the Python SDK can handle coders that are internal to

Re: Publishing release notes to the Github releases page

Sorry, I should have worded that better. What I meant was that Brian and I should focus on fixing the existing release process, but we'd welcome you to add the release notes as a new feature. On Tue, May 19, 2020 at 7:33 PM Julien Phalip wrote: > Sure, I can try to help :) Can you share some

Re: More metadata in Coder Proto

I have a PR that makes GBK a primitive in which the test_combine_globally is failing on the DataflowRunner. In particular, the DataflowRunner runs

Re: More metadata in Coder Proto

Since combine globally is a case where you don't need to know what the key or value is and could treat them as bytes allowing you to build and execute this pipeline (assuming you ignored properties such as is_deterministic). Regardless, I still think it makes sense to provide criteria on what

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats Robin! On Tue, May 19, 2020 at 2:39 PM Rui Wang wrote: > Nice! Congrats! > > > > -Rui > > On Tue, May 19, 2020 at 11:13 AM Pablo Estrada wrote: > >> yoohoo : ) >> >> On Tue, May 19, 2020 at 11:03 AM Yifan Zou wrote: >> >>> Congratulations, Robin! >>> >>> On Tue, May 19, 2020 at 10:53

Re: More metadata in Coder Proto

I think you are hitting GroupByKey [1] that is internal to the Java CombineGlobally implementation that takes a KV with a Void type (with VoidCoder) [2] as input. ExternalCoder was added to Python SDK to represent coders within external transforms that are not standard coders (in this case the

Re: Try Beam Katas Today

Hi Henry, Thanks for the quick response, I appreciate it. I believe that I pulled the latest from master a day or so ago, so I’ll make sure to pull the most recent changes in. As far as the placeholders, they aren’t currently present (as I don’t believe they were present in the Java ones