Support of KafkaIO Dynamic Read

Hi team, I'm working on KafkaIO dynamic read support which is tracked by BEAM-11325 and I started the documentation here: https://docs.google.com/document/d/1FU3GxVRetHPLVizP3Mdv6mP5tpjZ3fd99qNjUI5DT5k/edit?usp=sharing, which states the problem I

Re: pulllicenses fails while building

Unfortunately, setting that env variable didn't change anything. On Thu, Jan 7, 2021 at 7:09 PM Ahmet Altay wrote: > Googled this a bit. Setting this env variable might fix the problem by > bypassing the check: > export PYTHONHTTPSVERIFY=0 > > Could you try that? If it works we can make that

Re: pulllicenses fails while building

Googled this a bit. Setting this env variable might fix the problem by bypassing the check: export PYTHONHTTPSVERIFY=0 Could you try that? If it works we can make that part of the script. Also, I am not sure why licenses are pulled for regular development case. I thought it was not meant to run

pulllicenses fails while building

I recently upgraded to Python 3 on my build machine. Now all attempts to build Beam fail with the following. Anyone know how to ix this? urllib.error.URLError: ERROR:root:Invalid url for paranamer-2.7: https://raw.githubusercontent.com/paul-hammant/paranamer/master/LICENSE.txt.

Re: Why are all the website files failing RAT?

I just merged https://github.com/apache/beam/pull/13697 which should resolve this. On Thu, Jan 7, 2021 at 3:39 PM Ahmet Altay wrote: > > > On Thu, Jan 7, 2021 at 3:35 PM Brian Hulette wrote: > >> >> >> On Thu, Jan 7, 2021 at 2:00 PM Ahmet Altay wrote: >> >>> >>> >>> On Thu, Jan 7, 2021 at

Re: Website is MIT licensed?

I agree with you. I guess we missed this during the website migration. https://github.com/apache/beam/pull/13698 is to fix. On Thu, Jan 7, 2021 at 12:42 PM Kyle Weaver wrote: > Hi all, > > I discovered the Beam website's NPM module purports to be MIT licensed > [1]. I don't think we publish

Re: Why are all the website files failing RAT?

On Thu, Jan 7, 2021 at 3:35 PM Brian Hulette wrote: > > > On Thu, Jan 7, 2021 at 2:00 PM Ahmet Altay wrote: > >> >> >> On Thu, Jan 7, 2021 at 12:25 PM Kyle Weaver wrote: >> >>> I repro'd this by running "./gradlew :rat". If I understand correctly, >>> these are all Hugo dependencies that are

Re: Why are all the website files failing RAT?

On Thu, Jan 7, 2021 at 2:00 PM Ahmet Altay wrote: > > > On Thu, Jan 7, 2021 at 12:25 PM Kyle Weaver wrote: > >> I repro'd this by running "./gradlew :rat". If I understand correctly, >> these are all Hugo dependencies that are downloaded automatically. I looked >> at a few of them and they do

Re: [VOTE] Release 2.27.0, release candidate #4

+1 (non-binding) Tested Python DataFrame pipeline and sql_taxi example (multi-language, streaming) on Dataflow. On Thu, Jan 7, 2021 at 2:35 PM Chamikara Jayalath wrote: > +1 (non-binding). > > Tested Java quickstart for Direct/Dataflow runners and multi-language > pipelines. > > Thanks, > Cham

Re: Compatibility between Beam v2.23 and Beam v2.26

Hi Jan, I create this jira: https://issues.apache.org/jira/browse/BEAM-11583 Thanks. Antonio. On 2021/01/07 08:43:34, Jan Lukavský wrote: > Hi Antonio, > > can you please create one? > > Thanks, > >  Jan > > On 1/6/21 10:31 PM, Antonio Si wrote: > > Thanks for the information. Do we

Re: [VOTE] Release 2.27.0, release candidate #4

+1 (non-binding). Tested Java quickstart for Direct/Dataflow runners and multi-language pipelines. Thanks, Cham On Thu, Jan 7, 2021 at 10:47 AM Pablo Estrada wrote: > Thanks Valentyn, > these were built using Maven 3.6.0, and OpenJDK version 1.8.0_232. > Best > -P. > > On Thu, Jan 7, 2021 at

Re: Why are all the website files failing RAT?

On Thu, Jan 7, 2021 at 12:25 PM Kyle Weaver wrote: > I repro'd this by running "./gradlew :rat". If I understand correctly, > these are all Hugo dependencies that are downloaded automatically. I looked > at a few of them and they do have licenses, but I guess rat just doesn't > recognize them

Website is MIT licensed?

Hi all, I discovered the Beam website's NPM module purports to be MIT licensed [1]. I don't think we publish the website's NPM package anywhere, but wouldn't the website be Apache 2.0 licensed like everything else? Thanks, Kyle [1]

Re: Why are all the website files failing RAT?

I repro'd this by running "./gradlew :rat". If I understand correctly, these are all Hugo dependencies that are downloaded automatically. I looked at a few of them and they do have licenses, but I guess rat just doesn't recognize them for whatever reason. The rat task is supposed to ignore

Re: Standarizing the "Runner" concept across website content

To those unfamiliar with these concepts, I generally conflate everything to a "Runner" to keep things simple. Though, also mention "execution engine" at times. Glad there appears to be concrete consensus on how we want to talk about this. It will also help guide me in being consistent :-) On

Re: [VOTE] Release 2.27.0, release candidate #4

Thanks Valentyn, these were built using Maven 3.6.0, and OpenJDK version 1.8.0_232. Best -P. On Thu, Jan 7, 2021 at 9:45 AM Yichi Zhang wrote: > +1, I verified the python mobile game walkthrough on core runners. > > On Thu, Jan 7, 2021 at 8:29 AM Valentyn Tymofieiev > wrote: > >> On Thu, Jan

Why are all the website files failing RAT?

My builds are failing recently, with complaints of 2445 license violations. It appears to be a bunch of website files, see below. Any idea what is happening here? Unapproved Licenses: /Users/relax/beam/website/www/node_modules/callsites/index.js

Re: [VOTE] Release 2.27.0, release candidate #4

+1, I verified the python mobile game walkthrough on core runners. On Thu, Jan 7, 2021 at 8:29 AM Valentyn Tymofieiev wrote: > On Thu, Jan 7, 2021 at 8:06 AM Ismaël Mejía wrote: > >> > Also I wonder if we now need to clarify both Java 8 and Java 11 >> versions separately? >> >> You mean for

Re: Nexmark ratelimiting not working

It looks like the 'isRateLimited' flag was used on these tests inside google ~5 years ago, but I don't think we've ever used it in Beam. I don't believe there are any tests so it doesn't really surprise me that it is broken. The configs we normally run nexmark on Flink with are here:

Re: [VOTE] Release 2.27.0, release candidate #4

On Thu, Jan 7, 2021 at 8:06 AM Ismaël Mejía wrote: > > Also I wonder if we now need to clarify both Java 8 and Java 11 versions > separately? > > You mean for the docker images? Otherwise we should not be using Java > 11 at all to produce the artifacts. > Thanks for clarification, yes, looks

Re: [VOTE] Release 2.27.0, release candidate #4

+1 (binding) Regards JB > Le 6 janv. 2021 à 08:17, Pablo Estrada a écrit : > > Hi everyone, > Please review and vote on the release candidate #4 for the version 2.27.0, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > >

Re: [VOTE] Release 2.27.0, release candidate #4

> Also I wonder if we now need to clarify both Java 8 and Java 11 versions > separately? You mean for the docker images? Otherwise we should not be using Java 11 at all to produce the artifacts. On Thu, Jan 7, 2021 at 4:51 PM Valentyn Tymofieiev wrote: > > Noting that announcement does not

Re: [VOTE] Release 2.27.0, release candidate #4

Noting that announcement does not include the version of the Java compilers used - looks like the release guide still requires it: * Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle JDK JDK_VERSION. Could you please add this info to this thread for posterity? Also I wonder

Nexmark ratelimiting not working

Hello and happy new years! I've been trying to use Nexmark for evaluating some performance improvements I've made to Beam. I've discovered a problem where Nexmark won't work if you enable the ratelimiting mode. I've filled out a bug report for this[1]. To reproduce the problem is easy,

Re: Compatibility between Beam v2.23 and Beam v2.26

Thanks for mentioning me here @Boyan. In Beam there is no guarantee that checkpoints work across Beam releases. Checkpoint compatibility can break due to a lot of reasons (primarily DAG changes and serializer changes). Even though in this case the serialization id might have guaranteed

Re: [VOTE] Release 2.27.0, release candidate #4

+1 (non-binding). I've validated the RC against my dependent projects (mainly Java SDK, Flink and DirectRunner). Thanks,  Jan On 1/7/21 2:15 AM, Ahmet Altay wrote: +1 (binding) - validated python quickstarts. Thank you Pablo. On Wed, Jan 6, 2021 at 1:57 PM Pablo Estrada

Re: Compatibility between Beam v2.23 and Beam v2.26

Hi Antonio, can you please create one? Thanks,  Jan On 1/6/21 10:31 PM, Antonio Si wrote: Thanks for the information. Do we have a jira to track this issue or do you want me to create a jira for this? Thanks. Antonio. On 2021/01/06 17:59:47, Kenneth Knowles wrote: Agree with Boyuan &