Re: Java precomit failing, (though no test are failing)

For tasks without structured JUnit output, we have to scroll up / ctrl-f / grep for more logs. In this case it looks like it was probably a server-side issue. These links work for me, so I'm assuming the problem has been resolved. *11:31:04* >* Task :release:go-licenses:java:dockerRun**11:31:04*

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

ger Pod could be started together and the task manager > container need to connect to the worker pool service container sooner. > > Hope this clarifies things better. Let me know if you have more questions. > > Best, > Ke > > [1] https://issues.apache.org/jira/browse/FLINK-10

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

> > Therefore, if we bring up the external worker pool container together with > the runner container, which is one the supported approach by Flink Runner > on K8s Exactly which approach are you talking about in the doc? I feel like there could be some misunderstanding here. Here is the

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

PC calls. > > Created BEAM-12419 <https://issues.apache.org/jira/browse/BEAM-12419> to > track the work. > > Best, > Ke > > On May 27, 2021, at 4:30 PM, Kyle Weaver wrote: > > I don't think there's any specific reason we don't set a timeout, I'm > guessing it w

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

s sense to block since the wait/retry is handled in the previous > prepare(), however, is there any specific reason why we do not want to > support timeout in start worker request? > > Best, > Ke > > On May 14, 2021, at 11:25 AM, Kyle Weaver wrote: > > 1. and 2. are both f

Re: One Pager - Test Command Line Discoverability in Beam

Filed a wish JIRA (my favorite kind): BEAM-12412 On Wed, May 26, 2021 at 4:34 PM Kyle Weaver wrote: > > Also just naming Jenkins jobs exactly by the command to run them would > go a long way for me, personally. > > Good point. If I may complain provide some detail, AFAIK Jenkins

Re: One Pager - Test Command Line Discoverability in Beam

has required parameters. > > Also just naming Jenkins jobs exactly by the command to run them would go > a long way for me, personally. > > Kenn > > On Tue, May 25, 2021 at 2:46 PM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> Cool; will be good to

Re: One Pager - Test Command Line Discoverability in Beam

I left some comments. In summary, I think this is mostly a documentation problem. If running a test isn't as easy as "./gradlew $MODULE:integrationTest", there should be instructions in the test class's javadoc. On Tue, May 25, 2021 at 2:05 PM Udi Meiri wrote: > My first place to go would be

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

1. and 2. are both facilitated by GRPC, which takes care of most of the retry/wait logic. In some places we have a configurable timeout (which defaults to 60s) [1], while in other places we block [2][3]. [1] https://issues.apache.org/jira/browse/BEAM-7933 [2]

Re: Please help triage issues!

It's a little cumbersome, but you can query JIRA and export a CSV with the labels, and run a script to count them. Also, it won't let you export results from a query with more than 1000 results. Here's the list from query "project = beam and created > startOfYear()" dataframe-api 45 stale-P2 36

Re: BeamSQL: Error when using WHERE statements with OVER windows

nd Calcite. If you want to >> contribute to analytics function, there is a design doc that you can check >> to understand how query is mapped to Rel nodes: >> https://docs.google.com/document/d/1tJapdA7ZNwkU0NaK7p-em0XnpHqNE1pKIXw9hVJkIUg/edit?usp=sharing >> >> If you don'

Re: BeamSQL: Error when using WHERE statements with OVER windows

If I'm understanding correctly, this query is taken from an existing test [1], the only modification is the addition of "WHERE purchases > 3". This seems like a bug -- I'm guessing the planner/matching logic doesn't handle filters properly. To really figure out what's going on, you'd have to

Jenkins master is down

Just a quick PSA - Beam's Jenkins master is currently out of disk space, preventing any Jenkins CI jobs from running. We are looking into the issue ([1] for tracking). Thanks for your patience. Kyle [1] https://issues.apache.org/jira/browse/INFRA-21857

Re: Customizable Artifacts to Stage in Java Portable Runner

er/worker_pool_main.py#L56> > does > invoke artifact staging service to get artifacts from artifact endpoint > specified in StartWorkerRequest but not in the java ExternalWorkerService. > Is this discrepancy expected since java worker pool process does not likely > want to start the worke

Re: Customizable Artifacts to Stage in Java Portable Runner

> I am expecting FileStagingOptions#setFilesToStage in PortablePipelineOptions is the way to customize artifacts to be staged and resolved in portable pipeline,

Re: Portable Java Pipeline Support

astest way is to patch it ourselves? > > Best, > Ke > > > On Apr 26, 2021, at 10:17 AM, Kyle Weaver wrote: > > The reason is the Flink and Spark runners are written in Java. So when the > runner needs to execute user code written in Java, an EMBEDDED environment >

Re: Portable Java Pipeline Support

ented exception. > > Is there any specific reason that Java tests are using EMBEDDED mode while > python usually in LOOPBACK mode? > > Best, > Ke > > On Apr 23, 2021, at 4:01 PM, Kyle Weaver wrote: > > I couldn't find any existing ticket for this issue (you may be the

Re: Portable Java Pipeline Support

> LOG.info("Key {}: value {}", input.getKey(), input.getValue()); > return input.getKey() + ": " + input.getValue(); > } > } > > > This simple pipeline did work in Java LOOPBACK mode. > > Best, > Ke > > On Apr 23, 2021, at 1:16 PM,

Re: Portable Java Pipeline Support

Yes, we can expect to run java pipelines in portable mode. I'm guessing the method unimplemented exception is a bug, and we haven't caught it because (as far as I know) we don't test the Java loopback worker. As an alternative, you can try building the Java docker environment with "./gradlew

Re: [VOTE] Release 2.29.0, release candidate #1

+1 (non-) Ran Python wordcount on Flink and Spark. On Wed, Apr 21, 2021 at 5:20 PM Brian Hulette wrote: > +1 (non-binding) > > I ran a python pipeline exercising the DataFrame API, and another > exercising SQLTransform in Python, both on Dataflow. > > On Wed, Apr 21, 2021 at 12:55 PM Kenneth

Re: Flink runner configuration for closure cleaner

t; > On Tue, Apr 6, 2021 at 5:39 PM Kyle Weaver wrote: > >> I don't think this will require Beam to have its own configuration >> option. You should be able to set the property >> "pipeline.closure-cleaner-level" in your flink.conf and then pass it to >> Be

Re: Flink runner configuration for closure cleaner

I don't think this will require Beam to have its own configuration option. You should be able to set the property "pipeline.closure-cleaner-level" in your flink.conf and then pass it to Beam using Beam's "--flink-conf-dir" pipeline option. On Tue, Apr 6, 2021 at 2:28 PM Raman Gupta wrote: >

Re: Null checking in Beam

dards of quality. How about we try moving it to a separate CI job >>>>>>>> and >>>>>>>> see how it goes? >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>> In my experience stor

Re: Null checking in Beam

> > Big +1 for moving this to separate CI job. I really don't like what > annotations are currently added to the code we ship. Tools like Idea add > these annotations to code they generate when overriding classes and that's > very annoying. Users should not be exposed to internal tools like >

Re: Null checking in Beam

t from a shared distributed cache I'm guessing. Is there a JIRA or something for that? I feel like this has been discussed before, but I don't remember the details. On Fri, Mar 12, 2021 at 3:57 PM Kenneth Knowles wrote: > > > On Fri, Mar 12, 2021 at 3:05 PM Kyle Weaver wrote: > >

Re: Null checking in Beam

n > > On Fri, Mar 12, 2021 at 1:19 PM Pablo Estrada wrote: > >> Does it make sense to add a Jenkins precommit suite that runs null >> checking exclusively? >> >> On Fri, Mar 12, 2021 at 11:57 AM Kyle Weaver wrote: >> >>> I don't mind fixing

Re: Null checking in Beam

e.org/jira/browse/BEAM-11837 Since nullness checks seem to work fine for now on Jenkins, I propose making them opt-in rather than opt-out, and only run them on Jenkins by default. On Tue, Mar 2, 2021 at 12:13 PM Kyle Weaver wrote: > Can you try adding the generated classes to generated

Re: Java Tests are failing on Github checks

I don't think this is just a flake; it seems Github actions Java tests are permared right now. I filed a JIRA for it: https://issues.apache.org/jira/browse/BEAM-11921 On Thu, Mar 4, 2021 at 9:19 AM Robert Bradshaw wrote: > I've noticed this sometimes for Python as well: Jenkins is happy with

Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql

confuses me is: > > PAssert.that(stream).satisfies(row -> assertThat("array_agg_concat_field", > actual , > containsInAnyOrder(Arrays.asList(1L,2L,3L,4L,5L,6L; > > How come I can access *actual* here when output is not materialized. > > On T

Re: Null checking in Beam

Can you try adding the generated classes to generatedClassPatterns in the JavaNatureConfiguration? https://github.com/apache/beam/blob/03b883b415d27244ddabb17a0fb5bab147b86f89/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L92 On Tue, Mar 2, 2021 at 12:05 AM Reuven Lax

Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql

As you can see from existing tests, Beam doesn't materialize the output array directly. Instead you must use the PAssert API. I agree with Tyson's suggestion to use `satisfies`, which lets you do arbitrary assertions on the output data. On Tue, Mar 2, 2021 at 3:57 AM Sonam Ramchand <

Re: Not running tests that need DirectRunner in core?

Shouldn't the runner only be needed at runtime? On Fri, Feb 5, 2021 at 10:09 AM Daniel Collins wrote: > Ah looks like its a bit harder than this. But I'll try to make it work for > building all of beam locally then get back to you with a working PR. > > On Fri, Feb 5, 2021 at 1:07 PM Daniel

Re: Environment options for external transforms

1 at 3:52 PM Robert Bradshaw wrote: > On Thu, Feb 4, 2021 at 3:33 PM Kyle Weaver wrote: > >> This gets into the distinction of customizing what kind of environment >>> one wants to have (which could be generally applicable) vs. an absolute >>> designation of a

Re: Environment options for external transforms

1]. So in this particular case the user just wants to mount files into their SDK harness, which is a pretty common use case, so resource hints are probably a more appropriate solution. [1] https://lists.apache.org/thread.html/r690094f1c9ebc4e1d20f029a21ba8bc846672a65baafd57c4f52cb94%40%3Cuser.beam.apa

Re: Environment options for external transforms

wrote: > > > On Wed, Feb 3, 2021 at 12:34 PM Kyle Weaver wrote: > >> Hi Beamers, >> >> Recently we’ve had some requests on user@ and Slack for instructions on >> how to use custom-built containers in cross-language pipelines (typically >> calling J

Environment options for external transforms

Hi Beamers, Recently we’ve had some requests on user@ and Slack for instructions on how to use custom-built containers in cross-language pipelines (typically calling Java transforms from a predominantly Python pipeline). Currently, it seems like there is no way to change the container used by a

Re: Problems building latest beam source

This should be fixed now. On Tue, Jan 26, 2021 at 10:23 AM Kyle Weaver wrote: > I missed this thread and filed a JIRA for it: > https://issues.apache.org/jira/browse/BEAM-11689 > > > We could swap the spring.io repo for the pentaho nexus one > public.nexus.pentaho.org (her

Re: [Proposal] Release Guide improvements

> One bigger change to the content that certainly warrants discussion here: I'd like to remove the manual command level copies of the automation scripts in the release document. I approve. I already removed a large chunk of the copies in [1], but I may have missed some places. I think BEAM-11217

Re: Problems building latest beam source

I missed this thread and filed a JIRA for it: https://issues.apache.org/jira/browse/BEAM-11689 > We could swap the spring.io repo for the pentaho nexus one public.nexus.pentaho.org (here probably in Beam

Re: [PROPOSAL] Preparing for Beam 2.28.0 release

Thanks for stepping up Cham! Remember to mark critical JIRA issues as release blockers everybody! On Wed, Jan 13, 2021 at 11:25 AM Chamikara Jayalath wrote: > Hi All, > > Beam 2.28.0 release is scheduled to be cut on January 27th according to > the release calendar [1] > > I'd like to

Re: pulllicenses fails while building

FYI Emily fixed the bug with unintended license pulling. So bugs like the one Reuven originally reported should be bypassed by default. On Fri, Jan 8, 2021 at 5:01 PM Kyle Weaver wrote: > > $0.02 can we make it a separate target rather than an existing target > with flags? > >

Re: pulllicenses fails while building

les wrote: > $0.02 can we make it a separate target rather than an existing target with > flags? > > On Fri, Jan 8, 2021 at 11:06 AM Kyle Weaver wrote: > >> ./gradlew :sdks:java:container:build runs the pullLicenses task. >> https://issues.apache.org/jira/browse/BEAM-1

Re: pulllicenses fails while building

./gradlew :sdks:java:container:build runs the pullLicenses task. https://issues.apache.org/jira/browse/BEAM-11586 On Fri, Jan 8, 2021 at 10:59 AM Kyle Weaver wrote: > Which Gradle command are you running Reuven? And which Gradle task is > failing? > > > Also, I am not su

Re: pulllicenses fails while building

Which Gradle command are you running Reuven? And which Gradle task is failing? > Also, I am not sure why licenses are pulled for regular development case. I thought it was not meant to run by default. +1 this seems like a bug. IIRC you are supposed to pass "-Pdocker-pull-licenses" to Gradle to

Website is MIT licensed?

Hi all, I discovered the Beam website's NPM module purports to be MIT licensed [1]. I don't think we publish the website's NPM package anywhere, but wouldn't the website be Apache 2.0 licensed like everything else? Thanks, Kyle [1]

Re: Why are all the website files failing RAT?

I repro'd this by running "./gradlew :rat". If I understand correctly, these are all Hugo dependencies that are downloaded automatically. I looked at a few of them and they do have licenses, but I guess rat just doesn't recognize them for whatever reason. The rat task is supposed to ignore

Re: Compatibility between Beam v2.23 and Beam v2.26

This raises a few related questions from me: 1. Do we claim to support resuming Flink checkpoints made with previous Beam versions? 2. Does 1. require full binary compatibility between different versions of runner internals like CoderTypeSerializer? 3. Do we have tests for 1.? On Tue, Jan 5,

Re: Beam Website - Jira Question

There's an explanation for each resolution type here: https://issues.apache.org/jira/secure/ShowConstantsHelp.jspa?decorator=popup#ResolutionTypes Invalid: The problem isn't valid and it can't be fixed. Abandoned: Issues which have gone without any activity for an extended period of time On Tue,

Re: Need help with the Go Sdk

Commented on the PR (https://github.com/apache/beam/pull/12938). On Mon, Dec 28, 2020 at 11:41 AM Sonam Ramchand < sonam.ramch...@venturedive.com> wrote: > Hi Devs, > For ':sdks:go:resolveBuildDependencies' task, I have been getting: > > Exception in resolution, message is: > Cannot resolve

Re: [VOTE] Release 2.27.0, release candidate #1

catch Ismael. Thanks! >> >> I've created https://issues.apache.org/jira/browse/INFRA-21238 to >> request the repositories to be created. >> >> I am not sure what's the status of this work - should this block the >> release until we can get those containers out? >

Re: [VOTE] Release 2.27.0, release candidate #1

+1 (non-binding) Validated wordcount with Python source + Flink and Spark job server jars. Also checked that the ...:sql:udf jar was added and includes our cherry-picks. Thanks Pablo :) On Wed, Dec 23, 2020 at 12:02 PM Ahmet Altay wrote: > +1 (binding). > > I validated python quickstarts. Thank

Re: Python pipeline options documentation

pipeline with --help. > > [1] https://docs.python.org/3/library/argparse.html > > On Mon, Dec 21, 2020 at 1:06 PM Kyle Weaver wrote: > >> Java has a similar but less severe problem, where each option's getter >> and setter method are visible in the Javadoc, but the @

Re: Python pipeline options documentation

Java has a similar but less severe problem, where each option's getter and setter method are visible in the Javadoc, but the @Description is not. e.g. https://beam.apache.org/releases/javadoc/2.26.0/index.html?org/apache/beam/runners/direct/DirectOptions.html On Mon, Dec 21, 2020 at 1:00 PM Kyle

Python pipeline options documentation

Hello Beam developers, I was making some updates to Python's pipeline options documentation when I realized pydoc includes the pipeline options classes, but not their actual options (since option fields are set at runtime) [1]. But we do write a 'help' text for most (all?) of the options. So does

Re: Problem with :sdks:java:container:pullLicenses

Sometimes you have to ctrl-f in the logs to find an error message [1]. 02:54:05 > Task :sdks:java:container:pullLicenses 02:54:05 ERROR:root:['xz-1.5'] 02:54:05 ERROR:root: Licenses were not able to be pulled automatically for some dependencies. Please

Re: BeamSql query failure on complex data structures

Hi Aspen, if the issue has been fixed on Calcite's side, it is waiting for an overdue upgrade of the Calcite version Beam uses (tracked by [1]). Currently Beam depends on Calcite 1.20, and it looks like that issue was fixed in Calcite 1.21 [2]. [1] https://issues.apache.org/jira/browse/BEAM-9379

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

, 2020 at 1:42 PM Sonam Ramchand < sonam.ramch...@venturedive.com> wrote: > Thanks for your quick response. But, updating updating > kafka-clients:1.0.0 to 2.4.1 on the PR does not resolve the issue sadly. > Seems like there is some other issue. > > On Tue, Dec 15, 2020 at 2:03

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

We recently upgraded kafka and kafka-clients to version 2.4.1 [1]. It looks like there are a couple places in your PR that use the old version kafka-clients:1.0.0 [2]. You will need to update your PR to use version 2.4.1 instead. [1]

Re: Dynamic timers in python sdk.

Possibly a dumb question, but: if "the static timer is just a special case of the dynamic timer," why do we need to use different classes at all? On Wed, Dec 9, 2020 at 2:30 PM Yichi Zhang wrote: > Hi, Beam community, > > I'm trying to add the dynamic timer >

Re: Unit tests vs. Integration Tests

another question I should ask - is :test supposed to only run >>> unit tests? I've been assuming so since many modules have separate >>> :integrationTest tasks for *IT tests. >>> >>> On Wed, Dec 2, 2020 at 4:15 PM Kyle Weaver wrote: >>> >>>> &

Re: Unit tests vs. Integration Tests

> Should we (do we) require unit tests to be hermetic? We should. Unit tests are hermetic by definition. That begs the definition of hermetic, but clearly the internet is not. > Personally I think these tests should be classified as integration tests (renamed to *IT, and run with the

Re: Query regarding Array_Agg impl

I'm not sure there's a reason to use generics here, since this class will likely only ever be instantiated once. Have you tried using Object instead of T? On Wed, Nov 25, 2020 at 10:08 AM Sonam Ramchand < sonam.ramch...@venturedive.com> wrote: > Hi Devs, > I am trying to implement Array_Agg( >

Re: Best way to resolve conflict between PipelineOption classes

> One way would be to rename the conflicting property. Specifically, I can rename the property "region" to "awsRegion", which is the same with the ver1 aws module (sdk.io.aws). However, this may break existing pipelines that is using the "region field" Since aws2 is still marked as @Experimental,

Re: Question About HadoopFileSystemOptions#getHdfsConfiguration

Thanks for the detailed investigation Yuhong. This definitely sounds like a bug; the code means to deduplicate identical paths, but uses only String equality, not Path equality. I filed a JIRA issue in case someone wants to work on fixing it: https://issues.apache.org/jira/browse/BEAM-11329 On

Re: beam flink-runner distribution implementation

What are the advantages of using a Histogram instead of a Gauge? Also, check out this design doc for adding histogram metrics to Beam if you haven't already: http://s.apache.org/beam-metrics-api (Not sure what the current status is.) On Wed, Nov 18, 2020 at 1:37 PM Richard Moorhead wrote: >

Re: Problem being encountered while running the Query with COUNTIF function

COUNTIF was not defined properly. See my response to the other thread (let's continue discussion there to avoid context fragmentation). On Thu, Nov 12, 2020 at 10:16 AM Ahmet Altay wrote: > +Kenneth Knowles +Andrew Pilloud > +Robin > Qiu > > On Thu, Nov 12, 2020 at 9:32 AM Sonam Ramchand <

Re: Question about LOGICAL_AND

If you're defining a new built-in function in ZetaSQL, you can define an operator for it here: https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java Then add the operator the table here:

Re: Requesting Beam JIRA Permissions

What is your JIRA username? On Tue, Nov 10, 2020 at 1:28 PM Vachan Shetty wrote: > Hello, > I have recently joined the BigQuery Read API team and will be working on > new features in the Beam connector. Consequently, could I get JIRA > contribution access? > > Regards, > Vachan >

Re: Issues building/running 2.25 on java 8

Do you have JAVA_HOME set? (possibly related: https://issues.apache.org/jira/browse/BEAM-11080) On Fri, Nov 6, 2020 at 3:13 PM Steve Niemitz wrote: > I'm trying out 2.25 (built from source, using java 8), and running into > this error, both on the direct runner and dataflow: > > Caused by:

Re: Requesting Jira contributor access for Website revamp project team

> Can I ask you to provide me additional ones that will allow me to create a new project (Beam website revamp)? Usually we put all our issues under the BEAM project. Then we organize issues by setting their components and labels. I recommend using the existing "website" component along with a new

Re: [VOTE] Release 2.25.0, release candidate #1

> We should update the release guide to make this explicit for the person preparing the release so this does not happen again and eventually include some validation for this in the build. Instructions in the release guide can be easily missed. We should prioritize adding a version check to all

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

I agree with Kenn. Since any pipeline can be made into a template, it doesn't really make sense to have a separate "templates" directory. Based on a quick skim of your PR, the only thing that's specific to Dataflow templates is the instructions. Gradle does not make subprojects inherit

Re: Java precommit errors "cannot find symbol"

> I am not sure how to get more specific info, but they fail with "cannot find symbol" in files: I think these are not the cause of job failure, since they are also happening on passing builds (e.g. https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/14060/java/). Look in the console

Re: [UPDATE] Beam 2.25.0 release progress update

> +1 to the idea. We discussed it in the dev list [1]. I do not believe we discussed it with INFRA. We would have to collect everyone's Docker hub usernames first. > Could someone help Robin on this ticket? Would reaching out to infra on slack help? I messaged #asfinfra on Slack.

Re: Reviewers for SparkRunner

I think Alexey would be a good reviewer for this. Not sure about this particular case, but in general Beam's OWNERS files are pretty outdated. If I'm not sure who to ask for a review, usually I look at the git history of a file to see who has actively changed it recently. On Mon, Sep 21, 2020 at

Re: Adding repo to DockerHub for renaming Java8 image

Since we use the apache DockerHub account, the creation of new repositories needs to go through Infra. You can file a ticket with them like https://issues.apache.org/jira/browse/INFRA-20288. On Fri, Sep 18, 2020 at 10:22 AM Emily Ye wrote: > Hi dev@, > > Could someone with DockerHub credentials

Re: JIRA components

Why not use the existing sdk-py-core component and make Python types a label? On Thu, Sep 17, 2020 at 7:17 PM Kenneth Knowles wrote: > What does it mean? > > Kenn > > On Thu, Sep 17, 2020 at 6:18 PM Udi Meiri wrote: > >> Hi, I was going to create a Python types component in our JIRA. >>

Re: Infra Improvements

> Could someone create an ‘infra’ component in Jira (I don’t have access)? Could these issues go under the "testing" component? On Fri, Sep 11, 2020 at 3:11 PM Tyson Hamilton wrote: > Hi All, > > Below is a summary of the improvements made to Beam’s infrastructure in > the last few months.

Re: [VOTE] Release 2.24.0, release candidate #3

+1 (non-binding) Validated wordcount with Python 3.7.8 and Flink 1.10.0 (both loopback and using the Docker image). Also Python 3.7.8 loopback with an embedded Spark cluster. On Thu, Sep 10, 2020 at 2:32 PM Daniel Oliveira wrote: > By the way, most of the validation so far has covered Direct

Re: Enabling checkpointing while running Flink Runner

> But, from the configuration, there is no way to pass the checkpoint interval. Set the checkpointingInterval pipeline option. https://beam.apache.org/documentation/runners/flink/ On Wed, Sep 9, 2020 at 4:44 AM Sruthi Sree Kumar wrote: > Hi, > > How do we enable checkpointing for Flink

Re: [ANNOUNCE] Beam 2.23.0 Released

Hi Eleanore, there have been no changes to Beam's supported Flink versions since Beam 2.21.0. Beam supports Flink 1.8, 1.9, and 1.10. If you are looking for Flink 1.11 support, I didn't find an existing issue, so I filed https://issues.apache.org/jira/browse/BEAM-10612. On Thu, Jul 30, 2020 at

Re: [VOTE] Release 2.23.0, release candidate #2

+1 Ran Python wordcount {2.7, 3.7} x {Spark, Flink 1.10} job server containers. On Thu, Jul 23, 2020 at 10:42 AM Robert Bradshaw wrote: > +1 (binding) > > I validated the hashes and signatures of all the release artifacts, and > that the source tarball matches github > at

Re: Go error when building containers

I had a similar issue [1] that I resolved by running Brian's shell one-liner. However, it also deleted a bunch of checked-in vendor directories, which is presumably undesirable. > I thought there was a clean gradle target already, for the go gradle plugin. But that's probably mistaken.

Re: [BROKEN] Please add "Fix Version" when resolving or closing Jiras

> we could simplify those to a single state and let the Resolution field be the one source of truth for the nature of the resolution. +1 On Thu, Jul 23, 2020 at 2:14 PM Kenneth Knowles wrote: > This is unfortunate :-( > > Also unfortunate: I have just enough permissions to mess it up and not >

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

I prefer [3]. On Tue, Jul 14, 2020 at 10:53 AM Ning Kang wrote: > Hi everyone, > > Last week, I sent a design doc > > and proposals in this email thread >

Re: Versioning published Java containers

I'd be interested in looking at >> test coverage. The Flink & Spark portable ValidatesRunner suites use >> EMBEDDED environment, so they don't exercise the container. The first >> testing of the Java SDK harness container against the Python-based >> Universal Local Runner i

Re: Versioning published Java containers

My main question is, are we confident the Java 11 container is ready to release? AFAIK there are still a number of issues blocking full Java 11 support (cf [1] ; not sure how many of these, if any, affect the SDK harness specifically though.) For

Re: Errorprone plugin fails for release branches <2.22.0

> To fix on previous release branches, we would need to make a new release, is it not? Since hashes would change.. Would it be alright to patch the release branches on Github and leave the released source as-is? Github release branches themselves aren't release artifacts, so I think it should be

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

t; https://github.com/apache/beam/pull/11877). You should be able to >>>>>>>> build the wheel files using this github action instead of using the >>>>>>>> beam-wheels repo and Travis. Please give it a try during the release >>>>>>>> process. >

Re: beam submit TFX on yarn

Beam Python does not yet work with Spark on yarn. See https://issues.apache.org/jira/browse/BEAM-8970 for details. On Tue, Jul 7, 2020 at 8:52 PM sxqjq wrote: > > I forget， java can use spark-submit commit,but I use Python language > > > > - 原始邮件 - > > > *发件人：*sxqjq > >

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Thanks for all your contributions Aizhamal :) On Mon, Jun 29, 2020 at 9:56 AM Kenneth Knowles wrote: > Please join me and the rest of the Beam PMC in welcoming a new committer: > Aizhamal Nurmamat kyzy > > Over the last 15 months or so, Aizhamal has driven many efforts in the > Beam community

Re: Running Beam pipeline using Spark on YARN

t; "--environment_config={\"command\": \"./boot\"}")] > > I used my own boot script for customized python packaging. WIth this setup > I was able to get a simple hello-world program running. I haven't tried to > run the job server separately from the AM yet. So

Re: Running Beam pipeline using Spark on YARN

Hi Kamil, there is a JIRA for this: https://issues.apache.org/jira/browse/BEAM-8970 It's theoretically possible but remains untested as far as I know :) As I indicated in a comment, you can set --output_executable_path to create a jar that you can then submit to yarn via spark-submit. If you can

Re: Seasons of Technical Communications Project

Hi Vikas, Thank you for the introduction and your interest to work on Apache Beam documentation with Season of Docs. To participate in the program you need to follow the guides here [1] [2]. If you are new to the program, we suggest: 1. Start by studying our proposed project ideas and

Re: [Question] What are the supported and recommended JDK and Python versions

annual release cycle. > > On Tue, Jun 16, 2020, 4:37 PM Kyle Weaver wrote: > >> The Java [1] and Python [2] quickstart pages list these >> requirements, among other places. Even if we add a table, there's no >> guarantee that people will actually look at it be

Re: [Question] What are the supported and recommended JDK and Python versions

The Java [1] and Python [2] quickstart pages list these requirements, among other places. Even if we add a table, there's no guarantee that people will actually look at it before asking these questions on Stack Overflow. It might help though if we also add the supported versions to the Java and

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Sounds good, thanks Valentyn! On Mon, Jun 15, 2020 at 12:31 PM Valentyn Tymofieiev wrote: > Hi all, > > According to the Beam release calendar [1], the next (2.23.0) release > branch cut is scheduled for July 1. > > I would be happy to help with this release and volunteer myself to be the >

Re: Season of Docs: Interested in working with Apache Beam

Hi Cynthia, Thank you for the introduction and your interest to work on Apache Beam documentation with Season of Docs. To participate in the program you need to follow the guides here [1] [2]. If you are new to the program, we suggest: Start by studying our proposed project ideas and expected

Re: Application For Google Summer Of Docs

Hi Ayush, Thank you for the introduction and your interest to work on Apache Beam documentation with Season of Docs. To participate in the program you need to follow the guides here [1] [2]. If you are new to the program, we suggest: Start by studying our proposed project ideas and expected

Re: python precommit error - google-auth depenedency?

The fix to google-auth has been merged. Is the plan just to wait until a new version of google-auth is released and ignore the failing tests until then? (btw I filed a JIRA for this before I realized it was already being discussed here: https://issues.apache.org/jira/browse/BEAM-10232) On Wed,

  1   2   3   >