Re: Beam Emitted Metrics Reference

Also seems like each IO would benefit from an entry in a Transform Catalog with description of any IO-specific metrics it emits. (technically these may not be what you mean by "framework-emitted metrics") Kenn On Mon, Mar 2, 2020 at 9:40 AM Alex Amato wrote: > MonitoringInfoSpecs is

Re: Error logging from fn_api_runners

Yeah, this was an oversight on my part. I don't think we need to log this at all. https://github.com/apache/beam/pull/11021 for anyone to look at. On Mon, Mar 2, 2020 at 2:44 PM Heejong Lee wrote: > > I think it should be either info or debug but not error. > > On Mon, Mar 2, 2020 at 2:35 PM

Re: Python Static Typing: Next Steps

It seems people are conflating git pre-commit hooks (which IMHO should ideally be in the sub-second range, and run when an author does "git commit") with jenkins pre-commit tests (for which minutes is nothing compared to what we already do). I am +1 to adding mypy to the latter for sure, and think

Re: Java SplittableDoFn Watermark API

I don't have a strong preference for using a provider/having a set of tightly coupled methods in Java, other than that we be consistent (and we already use the methods style for restrictions). On Mon, Mar 2, 2020 at 3:32 PM Luke Cwik wrote: > > Jan, there are some parts of Apache Beam the

Re: Python Static Typing: Next Steps

Off-topic: Python lint via pre-commit should be much faster. (I wrote my own modified-file-only lint in the past) On Mon, Mar 2, 2020 at 2:08 PM Kyle Weaver wrote: > > Python lint takes 4-5mins to complete. I think if the mypy analysis is > really on the order of 10s, the additional time won't

Re: Java SplittableDoFn Watermark API

Jan, there are some parts of Apache Beam the watermarks package will likely rely on (@Experimental annotation, javadoc links) but fundamentally should not rely on core and someone could create a separate package for this. Ismael, the unification of bounded/unbounded within SplittableDoFn has

Re: Error logging from fn_api_runners

I think it should be either info or debug but not error. On Mon, Mar 2, 2020 at 2:35 PM Ning Kang wrote: > Hi, > > I just observed some error level loggings like these: > ``` > ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers > {'worker_5': > at 0x127fdaa58>} >

Error logging from fn_api_runners

Hi, I just observed some error level loggings like these: ``` ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers {'worker_5': } ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers {'worker_5': } ERROR:apache_beam.runners.portability.fn_api_runner:created 1

Re: Python Static Typing: Next Steps

> Python lint takes 4-5mins to complete. I think if the mypy analysis is really on the order of 10s, the additional time won't matter and could always be enabled. +1 of course it would be nice to make mypy as fast as possible, but I don't think speed needs to be a blocker. The productivity gains

Re: Python Static Typing: Next Steps

Python lint takes 4-5mins to complete. I think if the mypy analysis is really on the order of 10s, the additional time won't matter and could always be enabled. On Mon, Mar 2, 2020 at 1:21 PM Chad Dombrova wrote: > I believe that mypy via pre-commit hook will be faster than 10s since it >> only

Re: Python Static Typing: Next Steps

> > I believe that mypy via pre-commit hook will be faster than 10s since it > only applies to modified files. > Correct, with a few caveats: - pre-commit can be setup to only run if a python file changes. so modifying a java file won't trigger mypy to run. - if *any* python file

Fwd: Google Summer of Code 2020 Mentor Registration

FYI -- Forwarded message - From: Maxim Solodovnik Date: Thu, Feb 27, 2020 at 6:08 PM Subject: Google Summer of Code 2020 Mentor Registration To: Dear PMCs, I'm happy to announce that the ASF has made it onto the list of accepted organizations for Google Summer of Code 2020!

Re: [VOTE][BIP-1] Beam Schema Options

Anyone keen to review this PR: https://github.com/apache/beam/pull/10413 without this foundation I can't continue with the rest. _/ _/ Alex Van Boxel On Fri, Feb 28, 2020 at 11:40 PM Alex Van Boxel wrote: > Thank you everyone for voting: Accepted by majority vote +1 (7 votes, 3 > binding),

Re: Python Static Typing: Next Steps

Let's go forward with this and see. I volunteer to help as well. I believe that mypy via pre-commit hook will be faster than 10s since it only applies to modified files. On Mon, Mar 2, 2020 at 10:53 AM Robert Bradshaw wrote: > +1 > > We should enable this on jenkins, plus trivial instructions

Re: Python Static Typing: Next Steps

+1 We should enable this on jenkins, plus trivial instructions (ideally a one-liner tox command) to run it locally. Hopefully the errors will be easy enough for contributors to figure out (in particular local to and commensurate in complexity with the code that they're editing), and I agree it's

Re: Python Static Typing: Next Steps

+1 The typing information has really helped me several times figuring out that API contracts and expected types. On Mon, Mar 2, 2020 at 9:54 AM Pablo Estrada wrote: > I am in favor of enabling the test, and also am happy to start answering > questions too. > Thanks so much Chad for leading

Re: Python Static Typing: Next Steps

I am in favor of enabling the test, and also am happy to start answering questions too. Thanks so much Chad for leading this. Best -P. On Mon, Mar 2, 2020 at 9:44 AM Chad Dombrova wrote: > Good news everyone! > We nearly have the full beam codebase passing in mypy. > > As we are now approaching

Python Static Typing: Next Steps

Good news everyone! We nearly have the full beam codebase passing in mypy. As we are now approaching the zero-error event horizon, I'd like to open up a discussion around enabling mypy in the PythonLint job. Every day or so a PR is merged that introduces some new mypy errors, so enabling this

Re: Beam Emitted Metrics Reference

MonitoringInfoSpecs is effectively a list of metrics , but its purpose is to simply define how SDKs should populate MonitoringInfo protos for a RunnerHarness to interpret.

Re: JdbcIO for writing to Dynamic Schemas in Postgres

Hi You have the setPrepareStatement() method where you define the target tables. However, it’s in the same database (datasource) per pipeline. You can define several datasources and use a different datasource in each JdbcIO write. Meaning that you can divide in sub pipelines. Regards JB > Le

Re: GroupIntoBatches not Working properly for Direct Runner Java

Input : a-1, Timestamp : 1582994620366 Input : c-2, Timestamp : 1582994620367 Input : e-3, Timestamp : 1582994620367 Input : d-4, Timestamp : 1582994620367 Input : e-5, Timestamp : 1582994620367 Input : b-6, Timestamp : 1582994620368 Input : a-7, Timestamp : 1582994620368 Output : Timestamp :

JdbcIO for writing to Dynamic Schemas in Postgres

Hey folks, Can we use JdbcIO for writing data to multiple Schemas(For Postgres Database) dynamically using Apache beam Java Framework? Currently, I can't find any property that I could set to JdbcIO transform for providing schema or maybe I am missing something. Thanks

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Congrats Kamil!! On Mon, 2 Mar 2020 at 16:16, Karolina Rosół wrote: > Congratulations Kamil! Well deserved :-) > > Karolina Rosół > Polidea | Project Manager > > M: +48 606 630 236 <+48606630236> > E: karolina.ro...@polidea.com > [image: Polidea]

Re: Permission to self-assign JIRAs

Welcome, you have been added. On Mon, Mar 2, 2020 at 3:57 AM Jozef Vilcek wrote: > Can I please get a permission in JIRA for `jvilcek` user to self assign > JIRAs? >

Upcoming Apache Beam meetups in Warsaw

Hi everyone, I'm Project Manager at Polidea and work closely with three Apache Beam committers (Katarzyna Kucharczyk, Kamil Wasilewski and Michał Walenia). Together with folks from Polidea we'd like to announce our plans towards the upcoming Apache Beam meetups in Warsaw. The next date for the

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Congratulations Kamil! Well deserved :-) Karolina Rosół Polidea | Project Manager M: +48 606 630 236 <+48606630236> E: karolina.ro...@polidea.com [image: Polidea] Check out our projects! [image: Github]

Re: KafkaIO: Configurable timeout for setupInitialOffset()

Thanks Ismael! On Mon, Mar 2, 2020 at 2:15 PM Ismaël Mejía wrote: > Done, also assigned the issue you mentioned in the previous email to you. > > On Mon, Mar 2, 2020 at 12:56 PM Jozef Vilcek > wrote: > >> Recently I had a problem with Beam pipeline unable to start due to >> unhealthy broker in

Re: KafkaIO: Configurable timeout for setupInitialOffset()

Done, also assigned the issue you mentioned in the previous email to you. On Mon, Mar 2, 2020 at 12:56 PM Jozef Vilcek wrote: > Recently I had a problem with Beam pipeline unable to start due to > unhealthy broker in the list of configured bootstrap servers. I have > created a JIRA for it and

Beam Dependency Check Report (2020-03-02)

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Permission to self-assign JIRAs

Can I please get a permission in JIRA for `jvilcek` user to self assign JIRAs?

KafkaIO: Configurable timeout for setupInitialOffset()

Recently I had a problem with Beam pipeline unable to start due to unhealthy broker in the list of configured bootstrap servers. I have created a JIRA for it and plan to work on the fix. https://issues.apache.org/jira/browse/BEAM-9420 Please let me know in case it does not make sense of should

Re: Beam Emitted Metrics Reference

Hi, There is a doc about metrics here: https://beam.apache.org/documentation/programming-guide/#metrics You can also export the metrics to sinks (REST http endpoint and Graphite), see MetricsOptions class for configuration. Still, there is no doc for export on website, I'll add some Best

Re: GroupIntoBatches not Working properly for Direct Runner Java

Hi, +1 to what Kenn asked: your pipeline is in streaming mode and GIB preserves windowing, the elements are buffered until one of these conditions are true: batchsize reached or end of window. I your case I think it is the second one. Best Etienne On 28/02/2020 19:15, Kenneth Knowles

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Thank you all! I am very happy to be a part of the community :) On Mon, Mar 2, 2020 at 9:45 AM Ryan Skraba wrote: > Congratulations Kamil! > > On Mon, Mar 2, 2020 at 8:06 AM Michał Walenia > wrote: > >> Congratulations! >> >> On Sun, Mar 1, 2020 at 2:55 AM Reza Rokni wrote: >> >>>

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Congratulations Kamil! On Mon, Mar 2, 2020 at 8:06 AM Michał Walenia wrote: > Congratulations! > > On Sun, Mar 1, 2020 at 2:55 AM Reza Rokni wrote: > >> Congratilation Kamil >> >> On Sat, 29 Feb 2020, 06:18 Udi Meiri, wrote: >> >>> Welcome Kamil! >>> >>> On Fri, Feb 28, 2020 at 12:53 PM Mark