Re: Use Coder message for cross-lang ExternalConfigurationPayload?

2020-08-03 Thread Brian Hulette
I've opened BEAM-10571 [1] for this, and I'm most of the way to an implementation now. Aiming to have it done before the 2.24.0 cut since it will be the last release with python 2 support. [1] https://issues.apache.org/jira/browse/BEAM-10571 On Wed, Jul 15, 2020 at 9:03 AM Chamikara Jayalath

Re: Unknown accumulator coder error when running cross-language SpannerIO Write

2020-08-03 Thread Brian Hulette
The DirectRunner error looks like it's because the FnApiRunner doesn't support SDF. What is the coder id for the Flink error? It looks like the full stack trace should contain it. On Mon, Aug 3, 2020 at 10:09 AM Piotr Szuberski wrote: > I'm Writing SpannerIO.Write cross-language transform and

Re: Request Throttling in OSSIO

2020-08-03 Thread Luke Cwik
Since you are working on a new connector I would very strongly suggest writing it as a splittable DoFn instead of an UnboundedSource. See this thread[1] about additional details and some caveats on the recommendation. 1) You can return false from advance and the runner will execute advance at

Re: [External] Re: DISCUSS: Sorted MapState API

2020-08-03 Thread Catlyn Kong
Hey folks, Sry I'm late to this thread but this might be very helpful for the problem we're dealing with. Do we have a design doc or a jira ticket I can follow? Cheers, Catlyn On Thu, Jun 18, 2020 at 1:11 PM Jan Lukavský wrote: > My questions were just an example. I fully agree there is a

Re: Monitoring performance for releases

2020-08-03 Thread Robert Bradshaw
I have to admit I still have some qualms about tying detecting and fixing performance regressions as part of the release process (which is onerous enough as it is). Instead, I think we'd be better off with a separate process to detect and triage performance issues, which, when they occur, may

Re: Stateful Pardo Question

2020-08-03 Thread jmac...@godaddy.com
Yeah, unless I am misunderstanding something. The output from my repro code shows event timestamp and the context timestamp every time we process an event. Receiving event at: 2000-01-01T00:00:00.000Z Resetting timer to : 2000-01-01T00:15:00.000Z Receiving event at: 2000-01-01T00:05:00.000Z

Re: Development on Windows

2020-08-03 Thread Luke Cwik
It looks like Windows file locking is getting in your way since one gradle task is reading a file which is produced as a temporary file that is part of another gradle test task and that test task is trying to delete it. Try running the build with --no-parallel to disable parallel builds[1]. This

Re: Beam Dependency Check Report (2020-08-03)

2020-08-03 Thread Damian Gadomski
That's probably caused by this [1] PR, workspace had been deleted before the email was sent. +Udi Meiri Moving the workspace clean up to the very end of post-build actions should help. [1] https://github.com/apache/beam/pull/12326 On Mon, Aug 3, 2020 at 5:42 PM Brian Hulette wrote: > Does

Fwd: Contributor permission for Beam Jira tickets

2020-08-03 Thread Varun Sharma
Hi, This is Varun from THG. I have created a couple of tickets related to improvements in Beam Java SDK. Could someone add me as a contributor for Beam's Jira issue tracker? I would like to assign tickets for my work. My JIRA id- varun.sharma Regards Varun Sharma -- Forwarded message

Unknown accumulator coder error when running cross-language SpannerIO Write

2020-08-03 Thread Piotr Szuberski
I'm Writing SpannerIO.Write cross-language transform and when I try to run it from python I receive errors: On Flink: apache_beam.utils.subprocess_server: INFO: b'Caused by: java.lang.IllegalArgumentException: Transform external_1HolderCoder uses unknown accumulator coder id %s'

Re: Stateful Pardo Question

2020-08-03 Thread Reuven Lax
Are you sure that there is a 15 minute gap in your data? On Mon, Aug 3, 2020 at 6:20 AM jmac...@godaddy.com wrote: > I am confused about the behavior of timers on a simple stateful pardo. I > have put together a little repro here: > https://github.com/randomsamples/pardo_repro > > > > I

[BEAM-10292] change proposal to DefaultFilenamePolicy.ParamsCoder

2020-08-03 Thread David Janíček
Hello everyone, I've reported an issue https://issues.apache.org/jira/browse/BEAM-10292 which is about broken DefaultFilenamePolicy.ParamsCoder behavior. DefaultFilenamePolicy.ParamsCoder loses information whether DefaultFilenamePolicy.Params's baseFilename resource is file or directory on some

Re: Email about build runs on my fork.

2020-08-03 Thread Alex Amato
Thanks, havent seen any emails since rebasing to master. On Sun, Aug 2, 2020 at 5:09 AM Tobiasz Kędzierski < tobiasz.kedzier...@polidea.com> wrote: > Hi Alex, > > After rebase on the latest master scheduled workflow should not run, > condition for scheduled job was extended to prevent situations

Re: Monitoring performance for releases

2020-08-03 Thread Maximilian Michels
Here a first version of the updated release guide: https://github.com/apache/beam/pull/12455 Feel free to comment. -Max On 29.07.20 17:27, Maximilian Michels wrote: Thanks! I'm following up with this PR to display the Flink Pardo streaming data: https://github.com/apache/beam/pull/12408

Re: Beam Dependency Check Report (2020-08-03)

2020-08-03 Thread Brian Hulette
Does anyone know what went wrong here? It looks like the associated jenkins job [1] succeeded, and produced beam-dependency-check-report.html [1] https://ci-beam.apache.org/job/beam_Dependency_Check/279/ On Mon, Aug 3, 2020 at 5:28 AM Apache Jenkins Server < jenk...@builds.apache.org> wrote: >

Stateful Pardo Question

2020-08-03 Thread jmac...@godaddy.com
I am confused about the behavior of timers on a simple stateful pardo. I have put together a little repro here: https://github.com/randomsamples/pardo_repro I basically want to build something like a session window, accumulating events until quiescence of the stream for a given key and gap

Beam Dependency Check Report (2020-08-03)

2020-08-03 Thread Apache Jenkins Server
ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist