Re: GCP libraries up-to-date versions in Java

If we have a bunch of leftover junk in .m2 will that pollute the analysis? Should we rm -rf ~/.m2 first or does it work well anyhow? On Wed, Nov 20, 2019 at 4:52 PM Luke Cwik wrote: > I took a look at the linkage checker and have opened up this PR[1] to > allow contributors to aid in performing

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

] Beaver [ ] Hedgehog [ x] Lemur [x ] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [x ] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Tue, Nov 19, 2019 at 6:43 PM Kenneth Knowles wrote: > Please cast your votes of approval [1] for animals you would support as > Beam mascot. The

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

Responses are here . On Thu, Nov 21, 2019 at 7:05 PM David Cavazos wrote: > [x] Beaver > [x] Hedgehog > [x] Lemur > [x] Owl > [ ] Salmon > [ ] Trout > [ ] Robot dinosaur > [x] Firefly >

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[x] Beaver [x] Hedgehog [x] Lemur [x] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [x] Firefly [x] Cuttlefish [x] Dumbo Octopus [ ] Angler fish I created this Google Form if everyone

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [ ] Hedgehog [ ] Lemur [ ] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [ ] Cuttlefish [x] Dumbo Octopus [ ] Angler fish On Thu, Nov 21, 2019 at 6:14 PM Chamikara Jayalath wrote: > [X] Beaver > [ ] Hedgehog > [ ] Lemur > [X] Owl > [ ] Salmon > [ ] Trout > [ ] Robot

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[X] Beaver [ ] Hedgehog [ ] Lemur [X] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [X ] Cuttlefish [X ] Dumbo Octopus [ X] Angler fish Thanks, Cham On Thu, Nov 21, 2019 at 1:43 PM Michał Walenia wrote: > [X] Beaver > [ ] Hedgehog > [X] Lemur > [X] Owl > [ ] Salmon > [ ] Trout > [X]

Re: Pipeline AttributeError on Python3

We are currently verifying the patch. Will report back tomorrow. On Thu, Nov 21, 2019 at 8:40 AM Valentyn Tymofieiev wrote: > That would be helpful, thanks a lot! It should be a straightforward patch. > Also, thanks Guenther, for sharing your investigation on >

Re: Side inputs not working in CombineGlobally (Python)

Thanks for the report. You can work around this by specifying without_defaults() on the global combine (as the default is computed at pipeline construction time). Note that even in the cases where it works, it disables combiner lifting, so side inputs in combiners is generally discouraged. And

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

I wasn't clear: this has worked for a long time. I wasn't suggesting a new feature. I didn't look up the exact incantation. Kenn On Thu, Nov 21, 2019, 14:49 Robert Bradshaw wrote: > On Thu, Nov 21, 2019 at 2:46 PM Reuven Lax wrote: > > > > That is racy. It could also be that the side input

Re: [Discuss] Beam Summit 2020 Dates & locations

Maria put together this documents with related industry conferences [1], it would make sense to choose a time that doesn't conflict with other events around projects close to Beam. How about for June 21-22 (around Spark Summit) for North America, and October 5-6 or October 12-13 for Europe? [1]

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

On Thu, Nov 21, 2019 at 2:46 PM Reuven Lax wrote: > > That is racy. It could also be that the side input hasn't been computed yet > but will be available in a second. It's no more racy than any other side input lookup (as we still respect not declaring a window is empty until the watermark

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax wrote: > > In particular, since windows can be data based (e.g. session windows) the set > of windows is not always knowable in advance. It's [IntervalWindow(start, end) for start in range(MIN_TIMESTAMP, MAX_TIMESTAMP) for end in range(MIN_TIMESTAMP,

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

That is racy. It could also be that the side input hasn't been computed yet but will be available in a second. On Thu, Nov 21, 2019 at 2:37 PM Kenneth Knowles wrote: > It can work in pull mode when it is a side input, via > Combine.globally().asSingletonView(). Then the default is produced when

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

It can work in pull mode when it is a side input, via Combine.globally().asSingletonView(). Then the default is produced when the side input is requested and there is no data found. Kenn On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax wrote: > In particular, since windows can be data based (e.g.

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[X] Beaver [ ] Hedgehog [X] Lemur [X] Owl [ ] Salmon [ ] Trout [X] Robot dinosaur [X] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Thu, Nov 21, 2019 at 1:11 PM Aizhamal Nurmamat kyzy wrote: > [ ] Beaver > [X] Hedgehog > [ ] Lemur > [ ] Owl > [ ] Salmon > [ ] Trout > [ ] Robot

Re: Contributor permission for Beam Jira tickets

Done. Welcome! On Wed, Nov 20, 2019 at 3:58 PM Chuck Yang wrote: > Hello Beam developers, > > This is Chun from Cruise. I would like to make some contributions to > the Python SDK for Beam. Can someone add me as a contributor in the > Beam Jira? My username is cccyang. > > Thanks! > Chun > > --

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [X] Hedgehog [ ] Lemur [ ] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [X] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Thu, Nov 21, 2019 at 11:21 AM Robert Burke wrote: > [ X] Beaver > [] Hedgehog > [ x] Lemur > [ X] Owl > [ ] Salmon > [ ] Trout > [ ] Robot dinosaur >

Re: goVet and clickHouse tests failing

It would be nice if this failed gracefully, indicating what dependency is not installed. Kenn On Thu, Nov 21, 2019 at 4:38 AM Elliotte Rusty Harold wrote: > That would explain it. I didn't have Docker installed. I do now, so > let's see if it passes. Ultimately the root cause for a lot of this

Re: Portable runner bundle scheduling (Streaming/Python/Flink)

Hi Luke, Thanks for the background and it is exciting to see the progress on the SDF side. It will help with this use case and many other challenges. I imagine the Python user code would be able to determine that it is bogged down with high latency record processing (based on the duration it

Re: Default values not supported in Combine.globally() if not windowed by GlobalWindows

The semantics are a bit undefined--the sane extension of the model to support this is that the default value would show up in every window (otherwise which window would the default value belong to), but we don't have support for enumerating windows (or truncating infinite collections in batch

Re: [PROPOSAL] Preparing for Beam 2.18 release

Thanks Cham. Tomo, if there are any dependencies you believe are blockers please mark them. Also, only the sub-tasks

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congratulations, Daniel!!! On Thu, Nov 21, 2019 at 2:17 AM Connell O'Callaghan wrote: > Well done - congratulations Daniel!!! > Kenn thank you for sharing!!! > > On Thu, Nov 21, 2019 at 1:32 AM Robin Qiu wrote: > >> Congrats, Daniel! >> >> On Thu, Nov 21, 2019 at 10:11 AM Gleb Kanterov wrote:

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [ ] Hedgehog [x] Lemur [x] Owl [ ] Salmon [ ] Trout [x] Robot dinosaur! [ ] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish czw., 21 lis 2019 o 00:44 Augustin Lafanechere < augustin.lafanech...@kapten.com> napisał(a): > [ ] Beaver > [ ] Hedgehog > [ ] Lemur > [ ] Owl > [x]

Re: [PROPOSAL] Add support for writing flattened schemas to pubsub

A PR is up here [1]. Gleb: If I understand what you're saying, I think it's already implemented the way you're describing - PubsubIOJsonTable [2] is just a thin wrapper that connects PubsubIO with Beam SQL tables. Alex/Kenn: I agree with everything you've said :) The hard-coded event_timestamp is

Re: Pipeline AttributeError on Python3

That would be helpful, thanks a lot! It should be a straightforward patch. Also, thanks Guenther, for sharing your investigation on https://bugs.python.org/issue34572, it was very helpful. On Thu, Nov 21, 2019 at 8:25 AM Thomas Weise wrote: > Valentyn, thanks a lot for following up on this. > >

Re: Pipeline AttributeError on Python3

Valentyn, thanks a lot for following up on this. If the change can be cherry picked in isolation, we should be able to verify this soon (with 2.16). On Thu, Nov 21, 2019 at 8:12 AM Valentyn Tymofieiev wrote: > To close the loop here: To my knowledge this issue affects all Python 3 > users of

Re: Pipeline AttributeError on Python3

To close the loop here: To my knowledge this issue affects all Python 3 users of Portable Flink/Spark runners, and Dataflow Python Streaming users, including users on Python 3.7.3 and newer versions. The issue is addressed on Beam master, and we have a cherry-pick out for Beam 2.17.0. Workaround

Default values not supported in Combine.globally() if not windowed by GlobalWindows

Hi all, When I run a Combine.globally() transform with a default value set, on a PCollection windowed by a FixedWindow, I got this error message: java.lang.IllegalStateException: Default values are not supported in Combine.globally() if the input PCollection is not windowed by GlobalWindows ...

Re: goVet and clickHouse tests failing

That would explain it. I didn't have Docker installed. I do now, so let's see if it passes. Ultimately the root cause for a lot of this seems to be incomplete and conflicting docs. I hope we can invest some effort into improving this. On Thu, Nov 21, 2019 at 6:27 AM Gleb Kanterov wrote: > >

Re: goVet and clickHouse tests failing

Tentatively, the goVet issue does seem to have been an issue with my Go install I have now cleaned up. The clickhouse issue remains, as do several others I'm working through. I've filed https://issues.apache.org/jira/browse/BEAM-8798 to consolidate and update the instructions for getting to a

Re: goVet and clickHouse tests failing

:sdks:java:io:clickhouse:test is using testcontainers. Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container. Therefore, ClickHouse tests require a local

goVet and clickHouse tests failing

I'm slowly working my way through getting the tests to run and pass. We have a lot of work to do on the contributing docs to explain how to setup and run the build. There's clearly a lot of knowledge in developers' heads and workstations that hasn't yet made it into the docs. The latest is a

Re: [ANNOUNCE] New committer: Daniel Oliveira

Well done - congratulations Daniel!!! Kenn thank you for sharing!!! On Thu, Nov 21, 2019 at 1:32 AM Robin Qiu wrote: > Congrats, Daniel! > > On Thu, Nov 21, 2019 at 10:11 AM Gleb Kanterov wrote: > >> Congratulations! >> >> On Thu, Nov 21, 2019 at 6:24 AM Thomas Weise wrote: >> >>>

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congrats, Daniel! On Thu, Nov 21, 2019 at 10:11 AM Gleb Kanterov wrote: > Congratulations! > > On Thu, Nov 21, 2019 at 6:24 AM Thomas Weise wrote: > >> Congratulations! >> >> >> On Wed, Nov 20, 2019, 7:56 PM Chamikara Jayalath >> wrote: >> >>> Congrats!! >>> >>> On Wed, Nov 20, 2019 at 5:21

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congratulations! On Thu, Nov 21, 2019 at 6:24 AM Thomas Weise wrote: > Congratulations! > > > On Wed, Nov 20, 2019, 7:56 PM Chamikara Jayalath > wrote: > >> Congrats!! >> >> On Wed, Nov 20, 2019 at 5:21 PM Daniel Oliveira >> wrote: >> >>> Thank you everyone! I won't let you down. o7 >>> >>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [ ] Hedgehog [ ] Lemur [ ] Owl [x] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [x ] Angler fish > Le 20 nov. 2019 à 13:38, Maximilian Michels a écrit : > > [ ] Beaver > [ ] Hedgehog > [x] Lemur > [ ] Owl > [ ] Salmon > [ ] Trout > [ ] Robot