[BigQuery] TableRowJsonCoder question

Hi all, While playing with BigQueryIO I noticed something. When we create a TableRow (e.g. in a row function in bigQueryIO) using new TableRow().set(), for ex a long gets boxed into a Long. But when it is encoded using TableRowJsonCoder and then re-read it might be decoded as an Integer if

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Hi Pablo, I took only a quick look. "- The JAR from the non-LGPL findbugs does not contain the SuppressFBWarnings annotation" Unless I misunderstand you it looks like SuppressFBWarnings was added in Stephen's version in this commit [1] which was introduced in version 2.0.3-1 - I've checked is

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Hi Tim, you're right. Thanks for pointing that out. There's just one problem that I'm running into now: The 3.0.1-1 version does not seem to be available in Maven Central[1]. Looking at the website, I am not quite sure if there's another repository where they do stage the newer versions?[2] -P

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Hi Pablo, I'm afraid I couldn't find one either... there is an issue about it [1] which is old so it doesn't look likely to be resolved either. If you have time (sorry I am a bit busy) could you please verify the version does work if you install that version locally? I know the maven version of

Beam Dependency Check Report (2018-06-13)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Later Version Current Version Release Date Later Version Release Date google-cloud-bigquery0.25.01.3.02017-06-262018-06-08 httplib20.9.20.11.32015-09-282018-03-30 High Priority

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

>From my understanding, the @SuppressFBWarnings usage is in a dependency (ByteBuddy) rather than directly in our code; so we're not able to modify the usage. Pablo, feel free to disable failOnWarning for the sdks-java-core project temporarily. This isn't a major regression since we've only

Re: Beam Dependency Check Report (2018-06-13)

Hi, I want to follow up and explain this email. This is a sample email that reports the results of Beam SDK dependency check, which was proposed here . The goal is finding updates for

Re: Beam Dependency Check Report (2018-06-13)

Ahh very nice... thanks Yifan & Cham! Lots of old dependencies eh... very interesting. Best -P. On Wed, Jun 13, 2018 at 7:45 AM Yifan Zou wrote: > Hi, > > > I want to follow up and explain this email. > > > This is a sample email that reports the results of Beam SDK dependency > check, which

Re: [FYI] New Apache Beam Swag Store!

Great ! Thanks Gris and Matthias for putting this in place. Hope to get that hoodie soon. As a suggestion, more colors too, and eventually a t-shirt just with the big B logo. On Mon, Jun 11, 2018 at 6:50 PM Mikhail Gryzykhin wrote: > > That's nice! > > More colors are appreciated :) > > --Mikhail

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

Thanks for the example Romain. I took a look through it and was wondering whether it is only the root objects in the deserialization tree that need to implement SerializableService? Do lots of things need to implement SerializableService typically? What do you do with types that you don't control

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

if you have a javaagent you can otherwise you can't but beam can proxy all instances it sees which would be enough while the transforms* don't create their own classloaders without reusing the TCCL. On the deserialization side it is easy since it is beam land and it "just "needs to find back or

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Precommits for PR https://github.com/apache/beam/pull/5609 are now passing. For now I've simply set failOnWarning to false to cherrypick into the release, and fix in master later on. Best -P. On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner wrote: > From my understanding, the @SuppressFBWarnings

Re: Beam SQL Pipeline Options

I've turned this into a PR, more discussion going on over there: https://github.com/apache/beam/pull/5592 Andrew On Wed, Jun 6, 2018 at 9:46 PM Kenneth Knowles wrote: > This is a nice short design discussion doc, and perhaps a cooler piece of > news hidden in the paragraph :-) > > Kenn > > On

Re: Building and visualizing the Beam SQL graph

One of my goals is to make the graph easier to read and map back to the SQL EXPLAIN output. The way the graph is currently built (`toPTransform` vs `toPCollection`) does make a big difference in that graph. I think it is also important to have a common function to do the apply with consistent

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

I'm assuming that you have control over the application environment. Would it be possible to replace the ObjectInputStream that the JVM provides with your own version that uses the thread context class loader and manage the classloader per thread depending on what "application" owns that thread?

[Call for Speakers] Deep Learning in Production Meetup, Boston Area on June 26th

Hi Beam Community, Eila Arich-Landkof (from OrielResearch) and I are co-hosting the next edition of the Deep Learning in Production Meetup on June 26th at the Google Office in Cambridge, Massachusetts. *We are looking for speakers who would

Re: Beam Dependency Check Report (2018-06-13)

Thanks Yifan, this is great! My unsolicited feedback: - Could it warn against dependencies that did not get updates for a long time? For python there were examples of a dependency being abandoned by its own developers and it took us a while to figure it out and switch to maintained one.

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

(answered inline) Le mer. 13 juin 2018 à 18:42, Lukasz Cwik a écrit : > Thanks for the example Romain. > > I took a look through it and was wondering whether it is only the root > objects in the deserialization tree that need to implement > SerializableService? > Do lots of things need to

Re: Beam Dependency Check Report (2018-06-13)

Nifty. Here's some unsolicited feedback: * The report gives a nice view of the data and leaves it as an exercise to the reader to do the math on each row (v0.25.0 to v1.3.0 = 1 major version behind, 2017-06-26 to 2018-06-08 = 1 year behind). I would find the report more digestable if these

Re: Beam Dependency Check Report (2018-06-13)

Thanks Yifan. On Wed, Jun 13, 2018 at 10:21 AM Ahmet Altay wrote: > Thanks Yifan, this is great! > > My unsolicited feedback: > - Could it warn against dependencies that did not get updates for a long > time? For python there were examples of a dependency being abandoned by its > own developers

Re: [FYI] New Apache Beam Swag Store!

Thanks All! To close the loop on the suggestions, I'll order more t-shirts in black so we have some options. G On Wed, 13 Jun 2018 at 08:39, Ismaël Mejía wrote: > Great ! Thanks Gris and Matthias for putting this in place. > Hope to get that hoodie soon. As a suggestion, more colors too, and

Fwd: Proposing interactive beam runner

Resending after subscribing to dev list. -- Forwarded message -- From: Sindy Li Date: Fri, Jun 8, 2018 at 5:57 PM Subject: Proposing interactive beam runner To: dev@beam.apache.org Cc: Harsh Vardhan , Chamikara Jayalath < chamik...@google.com>, Anand Iyer , Robert Bradshaw <

Re: Proposing interactive beam runner

*Thanks, Andrew!* *Here is a link to the demo on Youtube for people interested:* *https://www.youtube.com/watch?v=c5CjA1e3Cqw=youtu.be * On Wed, Jun 13, 2018 at 1:23 PM, Andrew Pilloud wrote: > This sounds really interesting, thanks for

Re: Proposing interactive beam runner

This is awesome, thanks Sindy! I hope that the questions related to portability will get resolved in a way that will allow to reuse some of the work for other interactive Beam experiences, including SQL as Andrew says, and providing a REPL e.g. for users of Scala or other JVM-based languages.

Re: Proposing interactive beam runner

This sounds really interesting, thanks for sharing! We've just begun to explore making Beam SQL interactive. The Interactive Runner you've proposed sounds like it would solve a bunch of the problems SQL faces as well. SQL is written in Java right now, so we can't immediately reuse any code.

Re: Proposal: keeping post-commit tests green

Hello everybody, Thanks everyone. I didn't receive any more feedback on the design proposal document [1] and I believe we've reached consensus. I've added implementation tasks in JIRA (BEAM-4559 [2]) and will start coding soon. As a recap, the high-level plan is: - Split existing

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky test. As JB is the release manager, I'l let him make the call on what to do about it. Best -P. On Wed, Jun 13, 2018 at 3:34 PM Ahmet Altay wrote: > I would vote for second option, not a release blocker and disable the test >

Re: Proposal: keeping post-commit tests green

Hi Ahmet, I've checked on tests status and most of other tests are green 98% of the time. So I feel that we do not need any explicit actions for those tests. However java tests seem to have most of the problems. So I moved it to requirements explicitly. I do not bring in fixing failing tests as

Re: Building and visualizing the Beam SQL graph

Another thing to consider is that we might return something like a "SqlPCollection" that is the PCollection plus additional metadata that is useful to the shell / enumerable converter (such as if the PCollection has a known finite size due to LIMIT, even if it is "unbounded", and the shell can

Re: Building and visualizing the Beam SQL graph

>From the visualization perspective I really loved the interactive runner demo where it shows the graph: https://www.youtube.com/watch?v=c5CjA1e3Cqw=27s On Wed, Jun 13, 2018 at 4:36 PM Kenneth Knowles wrote: > Another thing to consider is that we might return something like a > "SqlPCollection"

Re: Precommits broken?

Probably due to the precommit filter applied in #5611 ? On Wed, Jun 13, 2018 at 5:02 PM Andrew Pilloud wrote: > Looks like statuses got posted between me writing this email and sending > it. Still wondering why the python and go jobs appear to be

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Hey all, Currently we have 3 PRs supposed to be cherrypicked into RC2: Pablo: https://github.com/apache/beam/pull/5609 (merged) Udi: https://github.com/apache/beam/pull/5607 (open) Charles: https://github.com/apache/beam/pull/5636 (open) Boyuan On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada

Re: Beam Dependency Check Report (2018-06-13)

Thanks everyone for feedbacks! We will embed some text to explain the details of the report and guide people what to do with it at this point. Cham and I prepare to start updating those dependencies, and have them grouped and find them owners if possible. Also, we will try to automate JIRA filing

Re: Apache Beam June Newsletter

Thanks Gris! Lots of interesting things. Best -P. On Wed, Jun 13, 2018 at 4:40 PM Griselda Cuevas wrote: > Hi Beam Community! > > Here > > [1] > is the June Edition of our Apache Beam

Re: Beam Dependency Check Report (2018-06-13)

Thanks Kenn. These are all direct dependencies. On Wed, Jun 13, 2018 at 4:40 PM Yifan Zou wrote: > Thanks everyone for feedbacks! > > We will embed some text to explain the details of the report and guide > people what to do with it at this point. Cham and I prepare to start > updating those

Apache Beam June Newsletter

Hi Beam Community! Here [1] is the June Edition of our Apache Beam Newsletter. This edition was curated by our community of contributors, committers and PMCs. Generally, it contains the work done

Re: SQL Filter Pushdowns in Apache Beam SQL

This has come up in a couple of in-person conversations. Pushing filtering and projection into to connectors is something we intend to do. Calcite's optimizer is designed to support this, we just don't have it set up. Your use case sounds like one that might test the limits of that, since the

Precommits broken?

Recent PRs don't appear to be running all the precommits, and success status isn't being pushed to PRs. Anyone know what is going on? See: https://github.com/apache/beam/pull/5592 https://github.com/apache/beam/pull/5622 Andrew

Re: Proposal: keeping post-commit tests green

On Wed, Jun 13, 2018 at 3:52 PM, Mikhail Gryzykhin wrote: > Hi Ahmet, > > I've checked on tests status and most of other tests are green 98% of the > time. So I feel that we do not need any explicit actions for those tests. > Is it going to be a one time action to fix existing flaky tests? Or

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

+1 to ignoring flaky test. FYI there's a fourth cherrypick: https://github.com/apache/beam/pull/5624 On Wed, Jun 13, 2018 at 3:45 PM Pablo Estrada wrote: > Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky > test. As JB is the release manager, I'l let him make the call on

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Hello all, cherrypicks for the release branch seem to be going well, but thanks to them we were able to surface a flaky test in the release branch. JIRA is filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558 Given that test issue, I see the following options: - Consider that this

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

I would vote for second option, not a release blocker and disable the test in the release branch. My reasoning is: - ReferenceRunner is not yet the official alternative to existing direct runners. - It is bad to have flaky tests on the release branch, and we would not get good signal during

Re: Proposal: keeping post-commit tests green

On Wed, Jun 13, 2018 at 3:45 PM, Mikhail Gryzykhin wrote: > Hello everybody, > > Thanks everyone. I didn't receive any more feedback on the design proposal > document [1] and I believe we've reached consensus. I've added > implementation tasks in JIRA (BEAM-4559 [2]) and will start coding soon.

Re: Beam Dependency Check Report (2018-06-13)

Wow, this is extremely readable and actionable. Are these all direct dependencies, or also transitive? Kenn On Wed, Jun 13, 2018 at 10:45 AM Chamikara Jayalath wrote: > > Thanks Yifan. > > On Wed, Jun 13, 2018 at 10:21 AM Ahmet Altay wrote: > >> Thanks Yifan, this is great! >> >> My

Re: Precommits broken?

Looks like statuses got posted between me writing this email and sending it. Still wondering why the python and go jobs appear to be missing? Andrew On Wed, Jun 13, 2018 at 5:00 PM Andrew Pilloud wrote: > Recent PRs don't appear to be running all the precommits, and success > status isn't

Re: Precommits broken?

Ah, so this is intended and I didn't break anything? Cool! Sorry for the false alarm, looks like a great build optimization! Andrew On Wed, Jun 13, 2018 at 5:06 PM Yifan Zou wrote: > Probably due to the precommit filter applied in #5611 > ? > > On Wed,

Re: Precommits broken?

Precommit filter is a really coool optimization! -Rui On Wed, Jun 13, 2018 at 5:21 PM Andrew Pilloud wrote: > Ah, so this is intended and I didn't break anything? Cool! Sorry for the > false alarm, looks like a great build optimization! > > Andrew > > On Wed, Jun 13, 2018 at 5:06 PM

Re: Precommits broken?

Indeed, I was going to send out an email about pre-commit filtering, but we've already found some kinks and may need to revert it. The change was submitted in PR#5611 [1] and enables Jenkins triggering to only run pre-commits based on modified files. However, Udi noticed that this also prevents

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

It looks good to me, I'm merging and moving forward. Regards JB On 14/06/2018 00:45, Pablo Estrada wrote: > Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky > test. As JB is the release manager, I'l let him make the call on what to > do about it. > Best > -P. > > On Wed,

Re: Proposing interactive beam runner

Thank you Sindy. I like the demo; it looks great. This would be interesting to a lot of users. What are your plans for moving this forward? What kind of an input you are looking for? Ahmet On Wed, Jun 13, 2018 at 2:32 PM, Eugene Kirpichov wrote: > This is awesome, thanks Sindy! I hope that

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

FWIW I have a fix to the flaky test in https://github.com/apache/beam/pull/5585 (open) On Wed, Jun 13, 2018 at 5:26 PM Udi Meiri wrote: > +1 to ignoring flaky test. > > FYI there's a fourth cherrypick: https://github.com/apache/beam/pull/5624 > > On Wed, Jun 13, 2018 at 3:45 PM Pablo Estrada