Re: [ANNOUNCE] New Committer: Byron Ellis

[Tomo Suzuki via dev]

Congratulations!


On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Congrats Byron!
>
> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Byron Ellis (b...@apache.org).
>>
>> Byron has been with Beam for over a year now. You may all know him as the
>> guy who just decided to write a Swift SDK :-). In addition to that big
>> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
>> pipeline authoring, and participated in our collective decision-making
>> process.
>>
>> Considering his contributions to the project over this timeframe, the
>> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]
>>
>> Thank you Byron! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

-- 
Regards,
Tomo

Re: [DISCUSSION][JAVA] Current state of Java 17 support

[Tomo Suzuki via dev]

> Do we still need to support Java 8 SDK?

Yes, for Google Cloud customers who still use Java 8, I want Apache Beam to
support Java 8. Do you observe any special burden maintaining Java 8?

Regards,
Tomo

On Tue, Nov 29, 2022 at 21:48 Austin Bennett  wrote:

> -1 for ongoing Java8 support [ or, said another way, +1 for dropping
> support of Java8 ]
>
> +1 for having tests that run for ANY JDK that we say we support.  Is there
> any reason the resources to support are too costly [ or outweigh the
> benefits of additional confidence in ensuring we support what we say we do
> ]?  I am not certain on whether this would only be critical for releases,
> or should be done as part of regular CI.
>
> On Tue, Nov 29, 2022 at 8:51 AM Alexey Romanenko 
> wrote:
>
>> Hello,
>>
>> I’m sorry if it’s already discussed somewhere but I find myself a little
>> bit lost in the subject.
>> So, I’d like to clarify this - what is a current official state of Java
>> 17 support at Beam?
>>
>> I recall that a great job was done to make Beam compatible with Java 17
>> [1] and Beam already provides “beam_java17_sdk” Docker image [2] but, iiuc,
>> Java 8 is still the default JVM to run all Java tests on Jenkins ("Java
>> PreCommit" in the first order) and there are only limited number of tests
>> that are running with JDK 11 and 17 on Jenkins by dedicated jobs.
>>
>> So, my question would sound like if Beam officially supports Java 17 (and
>> 11), do we need to run all Beam Java SDK related tests (VR and IT test
>> including) against all supported Java SDKs?
>>
>> Do we still need to support Java 8 SDK?
>>
>> In the same time, as we are heading to move everything from Jenkins to
>> GitHub actions, what would be the default JDK there or we will run all
>> Java-related actions against all supported JDKs?
>>
>> —
>> Alexey
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-12240
>> [2] https://hub.docker.com/r/apache/beam_java17_sdk
>>
>>
>>
>> --
Regards,
Tomo

Re: [DISCUSS] Jenkins -> GitHub Actions ?

[Tomo Suzuki via dev]

Kenn, thank you for the summary. SGTM. Looking forward to GitHub Actions.

On Mon, Nov 7, 2022 at 12:58 PM Kenneth Knowles  wrote:

> OK, it seems like there is general consensus. Not too much action on the
> document. I will summarize the gaps that don't have an answer in the doc,
> and my new opinion of how important they are:
>
>  - [required] Run specific non-default workflow on PR
>  - [required] View history of a workflow
>  - [required] Publish nightly snapshots
>  - [required] Run workflow on dedicated worker pool for performance testing
>  - [important but not required] Summarize flakiness statistics of one or
> all workflows
>  - [important but not required] History of all/many workflows in a single
> view
>  - [nice to have] History of specific test case (not just the workflow
> level)
>
> Do any of these seem like I got the importance wrong?
>
> Kenn
>
> On Mon, Nov 7, 2022 at 9:09 AM Austin Bennett  wrote:
>
>> +1
>>
>> Also would help address a good amount of what concerns me that was
>> [sorta] raised by
>> https://lists.apache.org/thread/7jr99nc5xsb3ft1d75kb0ml32bzw89rv
>>
>>
>> Once we think this is something we want to do, but might be
>> blocked/concerned because of lack of definitively comparable features, I'd
>> be happy to take a look at what exists in the wider ecosystem or could be
>> built.
>>
>> Cheers -
>>
>>
>>
>> On Fri, Oct 21, 2022 at 11:10 AM Ismaël Mejía  wrote:
>>
>>> +1 Github Actions are more intuitive and easy to modify and test for
>>> everyone.
>>> Also Beam wins because that makes one less system to maintain.
>>>
>>> Regards,
>>> Ismaël
>>>
>>> On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev
>>>  wrote:
>>> >
>>> > Thanks for kicking this conversation off. I'm +1 on migrating, but
>>> only once we've found a specific replacement for easy observability (which
>>> workflows have been failing lately, and how often) and trigger phrases (for
>>> retries and workflows that aren't automatically kicked off but should be
>>> run for extra validation, e.g. postcommits). Until we have viable
>>> replacements, I don't think we should make the move. Publishing nightly
>>> snapshots is eventually also a must to fully migrate, but probably doesn't
>>> need to block us from making progress here.
>>> >
>>> > With those caveats, the reason that I'm +1 on moving is that our
>>> Jenkins reliability has been rough. Since I joined the project in January,
>>> I can think of 3 different incidents that significantly harmed our ability
>>> to do work.
>>> >
>>> > 1. Jenkins triggers cause multi-day outage - this led to a multi-day
>>> code freeze, and we lost our trigger functionality for days afterwards.
>>> Investigating/restoring our state ate up a pretty full week for me.
>>> > 2. Jenkins plugin cause multi-day outage - this led to multiple days
>>> of Jenkins downtime before eventually being resolved by Infra.
>>> > 3. Cert issues cause many workers to go down - I don't have a thread
>>> for this because I handled most of the investigation the day of, but many
>>> of our workers went down for around a day and nobody noticed until queue
>>> time reached 6+ hours for each workflow.
>>> >
>>> > There may be others that I'm overlooking.
>>> >
>>> > GitHub Actions isn't a magic bullet to fix these problems, but it
>>> minimizes the amount of infra that we're maintaining ourselves, increases
>>> the isolation between workflows (catastrophic failure is less likely), has
>>> uptime guarantees, and is more likely to receive investment going forward
>>> (we're likely to get increasing benefits over time for free). We've also
>>> done a lot of exploration in this area already, so we're not starting from
>>> scratch.
>>> >
>>> > Thanks,
>>> > Danny
>>> >
>>> > On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles 
>>> wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> As you probably noticed, there's a lot of work going on around adding
>>> more GitHub Actions workflows.
>>> >>
>>> >> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues
>>> migration (but less user-facing) it would bring us on to "default"
>>> infrastructure that more people understand and is maintained by GitHub.
>>> >>
>>> >> So far we have hit some serious roadblocks. It isn't just a simple
>>> migration. We have to weigh doing the work to get there.
>>> >>
>>> >> I started a document with a table of the things we get from Jenkins
>>> that we need to be sure to have for GitHub Actions before we could think
>>> about migrating:
>>> >>
>>> >> https://s.apache.org/beam-jenkins-to-gha
>>> >>
>>> >> Can you please help me by adding things that we get from Jenkins, and
>>> if you know how to get them from GitHub Actions add that too.
>>> >>
>>> >> Thanks!
>>> >>
>>> >> Kenn
>>>
>>

-- 
Regards,
Tomo

Re: [ANNOUNCE] New committer: Yi Hu

[Tomo Suzuki via dev]

Congratulations!

On Wed, Nov 9, 2022 at 3:00 PM John Casey via dev 
wrote:

> Congrats! this is well deserved YI
>
> On Wed, Nov 9, 2022 at 2:58 PM Austin Bennett 
> wrote:
>
>> Congrats, and Thanks, Yi!
>>
>> On Wed, Nov 9, 2022 at 11:24 AM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I am with the Beam PMC on this, congratulations and very well deserved,
>>> Yi!
>>>
>>> On Wed, Nov 9, 2022 at 11:08 AM Byron Ellis via dev 
>>> wrote:
>>>
 Congratulations!

 On Wed, Nov 9, 2022 at 11:00 AM Pablo Estrada via dev <
 dev@beam.apache.org> wrote:

> +1 thanks Yi : D
>
> On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Yi! I've really appreciated the ways you've consistently
>> taken responsibility for improving our team's infra and working through
>> sharp edges in the codebase that others have ignored. This is definitely
>> well deserved!
>>
>> Thanks,
>> Danny
>>
>> On Wed, Nov 9, 2022 at 1:37 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Yi!
>>>
>>> On Wed, Nov 9, 2022 at 1:35 PM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Yi!

 On Wed, Nov 9, 2022 at 1:34 PM Ahmed Abualsaud via dev <
 dev@beam.apache.org> wrote:

> Congrats Yi!
>
> On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations Yi!
>>
>> On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Yi Hu (y...@apache.org)
>>>
>>> Yi started contributing to Beam in early 2022. Yi's
>>> contributions are very diverse! I/Os, performance tests, Jenkins, 
>>> support
>>> for Schema logical types. Not only code but a very large amount of 
>>> code
>>> review. Yi is also noted for picking up smaller issues that 
>>> normally would
>>> be left on the backburner and filing issues that he finds rather 
>>> than
>>> ignoring them.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Yi with the responsibilities of a 
>>> Beam
>>> committer. [1]
>>>
>>> Thank you Yi! And we are looking to see more of your
>>> contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

-- 
Regards,
Tomo

Re: [Dataflow][Guidance] Replacing beam-sdks-java-io-google-cloud-platform with local jar

[Tomo Suzuki via dev]

I don't come up with a solution (I'm not familiar with the method
you're using). However I often use "getProtectionDomain()"
https://stackoverflow.com/a/56000383/975074 to find the JAR file from a
class. This ensures the class you modified is actually used.

On Thu, Jul 21, 2022 at 3:35 PM Evan Galpin  wrote:

> Spoke too soon... still can't seem to get the new behaviour to appear in
> dataflow, possibly something is being overridden?
>
> On Thu, Jul 21, 2022 at 3:15 PM Evan Galpin  wrote:
>
>> Making a shadowJar from "beam-sdks-java-io-google-cloud-platform" looks
>> to be working. Added `  id 'com.github.johnrengelman.shadow'` to
>> `build.gradle` for "beam-sdks-java-io-google-cloud-platform" in the beam
>> source and used the resulting jar as a dependency replacement when
>> deploying the job to dataflow.  Looks ok.
>>
>> On Thu, Jul 21, 2022 at 3:02 PM Evan Galpin  wrote:
>>
>>> I believe I have the dependencySubstitution working, but it seems as
>>> though the substitution is removing transitive deps of
>>> "beam-sdks-java-io-google-cloud-platform", hmm...
>>>
>>> On Thu, Jul 21, 2022 at 1:15 PM Evan Galpin  wrote:
>>>
 Hi all,

 I'm trying to test a change I've made locally, but by validating it on
 Dataflow.  It works locally, but I want to validate on Dataflow.  I've
 tried a few different attempts at module substitution in the build.gradle
 config file for the pipeline I'm trying to deploy, but I haven't had any
 success yet.

 How might I be able to replace the
 beam-sdks-java-io-google-cloud-platform module usually installed from maven
 with a local jar generated from running:

 "./gradlew :sdk:java:io:google-cloud-platform:jar"

 Thanks,
 Evan

>>>

-- 
Regards,
Tomo

Re: Java post commit: HL7v2IOReadWriteIT failures

[Tomo Suzuki]

Here is a PR to fix: https://github.com/apache/beam/pull/15498 I confirmed
"Run Java Postcommit" passes.

On Fri, Sep 10, 2021 at 4:32 PM Tomo Suzuki  wrote:

> I'm troubleshooting Java postcommit build failure on HL7v2IOReadWriteIT.
>
> https://issues.apache.org/jira/browse/BEAM-12873
>
>
> org.apache.beam.sdk.io.gcp.healthcare.HL7v2IOReadIT.testHL7v2IO_ListHL7v2Messages_filtered
>
> org.apache.beam.sdk.io.gcp.healthcare.HL7v2IOReadIT.testHL7v2IO_ListHL7v2Messages
> org.apache.beam.sdk.io.gcp.healthcare.HL7v2IOReadWriteIT.testHL7v2IOE2E
>
>
> The build failures happened to include my commit
>
> https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c
> (in which I confirmed the pull request passed "Java Postcommit". Strange)
>
> I'm getting help from the server-side team (CC: Kenn and Ahmet). But if
> somebody knows some clues, please let me know.
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Aliasing Pub/Sub Lite IO in external repo

[Tomo Suzuki]

Hi Daniel,
(You helped me apply some change to this strange setup a few months back.
Thank you for working on rectifying the situation.)

I like that idea overall.

Question 1: How are you going to approach testing/CI?
The pull requests in the java-pubsublite repo do not trigger Beam repo's
CI. You want to deliver things to your customers after they are tested as
much as possible.


Question2 : in the code below, what is the purpose of keeping the
PubsubLiteIO in the Beam repo?

```
class PubsubLiteIO extends com.google.cloud.pubsublite.beam.PubsubLiteIO {}


The backward compatibility came to my mind but I thought you may have more
reasons.


My memo:
java-pubsublite repsitory has:
https://github.com/googleapis/java-pubsublite/blob/master/pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteIO.java
beam repo has:
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java
(and other files in the same directory)
google-cloud-pubsublite is not part of the Libraries BOM (yet) because of
its pre-1.0 status.


On Thu, Jun 17, 2021 at 5:07 PM Daniel Collins  wrote:

> I don't know that the cycle would cause a problem- wouldn't it override
> and cause it to use beam-sdks-java-core:2.30.0 (at least until beam goes to
> 3.X.X)?
>
> Something we can do if this is an issue is mark pubsublite-beam-io's dep
> on beam-sdks-java-core as 'provided'. But I'd prefer to avoid this and just
> let overriding fix it if that works.
>
> On Thu, Jun 17, 2021 at 4:15 PM Andrew Pilloud 
> wrote:
>
>> How do you plan to address the circular dependency? Won't this end up
>> with Beam depending on older versions of itself?
>>
>> beam-sdks-java-io-google-cloud-platform:2.30.0 ->
>> pubsublite-beam-io:0.16.0 -> beam-sdks-java-core:2.29.0
>>
>> On Thu, Jun 17, 2021 at 11:56 AM Daniel Collins 
>> wrote:
>>
>>> Hello beam developers,
>>>
>>> I'm the primary author of the Pub/Sub Lite I/O, and I'd like to get some
>>> feedback on a change to the model for hosting this I/O in beam. Our team
>>> has been frustrated by the fact that we have no way to release features or
>>> fixes for bugs to customers on time scales shorter than the 1-2 months of
>>> the beam release cycle, and that those fixes are necessarily coupled with a
>>> beam version upgrade. To work around this, I forked the I/O in beam to our
>>> own repo about 6 months ago and have been maintaining both copies in
>>> parallel.
>>>
>>> I'd like to retain our ability to quickly fix and improve the I/O while
>>> retaining end-user visibility within the beam repo. To do this, I'd like
>>> to remove all the implementation from the beam repo, and leave the I/O
>>> there implemented as:
>>>
>>> ```
>>> class PubsubLiteIO extends com.google.cloud.pubsublite.beam.PubsubLiteIO
>>> {}
>>> 
>>> , and add a dependency on our beam artifact.
>>>
>>> This enables beam users who want to just use the
>>> beam-sdks-java-io-google-cloud-platform artifact to do so, but they can
>>> also track the canonical version separately in our repo to get fixes and
>>> improvements at a faster rate. All static methods from the parent class
>>> would be available on the class in the beam repo.
>>>
>>> I'd be interested to hear anyones thoughts and suggestions surrounding
>>> this.
>>>
>>> -Daniel
>>>
>>

-- 
Regards,
Tomo

Re: [VOTE] Release 2.30.0, release candidate #1

[Tomo Suzuki]

+1 (non-binding)

Thank you for the preparation. With the GCP dependencies of my interest,
the GitHub checks worked.



On Thu, Jun 3, 2021 at 4:55 AM Heejong Lee  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 2.30.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if no issues are found.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint
> DBC03F1CCF4240FBD0F256F054550BE0F4C0A24D [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.30.0-RC1" [5],
> * website pull request listing the release [6], publishing the API
> reference manual [7], and the blog post [8].
> * Java artifacts were built with Maven 3.6.3 and OpenJDK 1.8.0_292.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
> * Validation sheet with a tab for 2.30.0 release to help with validation
> [9].
> * Docker images published to Docker Hub [10].
> * Python artifacts are published to pypi as a pre-release version [11].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Heejong
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349978
> [2] https://dist.apache.org/repos/dist/dev/beam/2.30.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1174/
> [5] https://github.com/apache/beam/tree/v2.30.0-RC1
> [6] https://github.com/apache/beam/pull/14894
> [7] https://github.com/apache/beam-site/pull/613
> [8] https://github.com/apache/beam/pull/14895
> [9]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=109662250
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image
> [11] https://pypi.org/project/apache-beam/2.30.0rc1/
>


-- 
Regards,
Tomo

Re: LGPL-2.1 in beam-vendor-grpc

[Tomo Suzuki]

As I sent in the separate email about
https://github.com/apache/beam/pull/14833, now the Beam's master branch
depends on the newer version of the vendored gRPC, which does not include
the problematic jboss artifacts.



On Tue, May 11, 2021 at 12:39 PM Kenneth Knowles  wrote:

> +1
>
> It seems we are pretty close on the upgrade. The same tricky problem as
> before, but it seems to be narrowed down.
>
> Kenn
>
> On Mon, May 10, 2021 at 8:26 AM Jean-Baptiste Onofre 
> wrote:
>
>> +1 fully agree.
>>
>> Regards
>> JB
>>
>> Le 10 mai 2021 à 16:02, Jan Lukavský  a écrit :
>>
>> +1 for blocking the release - I think we should not release something
>> about which we _know_ that it might be legally problematic. And we should
>> definitely create a check in the build process that will warn about such
>> issues in the future.
>>
>>  Jan
>> On 5/10/21 3:44 PM, Ismaël Mejía wrote:
>>
>> Tomo just confirmed in the ticket that if we update the gRPC vendored
>> version we won't need the JBoss dependency anymore so we should be good to
>> go with the upgrade. The open question is if this should be blocking for
>> the upcoming Beam 2.31.0 release or we can fix it afterwards.
>>
>>
>> On Mon, May 10, 2021 at 2:46 PM Ismaël Mejía  wrote:
>>
>>> We have been discussing about updating the vendored dependency in
>>> BEAM-11227 <https://issues.apache.org/jira/browse/BEAM-11227>, if I
>>> remember correctly the newer version of gRPC does not require the jboss
>>> dependency, so probably is the best upgrade path, can you confirm Tomo
>>> Suzuki
>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=suztomo> ?
>>>
>>> On Mon, May 10, 2021 at 2:33 PM Jarek Potiuk  wrote:
>>>
>>>> Also we have very similar discussion about it in
>>>> https://issues.apache.org/jira/browse/LEGAL-572
>>>> Just to be clear about the context of it, it's not a legal requirement
>>>> of Apache Licence, it's Apache Software Foundation policy, that we should
>>>> not limit our users in using our software. If the LGPL dependency is
>>>> "optional", it's fine to add such optional dependency. If it is "required"
>>>> to run your software, then it is not allowed as it limits the users of ASF
>>>> software in further redistributing the software in the way they want (this
>>>> is at least my understanding of it).
>>>>
>>>> On Mon, May 10, 2021 at 12:58 PM JB Onofré  wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> You can take a look on
>>>>>
>>>>> https://www.apache.org/legal/resolved.html
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> Le 10 mai 2021 à 12:56, Elliotte Rusty Harold  a
>>>>> écrit :
>>>>>
>>>>> Anyone have a link to the official Apache policy about this? Thanks.
>>>>>
>>>>> On Mon, May 10, 2021 at 10:07 AM Jan Lukavský  wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> we are bundling dependencies with LGPL-2.1, according to license header
>>>>>
>>>>> in META-INF/maven/org.jboss.modules/jboss-modules/pom.xml. I think is
>>>>>
>>>>> might be an issue, already reported here: [1]. I created [2] to track
>>>>> it
>>>>>
>>>>> on our side.
>>>>>
>>>>>
>>>>>  Jan
>>>>>
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-22555
>>>>>
>>>>>
>>>>> [2] https://issues.apache.org/jira/browse/BEAM-12316
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Elliotte Rusty Harold
>>>>> elh...@ibiblio.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> +48 660 796 129 <+48%20660%20796%20129>
>>>>
>>>
>>

-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Update:
I just merged Kiley's https://github.com/apache/beam/pull/14833, in which I
tried several "Run Java Precommit" and didn't observe the logging test
(BeamFnLoggingServiceTest) failures. Let's see how the builds go.


Kenn, Ismaël, and Kiley,
Thank you for the help and follow-up!


On Thu, May 13, 2021 at 10:39 AM Tomo Suzuki  wrote:

> I'm giving up! Can anyone troubleshoot this gRPC concurrency problem
> further?
> My current view of the problem (link
> <https://github.com/apache/beam/pull/14768#issuecomment-840576342>) is
> that "grpc-default-executor" threads stop processing the data. But I cannot
> tell why.
>
> I also raised an question to grpc-java on how best to troubleshoot such
> situation
> https://github.com/grpc/grpc-java/issues/8174
>
> On Wed, May 12, 2021 at 11:29 PM Tomo Suzuki  wrote:
>
>> Update: still the root cause of is unknown.
>>
>> From my observation with debug logging and thread dump,
>> "grpc-default-executor-XXX" threads disappear when the problematic tests
>> become hung.
>> More notes:
>> https://github.com/apache/beam/pull/14768#issuecomment-840228795
>>
>> Interestingly the "grpc-default-executor-XXX" threads reappear in the
>> logs when the pause triggers a 5-second timeout set by JUnit.
>>
>>
>> On Tue, May 11, 2021 at 1:12 PM Tomo Suzuki  wrote:
>>
>>> Thank you for the advice. Yes, the latch not being counted-down is the
>>> problem. (my memo:
>>> https://github.com/apache/beam/pull/14474#discussion_r619557479 ) I'll
>>> need to figure out why withOnError is not called.
>>>
>>>
>>> > Can you repro locally?
>>>
>>> No, the task succeeds in my environment (./gradlew
>>> :runners:google-cloud-dataflow-java:worker:test).
>>>
>>>
>>> On Tue, May 11, 2021 at 12:34 PM Kenneth Knowles 
>>> wrote:
>>>
>>>> I am not sure how much you read the code of the test. So apologies if I
>>>> am saying things you already know. The test does something like:
>>>>
>>>>  - start a logging service
>>>>  - set up some stub clients, each with onError wired up to release a
>>>> countdown latch
>>>>  - send error responses to all three of them (actually it sends the
>>>> error in the same task it creates the stub)
>>>>  - each task waits on the latch
>>>>
>>>> So if onError does not deliver or does not call to release the
>>>> countdown latch, it will hang. I notice in the gist you provide that all
>>>> three stub clients are hung awaiting the latch. That is suspicious to me. I
>>>> would want to confirm if the flakiness always occurs in a way that hangs
>>>> all three. Then there are gRPC workers waiting on empty queues, and the
>>>> main test thread waiting for the hung tasks to complete.
>>>>
>>>> The problem could be something about the test set up. Personally I
>>>> would add a ton of logs, or potentially use a debugger, to confirm exactly
>>>> the state of things when it hangs. Can you repro locally? I think this same
>>>> functionality could be tested in different ways that might remove some of
>>>> the variables. For example starting up all the waiting tasks, then sending
>>>> all the onError messages that should cause them to terminate.
>>>>
>>>> Since this is a unit test, adding a timeout to just that method should
>>>> save time (but will make it harder to capture stack traces, etc). I've
>>>> opened up https://github.com/apache/beam/pull/14781 for that. There
>>>> may be a nice way to add a timeout to the executor to capture the hung
>>>> stack, but I didn't look for it.
>>>>
>>>> Kenn
>>>>
>>>> On Tue, May 11, 2021 at 7:36 AM Tomo Suzuki  wrote:
>>>>
>>>>> gRPC 1.37.0 showed the same problem:
>>>>> BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
>>>>> waits tasks forever, causing timeout in Java precommit.
>>>>>
>>>>> While I continue my investigation, I appreciate if someone knows the
>>>>> cause of the problem, I pasted the thread dump of the Java process when 
>>>>> the
>>>>> test was frozen:
>>>>> https://github.com/apache/beam/pull/14768
>>>>>
>>>>> If this mystery is never solved, vendoring (a bit old) gRPC 1.32.2
>>>>> without the jboss dependencies is an alterna

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.11.0

[Tomo Suzuki]

+1
All classes in the JAR are shaded correctly under org/apache/beam/vendor

On Wed, May 19, 2021 at 2:54 PM Ismaël Mejía  wrote:

> This release is only to publish the vendored dependency artifacts. We need
> those to integrate it and be able to verify if it causes problems or not.
> The PR for this is already opened but it needs the artifacts of this vote
> to be ran.
> https://github.com/apache/beam/pull/14824
>
> For ref there is a document on how to release and validate releases of
> Beam's vendored dependencies that can be handy to anyone wishing to help
> validate:
> https://s.apache.org/beam-release-vendored-artifacts
>
> On Wed, May 19, 2021 at 8:45 PM Tyson Hamilton  wrote:
>
>> I'd like to help, but I don't know how to determine whether this upgrade
>> is going to cause problems or not. Are there tests I should look at, or
>> some validation I should perform?
>>
>> On Wed, May 19, 2021 at 11:29 AM Ismaël Mejía  wrote:
>>
>>> Kind reminder, the vote is ongoing
>>>
>>> On Mon, May 17, 2021 at 5:32 PM Ismaël Mejía  wrote:
>>>
 Please review the release of the following artifacts that we vendor:
  * beam-vendor-bytebuddy-1_11_0

 Hi everyone,
 Please review and vote on the release candidate #1 for the version 0.1,
 as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)

 The complete staging area is available for your review, which includes:
 * the official Apache source release to be deployed to dist.apache.org
 [1], which is signed with the key with fingerprint
 3415631729E15B33051ADB670A9DAF6713B86349 [2],
 * all artifacts to be deployed to the Maven Central Repository [3],
 * commit hash "d93c591deb21237ddb656583d7ef7a4debba" [4],

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 Thanks,
 Release Manager

 [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
 [2] https://dist.apache.org/repos/dist/release/beam/KEYS
 [3]
 https://repository.apache.org/content/repositories/orgapachebeam-1166/
 [4]
 https://github.com/apache/beam/commit/d93c591deb21237ddb656583d7ef7a4debba



-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

I'm giving up! Can anyone troubleshoot this gRPC concurrency problem
further?
My current view of the problem (link
<https://github.com/apache/beam/pull/14768#issuecomment-840576342>) is that
"grpc-default-executor" threads stop processing the data. But I cannot tell
why.

I also raised an question to grpc-java on how best to troubleshoot such
situation
https://github.com/grpc/grpc-java/issues/8174

On Wed, May 12, 2021 at 11:29 PM Tomo Suzuki  wrote:

> Update: still the root cause of is unknown.
>
> From my observation with debug logging and thread dump,
> "grpc-default-executor-XXX" threads disappear when the problematic tests
> become hung.
> More notes:
> https://github.com/apache/beam/pull/14768#issuecomment-840228795
>
> Interestingly the "grpc-default-executor-XXX" threads reappear in the logs
> when the pause triggers a 5-second timeout set by JUnit.
>
>
> On Tue, May 11, 2021 at 1:12 PM Tomo Suzuki  wrote:
>
>> Thank you for the advice. Yes, the latch not being counted-down is the
>> problem. (my memo:
>> https://github.com/apache/beam/pull/14474#discussion_r619557479 ) I'll
>> need to figure out why withOnError is not called.
>>
>>
>> > Can you repro locally?
>>
>> No, the task succeeds in my environment (./gradlew
>> :runners:google-cloud-dataflow-java:worker:test).
>>
>>
>> On Tue, May 11, 2021 at 12:34 PM Kenneth Knowles  wrote:
>>
>>> I am not sure how much you read the code of the test. So apologies if I
>>> am saying things you already know. The test does something like:
>>>
>>>  - start a logging service
>>>  - set up some stub clients, each with onError wired up to release a
>>> countdown latch
>>>  - send error responses to all three of them (actually it sends the
>>> error in the same task it creates the stub)
>>>  - each task waits on the latch
>>>
>>> So if onError does not deliver or does not call to release the countdown
>>> latch, it will hang. I notice in the gist you provide that all three stub
>>> clients are hung awaiting the latch. That is suspicious to me. I would want
>>> to confirm if the flakiness always occurs in a way that hangs all three.
>>> Then there are gRPC workers waiting on empty queues, and the main test
>>> thread waiting for the hung tasks to complete.
>>>
>>> The problem could be something about the test set up. Personally I would
>>> add a ton of logs, or potentially use a debugger, to confirm exactly the
>>> state of things when it hangs. Can you repro locally? I think this same
>>> functionality could be tested in different ways that might remove some of
>>> the variables. For example starting up all the waiting tasks, then sending
>>> all the onError messages that should cause them to terminate.
>>>
>>> Since this is a unit test, adding a timeout to just that method should
>>> save time (but will make it harder to capture stack traces, etc). I've
>>> opened up https://github.com/apache/beam/pull/14781 for that. There may
>>> be a nice way to add a timeout to the executor to capture the hung stack,
>>> but I didn't look for it.
>>>
>>> Kenn
>>>
>>> On Tue, May 11, 2021 at 7:36 AM Tomo Suzuki  wrote:
>>>
>>>> gRPC 1.37.0 showed the same problem:
>>>> BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
>>>> waits tasks forever, causing timeout in Java precommit.
>>>>
>>>> While I continue my investigation, I appreciate if someone knows the
>>>> cause of the problem, I pasted the thread dump of the Java process when the
>>>> test was frozen:
>>>> https://github.com/apache/beam/pull/14768
>>>>
>>>> If this mystery is never solved, vendoring (a bit old) gRPC 1.32.2
>>>> without the jboss dependencies is an alternate option, (suggestion by Kenn;
>>>> memo
>>>> <https://issues.apache.org/jira/browse/BEAM-11227?focusedCommentId=17318238=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17318238>
>>>> )
>>>>
>>>> Regards,
>>>> Tomo
>>>>
>>>>
>>>> On Mon, May 10, 2021 at 9:40 AM Tomo Suzuki  wrote:
>>>>
>>>>> I was investigating the strange timeout (
>>>>> https://github.com/apache/beam/pull/14474) but was occupied with
>>>>> something else lately.
>>>>> Let me try the new version today to see any improvements.
>>>>>
>

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Update: still the root cause of is unknown.

>From my observation with debug logging and thread dump,
"grpc-default-executor-XXX" threads disappear when the problematic tests
become hung.
More notes: https://github.com/apache/beam/pull/14768#issuecomment-840228795

Interestingly the "grpc-default-executor-XXX" threads reappear in the logs
when the pause triggers a 5-second timeout set by JUnit.


On Tue, May 11, 2021 at 1:12 PM Tomo Suzuki  wrote:

> Thank you for the advice. Yes, the latch not being counted-down is the
> problem. (my memo:
> https://github.com/apache/beam/pull/14474#discussion_r619557479 ) I'll
> need to figure out why withOnError is not called.
>
>
> > Can you repro locally?
>
> No, the task succeeds in my environment (./gradlew
> :runners:google-cloud-dataflow-java:worker:test).
>
>
> On Tue, May 11, 2021 at 12:34 PM Kenneth Knowles  wrote:
>
>> I am not sure how much you read the code of the test. So apologies if I
>> am saying things you already know. The test does something like:
>>
>>  - start a logging service
>>  - set up some stub clients, each with onError wired up to release a
>> countdown latch
>>  - send error responses to all three of them (actually it sends the error
>> in the same task it creates the stub)
>>  - each task waits on the latch
>>
>> So if onError does not deliver or does not call to release the countdown
>> latch, it will hang. I notice in the gist you provide that all three stub
>> clients are hung awaiting the latch. That is suspicious to me. I would want
>> to confirm if the flakiness always occurs in a way that hangs all three.
>> Then there are gRPC workers waiting on empty queues, and the main test
>> thread waiting for the hung tasks to complete.
>>
>> The problem could be something about the test set up. Personally I would
>> add a ton of logs, or potentially use a debugger, to confirm exactly the
>> state of things when it hangs. Can you repro locally? I think this same
>> functionality could be tested in different ways that might remove some of
>> the variables. For example starting up all the waiting tasks, then sending
>> all the onError messages that should cause them to terminate.
>>
>> Since this is a unit test, adding a timeout to just that method should
>> save time (but will make it harder to capture stack traces, etc). I've
>> opened up https://github.com/apache/beam/pull/14781 for that. There may
>> be a nice way to add a timeout to the executor to capture the hung stack,
>> but I didn't look for it.
>>
>> Kenn
>>
>> On Tue, May 11, 2021 at 7:36 AM Tomo Suzuki  wrote:
>>
>>> gRPC 1.37.0 showed the same problem:
>>> BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
>>> waits tasks forever, causing timeout in Java precommit.
>>>
>>> While I continue my investigation, I appreciate if someone knows the
>>> cause of the problem, I pasted the thread dump of the Java process when the
>>> test was frozen:
>>> https://github.com/apache/beam/pull/14768
>>>
>>> If this mystery is never solved, vendoring (a bit old) gRPC 1.32.2
>>> without the jboss dependencies is an alternate option, (suggestion by Kenn;
>>> memo
>>> <https://issues.apache.org/jira/browse/BEAM-11227?focusedCommentId=17318238=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17318238>
>>> )
>>>
>>> Regards,
>>> Tomo
>>>
>>>
>>> On Mon, May 10, 2021 at 9:40 AM Tomo Suzuki  wrote:
>>>
>>>> I was investigating the strange timeout (
>>>> https://github.com/apache/beam/pull/14474) but was occupied with
>>>> something else lately.
>>>> Let me try the new version today to see any improvements.
>>>>
>>>>
>>>> On Mon, May 10, 2021 at 4:57 AM Ismaël Mejía  wrote:
>>>>
>>>>> I just saw that gRPC 1.37.1 is out now (and with aarch64 support for
>>>>> python!) that made me wonder about this, what is the current status of
>>>>> upgrading the vendored dependency Tomo?
>>>>>
>>>>>
>>>>> On Thu, Apr 8, 2021 at 4:16 PM Tomo Suzuki  wrote:
>>>>>
>>>>>> We observed the cron job of Java Precommit for the master branch
>>>>>> started timing out often (not always) since upgrading the gRPC version.
>>>>>> https://github.com/apache/beam/pull/14466#issuecomment-815343974
>>>>>>
>>>>>> Exchanged messages

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Thank you for the advice. Yes, the latch not being counted-down is the
problem. (my memo:
https://github.com/apache/beam/pull/14474#discussion_r619557479 ) I'll need
to figure out why withOnError is not called.


> Can you repro locally?

No, the task succeeds in my environment (./gradlew
:runners:google-cloud-dataflow-java:worker:test).


On Tue, May 11, 2021 at 12:34 PM Kenneth Knowles  wrote:

> I am not sure how much you read the code of the test. So apologies if I am
> saying things you already know. The test does something like:
>
>  - start a logging service
>  - set up some stub clients, each with onError wired up to release a
> countdown latch
>  - send error responses to all three of them (actually it sends the error
> in the same task it creates the stub)
>  - each task waits on the latch
>
> So if onError does not deliver or does not call to release the countdown
> latch, it will hang. I notice in the gist you provide that all three stub
> clients are hung awaiting the latch. That is suspicious to me. I would want
> to confirm if the flakiness always occurs in a way that hangs all three.
> Then there are gRPC workers waiting on empty queues, and the main test
> thread waiting for the hung tasks to complete.
>
> The problem could be something about the test set up. Personally I would
> add a ton of logs, or potentially use a debugger, to confirm exactly the
> state of things when it hangs. Can you repro locally? I think this same
> functionality could be tested in different ways that might remove some of
> the variables. For example starting up all the waiting tasks, then sending
> all the onError messages that should cause them to terminate.
>
> Since this is a unit test, adding a timeout to just that method should
> save time (but will make it harder to capture stack traces, etc). I've
> opened up https://github.com/apache/beam/pull/14781 for that. There may
> be a nice way to add a timeout to the executor to capture the hung stack,
> but I didn't look for it.
>
> Kenn
>
> On Tue, May 11, 2021 at 7:36 AM Tomo Suzuki  wrote:
>
>> gRPC 1.37.0 showed the same problem:
>> BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
>> waits tasks forever, causing timeout in Java precommit.
>>
>> While I continue my investigation, I appreciate if someone knows the
>> cause of the problem, I pasted the thread dump of the Java process when the
>> test was frozen:
>> https://github.com/apache/beam/pull/14768
>>
>> If this mystery is never solved, vendoring (a bit old) gRPC 1.32.2
>> without the jboss dependencies is an alternate option, (suggestion by Kenn;
>> memo
>> <https://issues.apache.org/jira/browse/BEAM-11227?focusedCommentId=17318238=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17318238>
>> )
>>
>> Regards,
>> Tomo
>>
>>
>> On Mon, May 10, 2021 at 9:40 AM Tomo Suzuki  wrote:
>>
>>> I was investigating the strange timeout (
>>> https://github.com/apache/beam/pull/14474) but was occupied with
>>> something else lately.
>>> Let me try the new version today to see any improvements.
>>>
>>>
>>> On Mon, May 10, 2021 at 4:57 AM Ismaël Mejía  wrote:
>>>
>>>> I just saw that gRPC 1.37.1 is out now (and with aarch64 support for
>>>> python!) that made me wonder about this, what is the current status of
>>>> upgrading the vendored dependency Tomo?
>>>>
>>>>
>>>> On Thu, Apr 8, 2021 at 4:16 PM Tomo Suzuki  wrote:
>>>>
>>>>> We observed the cron job of Java Precommit for the master branch
>>>>> started timing out often (not always) since upgrading the gRPC version.
>>>>> https://github.com/apache/beam/pull/14466#issuecomment-815343974
>>>>>
>>>>> Exchanged messages with Kenn, I reverted to the change; now the master
>>>>> branch uses the vendored gRPC 1.26.
>>>>>
>>>>>
>>>>> On Wed, Mar 31, 2021 at 11:40 AM Kenneth Knowles 
>>>>> wrote:
>>>>>
>>>>>> Merged. Let's keep an eye for trouble, and I will incorporate to the
>>>>>> release branch.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Wed, Mar 31, 2021 at 6:45 AM Tomo Suzuki 
>>>>>> wrote:
>>>>>>
>>>>>>> Regarding troubleshooting on build timeout, it seems that Docker
>>>>>>> cache in Jenkins machines might be playing a role. As I run more "Java
>>>>>>> Pre

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

I was investigating the strange timeout (
https://github.com/apache/beam/pull/14474) but was occupied with something
else lately.
Let me try the new version today to see any improvements.


On Mon, May 10, 2021 at 4:57 AM Ismaël Mejía  wrote:

> I just saw that gRPC 1.37.1 is out now (and with aarch64 support for
> python!) that made me wonder about this, what is the current status of
> upgrading the vendored dependency Tomo?
>
>
> On Thu, Apr 8, 2021 at 4:16 PM Tomo Suzuki  wrote:
>
>> We observed the cron job of Java Precommit for the master branch started
>> timing out often (not always) since upgrading the gRPC version.
>> https://github.com/apache/beam/pull/14466#issuecomment-815343974
>>
>> Exchanged messages with Kenn, I reverted to the change; now the master
>> branch uses the vendored gRPC 1.26.
>>
>>
>> On Wed, Mar 31, 2021 at 11:40 AM Kenneth Knowles  wrote:
>>
>>> Merged. Let's keep an eye for trouble, and I will incorporate to the
>>> release branch.
>>>
>>> Kenn
>>>
>>> On Wed, Mar 31, 2021 at 6:45 AM Tomo Suzuki  wrote:
>>>
>>>> Regarding troubleshooting on build timeout, it seems that Docker cache
>>>> in Jenkins machines might be playing a role. As I run more "Java
>>>> Presubmit", I no longer observe timeouts in the PR.
>>>>
>>>> Kenn, would you merge the PR?
>>>> https://github.com/apache/beam/pull/14295 (all checks green, including
>>>> the new Java postcommit checks)
>>>>
>>>> On Thu, Mar 25, 2021 at 5:24 PM Kenneth Knowles 
>>>> wrote:
>>>>
>>>>> Yes, I agree this might be a good idea. This is not the only major
>>>>> issue on the release-2.29.0 branch.
>>>>>
>>>>> The counter argument is that we will be pulling in all the bugs
>>>>> introduced to `master` since the branch cut.
>>>>>
>>>>> As far as effort goes, I have been mostly focused on burning down the
>>>>> bugs so I would not lose much work in the release process.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Thu, Mar 25, 2021 at 1:42 PM Ismaël Mejía 
>>>>> wrote:
>>>>>
>>>>>> Precommit is quite unstable in the last days, so worth to check if
>>>>>> something is wrong in the CI.
>>>>>>
>>>>>> I have a question Kenn. Given that cherry picking this might be a bit
>>>>>> big as a change can we just reconsider cutting the 2.29.0 branch again
>>>>>> after the updated gRPC version use gets merged and mark the issues
>>>>>> already fixed for version 2.30.0 to version 2.29.0 ? Seems like an
>>>>>> easier upgrade path (and we will get some nice fixes/improvements like
>>>>>> official Spark 3 support for free on the release).
>>>>>>
>>>>>> WDYT?
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 24, 2021 at 8:06 PM Tomo Suzuki 
>>>>>> wrote:
>>>>>> >
>>>>>> > Update: I observe that Java precommit check is unstable in the PR
>>>>>> to upgrade vendored gRPC (compared with an PR with an empty change).
>>>>>> There's no constant failures; sometimes it succeeds and other times it
>>>>>> faces timeout and flaky test failures.
>>>>>> >
>>>>>> > https://github.com/apache/beam/pull/14295#issuecomment-806071087
>>>>>> >
>>>>>> >
>>>>>> > On Mon, Mar 22, 2021 at 10:46 AM Tomo Suzuki 
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Thank you for the voting and I see the artifact available in Maven
>>>>>> Central. I'll work on the PR to use the published artifact today.
>>>>>> >>
>>>>>> https://search.maven.org/artifact/org.apache.beam/beam-vendor-grpc-1_36_0/0.1/jar
>>>>>> >>
>>>>>> >> On Tue, Mar 16, 2021 at 3:07 PM Kenneth Knowles 
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Update on this: there are some minor issues and then I'll send
>>>>>> out the RC.
>>>>>> >>>
>>>>>> >>> I think this is worth blocking 2.29.0 release on, so I will do
>>>>>> this first. We are still eliminating other blockers from 2.29.0 anyhow.
>>>>>> >>>
>>>>>> >>> Kenn
>>>>>> >>>
>>>>>> >>> On Mon, Mar 15, 2021 at 7:17 AM Tomo Suzuki 
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi Beam developers,
>>>>>> >>>>
>>>>>> >>>> I'm working on upgrading the vendored gRPC 1.36.0
>>>>>> >>>> https://issues.apache.org/jira/browse/BEAM-11227 (PR:
>>>>>> https://github.com/apache/beam/pull/14028)
>>>>>> >>>> Let me know if you have any questions or concerns.
>>>>>> >>>>
>>>>>> >>>> Background:
>>>>>> >>>> Exchanged messages with Ismaël in BEAM-11227, it seems that it
>>>>>> the ticket created by some automation is false positive, but it's nice to
>>>>>> use an artifact without being marked with CVE.
>>>>>> >>>>
>>>>>> >>>> Kenn offered to work as the release manager (as in
>>>>>> https://s.apache.org/beam-release-vendored-artifacts) of the
>>>>>> vendored artifact.
>>>>>> >>>>
>>>>>> >>>> --
>>>>>> >>>> Regards,
>>>>>> >>>> Tomo
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Regards,
>>>>>> >> Tomo
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Regards,
>>>>>> > Tomo
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

[Tomo Suzuki]

Thank you for the preparation!

> a few responses that some high priority changes

Would you be willing to share the items for visibility?


On Wed, Apr 21, 2021 at 7:21 PM Kenneth Knowles  wrote:
>
> Also the 2.29.0 was re-cut.
>
> Usually a delay in one release should not delay the next release, because 
> each release represents a certain quantity of changes. But in this case, the 
> actual quantity of changes is affected by the re-cut, too.
>
> On Wed, Apr 21, 2021 at 4:12 PM Heejong Lee  wrote:
>>
>> Update on the 2.30.0 branch cut schedule:
>>
>> I'm thinking of delaying the branch cut a week since I've got a few 
>> responses that some high priority changes are still ongoing.
>>
>> The new cut date is April 28.
>>
>>
>> On Tue, Apr 20, 2021 at 6:07 PM Ahmet Altay  wrote:
>>>
>>> +1 and thank you!
>>>
>>> On Tue, Apr 20, 2021 at 4:55 PM Heejong Lee  wrote:

 Hi All,

 Beam 2.30.0 release is scheduled to be cut on April 21 according to the 
 release calendar [1]

 I'd like to volunteer myself to be the release manager for this release. I 
 plan on cutting the release branch on the scheduled date.

 Any comments or objections ?

 Thanks,
 Heejong

 [1] 
 https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com=America/Los_Angeles



--
Regards,
Tomo

Re: Issues and PR names and descriptions (or should we change the contribution guide)

[Tomo Suzuki]

BEAM-12173 is on me. I'm sorry about that. Re-reading committer guide
[1], I see I was not following this

> The reviewer should give the LGTM and then request that the author of the 
> pull request rebase, squash, split, etc, the commits, so that the history is 
> most useful


Thank you for the feedback on this matter! (And I don't think we
should change the contribution guide)

[1] https://beam.apache.org/contribute/committer-guide/

On Wed, Apr 21, 2021 at 10:35 AM Ismaël Mejía  wrote:
>
> Hello,
>
> I have noticed an ongoing pattern of carelessness around issues/PR titles and
> descriptions. It is really painful to see more and more examples like:
>
> BEAM-12160 Add TODO for fixing the warning
> BEAM-12165 Fix ParquetIO
> BEAM-12173 avoid intermediate conversion (PR) and BEAM-12173 use
> toMinutes (commit)
>
> In all these cases with just a bit of detail in the title it would be enough 
> to
> make other contributors or reviewers life easierm as well as to have a better
> project history.  What astonishes me apart of the lack of care is that some of
> those are from Beam commmitters.
>
> We already have discussed about not paying attention during commit merges 
> where
> some PRs end up merging tons of 'unwanted' fixup commits, and nothing has
> changed so I am wondering if we should maybe just totally remove that rule 
> (for
> commits) and also eventually for titles and descriptions.
>
> Ismaël
>
> [1] https://beam.apache.org/contribute/



-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

We observed the cron job of Java Precommit for the master branch started
timing out often (not always) since upgrading the gRPC version.
https://github.com/apache/beam/pull/14466#issuecomment-815343974

Exchanged messages with Kenn, I reverted to the change; now the master
branch uses the vendored gRPC 1.26.


On Wed, Mar 31, 2021 at 11:40 AM Kenneth Knowles  wrote:

> Merged. Let's keep an eye for trouble, and I will incorporate to the
> release branch.
>
> Kenn
>
> On Wed, Mar 31, 2021 at 6:45 AM Tomo Suzuki  wrote:
>
>> Regarding troubleshooting on build timeout, it seems that Docker cache in
>> Jenkins machines might be playing a role. As I run more "Java Presubmit", I
>> no longer observe timeouts in the PR.
>>
>> Kenn, would you merge the PR?
>> https://github.com/apache/beam/pull/14295 (all checks green, including
>> the new Java postcommit checks)
>>
>> On Thu, Mar 25, 2021 at 5:24 PM Kenneth Knowles  wrote:
>>
>>> Yes, I agree this might be a good idea. This is not the only major issue
>>> on the release-2.29.0 branch.
>>>
>>> The counter argument is that we will be pulling in all the bugs
>>> introduced to `master` since the branch cut.
>>>
>>> As far as effort goes, I have been mostly focused on burning down the
>>> bugs so I would not lose much work in the release process.
>>>
>>> Kenn
>>>
>>> On Thu, Mar 25, 2021 at 1:42 PM Ismaël Mejía  wrote:
>>>
>>>> Precommit is quite unstable in the last days, so worth to check if
>>>> something is wrong in the CI.
>>>>
>>>> I have a question Kenn. Given that cherry picking this might be a bit
>>>> big as a change can we just reconsider cutting the 2.29.0 branch again
>>>> after the updated gRPC version use gets merged and mark the issues
>>>> already fixed for version 2.30.0 to version 2.29.0 ? Seems like an
>>>> easier upgrade path (and we will get some nice fixes/improvements like
>>>> official Spark 3 support for free on the release).
>>>>
>>>> WDYT?
>>>>
>>>>
>>>> On Wed, Mar 24, 2021 at 8:06 PM Tomo Suzuki  wrote:
>>>> >
>>>> > Update: I observe that Java precommit check is unstable in the PR to
>>>> upgrade vendored gRPC (compared with an PR with an empty change). There's
>>>> no constant failures; sometimes it succeeds and other times it faces
>>>> timeout and flaky test failures.
>>>> >
>>>> > https://github.com/apache/beam/pull/14295#issuecomment-806071087
>>>> >
>>>> >
>>>> > On Mon, Mar 22, 2021 at 10:46 AM Tomo Suzuki 
>>>> wrote:
>>>> >>
>>>> >> Thank you for the voting and I see the artifact available in Maven
>>>> Central. I'll work on the PR to use the published artifact today.
>>>> >>
>>>> https://search.maven.org/artifact/org.apache.beam/beam-vendor-grpc-1_36_0/0.1/jar
>>>> >>
>>>> >> On Tue, Mar 16, 2021 at 3:07 PM Kenneth Knowles 
>>>> wrote:
>>>> >>>
>>>> >>> Update on this: there are some minor issues and then I'll send out
>>>> the RC.
>>>> >>>
>>>> >>> I think this is worth blocking 2.29.0 release on, so I will do this
>>>> first. We are still eliminating other blockers from 2.29.0 anyhow.
>>>> >>>
>>>> >>> Kenn
>>>> >>>
>>>> >>> On Mon, Mar 15, 2021 at 7:17 AM Tomo Suzuki 
>>>> wrote:
>>>> >>>>
>>>> >>>> Hi Beam developers,
>>>> >>>>
>>>> >>>> I'm working on upgrading the vendored gRPC 1.36.0
>>>> >>>> https://issues.apache.org/jira/browse/BEAM-11227 (PR:
>>>> https://github.com/apache/beam/pull/14028)
>>>> >>>> Let me know if you have any questions or concerns.
>>>> >>>>
>>>> >>>> Background:
>>>> >>>> Exchanged messages with Ismaël in BEAM-11227, it seems that it the
>>>> ticket created by some automation is false positive, but it's nice to use
>>>> an artifact without being marked with CVE.
>>>> >>>>
>>>> >>>> Kenn offered to work as the release manager (as in
>>>> https://s.apache.org/beam-release-vendored-artifacts) of the vendored
>>>> artifact.
>>>> >>>>
>>>> >>>> --
>>>> >>>> Regards,
>>>> >>>> Tomo
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Regards,
>>>> >> Tomo
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Regards,
>>>> > Tomo
>>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: [ANNOUNCE] New committer: Tomo Suzuki

[Tomo Suzuki]

Thank you all! I'm looking forward to working with you further.

On Fri, Apr 2, 2021 at 5:29 PM Robin Qiu  wrote:

> Congrats Tomo, well deserved!
>
> On Fri, Apr 2, 2021 at 2:28 PM Ismaël Mejía  wrote:
>
>> Congrats Tomo, so well deserved. It has been a pleasure to work with you!
>>
>>
>> On Fri, Apr 2, 2021 at 8:29 PM Tyson Hamilton  wrote:
>>
>>> Congrats!
>>>
>>> On Fri, Apr 2, 2021 at 11:02 AM Pablo Estrada 
>>> wrote:
>>>
>>>> Thank you Tomo! And congrats : )
>>>>
>>>> On Fri, Apr 2, 2021 at 10:24 AM Robert Bradshaw 
>>>> wrote:
>>>>
>>>>> Congratulations!
>>>>>
>>>>> On Fri, Apr 2, 2021 at 10:19 AM Chamikara Jayalath <
>>>>> chamik...@google.com> wrote:
>>>>>
>>>>>> Congrats Tomo!
>>>>>>
>>>>>> On Fri, Apr 2, 2021 at 9:54 AM Brian Hulette 
>>>>>> wrote:
>>>>>>
>>>>>>> Congratulations Tomo! Well deserved :)
>>>>>>>
>>>>>>> On Fri, Apr 2, 2021 at 9:51 AM Yichi Zhang 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Congratulations!
>>>>>>>>
>>>>>>>> On Fri, Apr 2, 2021 at 9:42 AM Ahmet Altay 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Congratulations! 
>>>>>>>>>
>>>>>>>>> On Fri, Apr 2, 2021 at 9:38 AM Kenneth Knowles 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Please join me and the rest of the Beam PMC in welcoming a new
>>>>>>>>>> committer: Tomo Suzuki
>>>>>>>>>>
>>>>>>>>>> Since joining the Beam community in 2019, Tomo has done lots of
>>>>>>>>>> critical work on Beam's dependencies: maintaining the dependency 
>>>>>>>>>> checker
>>>>>>>>>> that files Jiras and sends emails, upgrading dependencies, fixing
>>>>>>>>>> dependency configuration errors, maintaining our linkage checker. 
>>>>>>>>>> Most
>>>>>>>>>> recently, an epic upgrade of gRPC.
>>>>>>>>>>
>>>>>>>>>> Considering these highlighted contributions and the rest, the
>>>>>>>>>> Beam PMC trusts Tomo with the responsibilities of a Beam
>>>>>>>>>> committer [1].
>>>>>>>>>>
>>>>>>>>>> Thank you, Tomo, for your contributions.
>>>>>>>>>>
>>>>>>>>>> Kenn
>>>>>>>>>>
>>>>>>>>>> [1] https://beam.apache.org/contribute/become-a-committer
>>>>>>>>>> /#an-apache-beam-committer
>>>>>>>>>>
>>>>>>>>>

-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Regarding troubleshooting on build timeout, it seems that Docker cache in
Jenkins machines might be playing a role. As I run more "Java Presubmit", I
no longer observe timeouts in the PR.

Kenn, would you merge the PR?
https://github.com/apache/beam/pull/14295 (all checks green, including the
new Java postcommit checks)

On Thu, Mar 25, 2021 at 5:24 PM Kenneth Knowles  wrote:

> Yes, I agree this might be a good idea. This is not the only major issue
> on the release-2.29.0 branch.
>
> The counter argument is that we will be pulling in all the bugs introduced
> to `master` since the branch cut.
>
> As far as effort goes, I have been mostly focused on burning down the bugs
> so I would not lose much work in the release process.
>
> Kenn
>
> On Thu, Mar 25, 2021 at 1:42 PM Ismaël Mejía  wrote:
>
>> Precommit is quite unstable in the last days, so worth to check if
>> something is wrong in the CI.
>>
>> I have a question Kenn. Given that cherry picking this might be a bit
>> big as a change can we just reconsider cutting the 2.29.0 branch again
>> after the updated gRPC version use gets merged and mark the issues
>> already fixed for version 2.30.0 to version 2.29.0 ? Seems like an
>> easier upgrade path (and we will get some nice fixes/improvements like
>> official Spark 3 support for free on the release).
>>
>> WDYT?
>>
>>
>> On Wed, Mar 24, 2021 at 8:06 PM Tomo Suzuki  wrote:
>> >
>> > Update: I observe that Java precommit check is unstable in the PR to
>> upgrade vendored gRPC (compared with an PR with an empty change). There's
>> no constant failures; sometimes it succeeds and other times it faces
>> timeout and flaky test failures.
>> >
>> > https://github.com/apache/beam/pull/14295#issuecomment-806071087
>> >
>> >
>> > On Mon, Mar 22, 2021 at 10:46 AM Tomo Suzuki 
>> wrote:
>> >>
>> >> Thank you for the voting and I see the artifact available in Maven
>> Central. I'll work on the PR to use the published artifact today.
>> >>
>> https://search.maven.org/artifact/org.apache.beam/beam-vendor-grpc-1_36_0/0.1/jar
>> >>
>> >> On Tue, Mar 16, 2021 at 3:07 PM Kenneth Knowles 
>> wrote:
>> >>>
>> >>> Update on this: there are some minor issues and then I'll send out
>> the RC.
>> >>>
>> >>> I think this is worth blocking 2.29.0 release on, so I will do this
>> first. We are still eliminating other blockers from 2.29.0 anyhow.
>> >>>
>> >>> Kenn
>> >>>
>> >>> On Mon, Mar 15, 2021 at 7:17 AM Tomo Suzuki 
>> wrote:
>> >>>>
>> >>>> Hi Beam developers,
>> >>>>
>> >>>> I'm working on upgrading the vendored gRPC 1.36.0
>> >>>> https://issues.apache.org/jira/browse/BEAM-11227 (PR:
>> https://github.com/apache/beam/pull/14028)
>> >>>> Let me know if you have any questions or concerns.
>> >>>>
>> >>>> Background:
>> >>>> Exchanged messages with Ismaël in BEAM-11227, it seems that it the
>> ticket created by some automation is false positive, but it's nice to use
>> an artifact without being marked with CVE.
>> >>>>
>> >>>> Kenn offered to work as the release manager (as in
>> https://s.apache.org/beam-release-vendored-artifacts) of the vendored
>> artifact.
>> >>>>
>> >>>> --
>> >>>> Regards,
>> >>>> Tomo
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Tomo
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Tomo
>>
>

-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Update: I observe that Java precommit check is unstable in the PR to
upgrade vendored gRPC (compared with an PR with an empty change). There's
no constant failures; sometimes it succeeds and other times it faces
timeout and flaky test failures.

https://github.com/apache/beam/pull/14295#issuecomment-806071087


On Mon, Mar 22, 2021 at 10:46 AM Tomo Suzuki  wrote:

> Thank you for the voting and I see the artifact available in Maven
> Central. I'll work on the PR to use the published artifact today.
>
> https://search.maven.org/artifact/org.apache.beam/beam-vendor-grpc-1_36_0/0.1/jar
>
> On Tue, Mar 16, 2021 at 3:07 PM Kenneth Knowles  wrote:
>
>> Update on this: there are some minor issues and then I'll send out the RC.
>>
>> I think this is worth blocking 2.29.0 release on, so I will do this
>> first. We are still eliminating other blockers from 2.29.0 anyhow.
>>
>> Kenn
>>
>> On Mon, Mar 15, 2021 at 7:17 AM Tomo Suzuki  wrote:
>>
>>> Hi Beam developers,
>>>
>>> I'm working on upgrading the vendored gRPC 1.36.0
>>> https://issues.apache.org/jira/browse/BEAM-11227 (PR:
>>> https://github.com/apache/beam/pull/14028)
>>> Let me know if you have any questions or concerns.
>>>
>>> Background:
>>> Exchanged messages with Ismaël in BEAM-11227, it seems that it the
>>> ticket created by some automation is false positive, but it's nice to use
>>> an artifact without being marked with CVE.
>>>
>>> Kenn offered to work as the release manager (as in
>>> https://s.apache.org/beam-release-vendored-artifacts) of the vendored
>>> artifact.
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Thank you for the voting and I see the artifact available in Maven Central.
I'll work on the PR to use the published artifact today.
https://search.maven.org/artifact/org.apache.beam/beam-vendor-grpc-1_36_0/0.1/jar

On Tue, Mar 16, 2021 at 3:07 PM Kenneth Knowles  wrote:

> Update on this: there are some minor issues and then I'll send out the RC.
>
> I think this is worth blocking 2.29.0 release on, so I will do this first.
> We are still eliminating other blockers from 2.29.0 anyhow.
>
> Kenn
>
> On Mon, Mar 15, 2021 at 7:17 AM Tomo Suzuki  wrote:
>
>> Hi Beam developers,
>>
>> I'm working on upgrading the vendored gRPC 1.36.0
>> https://issues.apache.org/jira/browse/BEAM-11227 (PR:
>> https://github.com/apache/beam/pull/14028)
>> Let me know if you have any questions or concerns.
>>
>> Background:
>> Exchanged messages with Ismaël in BEAM-11227, it seems that it the ticket
>> created by some automation is false positive, but it's nice to use an
>> artifact without being marked with CVE.
>>
>> Kenn offered to work as the release manager (as in
>> https://s.apache.org/beam-release-vendored-artifacts) of the vendored
>> artifact.
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Upgrading vendored gRPC from 1.26.0 to 1.36.0

[Tomo Suzuki]

Hi Beam developers,

I'm working on upgrading the vendored gRPC 1.36.0
https://issues.apache.org/jira/browse/BEAM-11227 (PR:
https://github.com/apache/beam/pull/14028)
Let me know if you have any questions or concerns.

Background:
Exchanged messages with Ismaël in BEAM-11227, it seems that it the ticket
created by some automation is false positive, but it's nice to use an
artifact without being marked with CVE.

Kenn offered to work as the release manager (as in
https://s.apache.org/beam-release-vendored-artifacts) of the vendored
artifact.

-- 
Regards,
Tomo

Re: Java Postcommit: MapClassIntegrationIT has been failing

[Tomo Suzuki]

Thank you for your response. Looking forward to the fix.

On Mon, Mar 8, 2021 at 2:16 AM Reuven Lax  wrote:

> That test was added by mistake - it was originally added for debugging
> (all it did was run forever and print its results to the log), and I didn't
> mean to submit it with the PR. I'll send a PR to remove that file.
>
> On Sun, Mar 7, 2021 at 8:15 PM Kenneth Knowles  wrote:
>
>> It was added in https://github.com/apache/beam/pull/13802 but was not
>> working:
>>
>>  - it was not excluded from the runner v2 test suite (where it is
>> detected and rejected with a good error message) so I excluded it in
>> https://github.com/apache/beam/pull/14111
>>  - it turns out it was also already failing on the runner v1 suite. I
>> have not followed up further.
>>
>> Kenn
>>
>> On Thu, Mar 4, 2021 at 12:31 PM Boyuan Zhang  wrote:
>>
>>> +Reuven Lax 
>>> Reuven, do you have some insights on why the test is failing?
>>>
>>> On Thu, Mar 4, 2021 at 12:28 PM Tomo Suzuki  wrote:
>>>
>>>> Hi Beam developers,
>>>>
>>>> MapClassIntegrationIT in the Java Postcommit job has been failing since
>>>> Feb 27th (5 days ago).
>>>>
>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7269/testReport/org.apache.beam.examples.cookbook/MapClassIntegrationIT/testDataflowMapState/history/
>>>>
>>>> I checked the logs (see the ticket below) but could not identify error.
>>>> The underlying Dataflow job seem to become unresponsive.
>>>>
>>>> Created a ticket: https://issues.apache.org/jira/browse/BEAM-11922
>>>>
>>>> Does anyone know how to troubleshoot this issue? (I think we need to
>>>> read the Dataflow log via Google Cloud console)
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>

-- 
Regards,
Tomo

Java Postcommit: MapClassIntegrationIT has been failing

[Tomo Suzuki]

Hi Beam developers,

MapClassIntegrationIT in the Java Postcommit job has been failing since Feb
27th (5 days ago).
https://ci-beam.apache.org/job/beam_PostCommit_Java/7269/testReport/org.apache.beam.examples.cookbook/MapClassIntegrationIT/testDataflowMapState/history/

I checked the logs (see the ticket below) but could not identify error. The
underlying Dataflow job seem to become unresponsive.

Created a ticket: https://issues.apache.org/jira/browse/BEAM-11922

Does anyone know how to troubleshoot this issue? (I think we need to read
the Dataflow log via Google Cloud console)

-- 
Regards,
Tomo

Re: Request edit access to Beam wiki

[Tomo Suzuki]

Thank you. It worked.

On Tue, Feb 16, 2021 at 11:55 PM Kenneth Knowles  wrote:

> Done. Let me know if I did not do it right.
>
> Kenn
>
> On Tue, Feb 16, 2021 at 2:08 PM Tomo Suzuki  wrote:
>
>> Hi Beam wiki admins,
>>
>> Would you grant me edit permission to the Beam wiki (Confluence)? My user
>> name is suztomo
>>
>> I'd like to update the command parameter for beam-linkage-check.sh
>> command to reflect the latest version (it now takes branch names):
>> https://cwiki.apache.org/confluence/display/BEAM/Dependency+Upgrades
>>
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Request edit access to Beam wiki

[Tomo Suzuki]

Hi Beam wiki admins,

Would you grant me edit permission to the Beam wiki (Confluence)? My user
name is suztomo

I'd like to update the command parameter for beam-linkage-check.sh command
to reflect the latest version (it now takes branch names):
https://cwiki.apache.org/confluence/display/BEAM/Dependency+Upgrades

Regards,
Tomo

Re: Upgrading non-vendored Guava version

[Tomo Suzuki]

The PR has moved to https://github.com/apache/beam/pull/13804, after
reflecting Kyle's feedback. Kyle approved this.

I look for feedback, and merge if there's no problem.

Regards,
Tomo

On Tue, Jan 19, 2021 at 6:10 PM Ahmet Altay  wrote:

> +Kenneth Knowles  +Tyson Hamilton  +Kiley
> Sok 
>
> On Thu, Jan 14, 2021 at 4:23 PM Tomo Suzuki  wrote:
>
>> Hi Beam developers,
>>
>> I'm preparing a pull request (#13740)
>> <https://github.com/apache/beam/pull/13740> that upgrades the
>> non-vendored Guava version (currently 25.1-jre) to the latest version
>> (30.1-jre). I want Beam to be built, tested, and run with the newer version
>> of Guava so that its Guava dependency works with other libraries that
>> depend on newer Guava (e.g., gcsio).
>>
>> However, certain Hadoop/Cassandra-related modules in the Beam project
>> require Guava 25 or older (Details in BEAM-11626
>> <https://issues.apache.org/jira/browse/BEAM-11626>) when they run tests.
>> Therefore, this PR attempts to split guava versions: 25
>> for Hadoop/Cassandra-related modules and 30 for the rest of the project.
>>
>> I analyzed the effect of the change in the PR below. So far I see the
>> effect is minimal thanks to the fact that Beam vendors Guava and only 3
>> modules declares non-test dependency to the non-vendored Guava.
>>
>> https://github.com/apache/beam/pull/13740
>>
>> As I am not a Hadoop/Cassandra user, I might have missed
>> important points. I appreciate it if you can share your perspective on this.
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Problems building latest beam source

[Tomo Suzuki]

Thank you. I confirmed that I can build the source now.

On Tue, Jan 26, 2021 at 7:59 PM Kyle Weaver  wrote:

> This should be fixed now.
>
> On Tue, Jan 26, 2021 at 10:23 AM Kyle Weaver  wrote:
>
>> I missed this thread and filed a JIRA for it:
>> https://issues.apache.org/jira/browse/BEAM-11689
>>
>> > We could swap the spring.io repo for the pentaho nexus one
>> public.nexus.pentaho.org (here probably in Beam
>> https://github.com/apache/beam/blob/67989cafecf3f5d4bce879e9f6b9a690955e84d5/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy#L45
>> )
>>
>> +1 I agree this is the correct solution here.
>>
>> On Tue, Jan 26, 2021 at 10:01 AM Tyson Hamilton 
>> wrote:
>>
>>> I saw this issue on another OSS repo (
>>> https://github.com/apache/hudi/issues/2479), they just removed the
>>> spring.io repository but I'm not sure where their dep would come from
>>> then. We could swap the spring.io repo for the pentaho nexus one
>>> public.nexus.pentaho.org (here probably in Beam
>>> https://github.com/apache/beam/blob/67989cafecf3f5d4bce879e9f6b9a690955e84d5/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy#L45
>>> )
>>>
>>> On Mon, Jan 25, 2021 at 2:19 PM Tomo Suzuki  wrote:
>>>
>>>> I'm seeing the same problem.
>>>>
>>>> * What went wrong:
>>>> Execution failed for task ':sdks:java:io:hcatalog:compileJava'.
>>>> > Could not resolve all files for configuration
>>>> ':sdks:java:io:hcatalog:provided'.
>>>>> Could not resolve
>>>> org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
>>>>  Required by:
>>>>  project :sdks:java:io:hcatalog >
>>>> org.apache.hive:hive-exec:2.1.0 > org.apache.calcite:calcite-core:1.6.0
>>>>   > Could not resolve
>>>> org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
>>>>  > Could not get resource '
>>>> https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>>>> '.
>>>> > Could not HEAD '
>>>> https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'.
>>>> Received status code 401 from server: Unauthorized
>>>>
>>>> On Mon, Jan 25, 2021 at 10:34 AM Steve Niemitz 
>>>> wrote:
>>>>
>>>>> I ran into issues
>>>>> resolving org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde, it looks
>>>>> like the spring repo hosting it has locked the artifact, I get a 401
>>>>> unauthorized trying to download it.
>>>>>
>>>>> Has anyone else run into this?  I assume many people have the artifact
>>>>> cached locally and so haven't run into it yet.
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>

-- 
Regards,
Tomo

Re: Problems building latest beam source

[Tomo Suzuki]

I'm seeing the same problem.

* What went wrong:
Execution failed for task ':sdks:java:io:hcatalog:compileJava'.
> Could not resolve all files for configuration
':sdks:java:io:hcatalog:provided'.
   > Could not resolve
org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
 Required by:
 project :sdks:java:io:hcatalog > org.apache.hive:hive-exec:2.1.0 >
org.apache.calcite:calcite-core:1.6.0
  > Could not resolve
org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
 > Could not get resource '
https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
'.
> Could not HEAD '
https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'.
Received status code 401 from server: Unauthorized

On Mon, Jan 25, 2021 at 10:34 AM Steve Niemitz  wrote:

> I ran into issues
> resolving org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde, it looks
> like the spring repo hosting it has locked the artifact, I get a 401
> unauthorized trying to download it.
>
> Has anyone else run into this?  I assume many people have the artifact
> cached locally and so haven't run into it yet.
>


-- 
Regards,
Tomo

Upgrading non-vendored Guava version

[Tomo Suzuki]

Hi Beam developers,

I'm preparing a pull request (#13740)
 that upgrades the non-vendored
Guava version (currently 25.1-jre) to the latest version (30.1-jre). I want
Beam to be built, tested, and run with the newer version of Guava so that
its Guava dependency works with other libraries that depend on newer Guava
(e.g., gcsio).

However, certain Hadoop/Cassandra-related modules in the Beam project
require Guava 25 or older (Details in BEAM-11626
) when they run tests.
Therefore, this PR attempts to split guava versions: 25
for Hadoop/Cassandra-related modules and 30 for the rest of the project.

I analyzed the effect of the change in the PR below. So far I see the
effect is minimal thanks to the fact that Beam vendors Guava and only 3
modules declares non-test dependency to the non-vendored Guava.

https://github.com/apache/beam/pull/13740

As I am not a Hadoop/Cassandra user, I might have missed
important points. I appreciate it if you can share your perspective on this.

-- 
Regards,
Tomo

Re: Getting ClassCastException

[Tomo Suzuki]

Hi Sonam,

Do you want to share the stack trace of the error? It would help which line
it hit the error and the function calls.

On Fri, Nov 13, 2020 at 6:40 AM Sonam Ramchand <
sonam.ramch...@venturedive.com> wrote:

> Hi Devs,
>
> I am trying to implement LOGICAL_AND  functions for Beam SQL ZetaSQL
> dialect, as CombineFn.
>
> https://github.com/sonam-vend/beam/commit/9ad8ee1d8fa617aca7fcafc8e7efe8bf388b3afb
>
> When I try to execute testLogicalAndZetaSQL, I am getting
> org.apache.beam.sdk.extensions.sql.zetasql.translation.SqlOperators$1
> cannot be cast to
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAggFunction
> java.lang.ClassCastException:
> org.apache.beam.sdk.extensions.sql.zetasql.translation.SqlOperators$1
> cannot be cast to
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAggFunction
>
> I cannot understand the reason behind occurrence of this exception.
>
> Any sort of help would be appreciated/
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296 <+92%20308%208337296>
>
> 
>


-- 
Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

This has been revised as https://github.com/apache/beam/pull/13075 and it's
merged yesterday. Now many GCP-related dependencies are managed by the BOM.
Thank you Kiley!

On Mon, Mar 30, 2020 at 10:59 AM Tomo Suzuki  wrote:

> Hi Chamikara,
>
> Ahmet is asking for your input in
> https://github.com/apache/beam/pull/11156#issuecomment-602716275 . Would
> you check this?
>
> On Mon, Mar 23, 2020 at 3:05 PM Tomo Suzuki  wrote:
>
>> (Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)
>>
>> Hi Luke,
>>
>> > the pom files that are produced
>>
>> Build.gradle that uses the BOM will produce pom files that have
>> corresponding dependencies without versions, and have a
>> "dependencyManagement" section through which they import the GCP Libraries
>> BOM. This fills the versions of the versionless dependencies when build
>> systems evaluates the pom files.
>>
>> > how the GCP BOM impacts the release process
>>
>> I'm afraid I don't have knowledge on how dependencies affect the Beam
>> release process. Would you be willing to break down this concern into some
>> questions I can answer?
>>
>>
>>
>>
>> On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik  wrote:
>>
>>> My concern would be to validate how the GCP BOM impacts the release
>>> process and the pom files that are produced otherwise the next person
>>> running a release may run into trouble.
>>>
>>> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada 
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía  wrote:
>>>>
>>>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>>>> confident to LGTM it. but given that 22 checks pass and given that
>>>>> Tomo addressed most comments and he has already a consistent track of
>>>>> good work on dependency improvements I think it is worth to merge it
>>>>> as it is. We still have some time to fix stuff if we find any
>>>>> regression. WDYT?
>>>>>
>>>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik  wrote:
>>>>> >
>>>>> > As much as I would like to spend time on these reviews, I believe
>>>>> I'll be delayed from reviewing them thoroughly till I finish other work
>>>>> that I'm targeting for the 2.21 release related to portability. It would 
>>>>> be
>>>>> greatly appreciated if there are others that could review this in the
>>>>> meantime.
>>>>> >
>>>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki 
>>>>> wrote:
>>>>> >>
>>>>> >> PR is ready (22 successful check)
>>>>> >> https://github.com/apache/beam/pull/11156
>>>>> >> (Luke assigned himself as a reviewer)
>>>>> >>
>>>>> >> Regards,
>>>>> >> Tomo
>>>>> >>
>>>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki 
>>>>> wrote:
>>>>> >>>
>>>>> >>> Thank you for favorable responses. I'll start implementation.
>>>>> >>>
>>>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki 
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> > Do Spark or Flink have BOMs?
>>>>> >>>>
>>>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1,
>>>>> 2].
>>>>> >>>>
>>>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles 
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> +1 and you have phrased the benefits and limitations well. We
>>>>> have plenty of not-Google-related dependencies that use Guava and protobuf
>>>>> (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work 
>>>>> in
>>>>> managing deps, but the BOM should make it a lot easier to upgrade all 
>>>>> these
>>>>> tightly coupled libraries that Google ships from their head.
>>>>> >>>

Re: BEAM-9958: Code Review Wanted for PR 11674

[Tomo Suzuki]

Ahmet,

Thank you for the review and merging.

On Fri, May 15, 2020 at 2:06 PM Tomo Suzuki  wrote:

> Hi Luke and Beam committers,
>
> Would you check this PR to use Linkage Checker's exclusion file?
> https://github.com/apache/beam/pull/11674
> This script used to use "diff" command to identify new linkage errors by
> comparing line by line. With this PR, it identifies new linkage errors in
> an appropriate manner. This PR opens up an opportunity to set up a Jenkins
> job to check dependency conflicts.
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

BEAM-9958: Code Review Wanted for PR 11674

[Tomo Suzuki]

Hi Luke and Beam committers,

Would you check this PR to use Linkage Checker's exclusion file?
https://github.com/apache/beam/pull/11674
This script used to use "diff" command to identify new linkage errors by
comparing line by line. With this PR, it identifies new linkage errors in
an appropriate manner. This PR opens up an opportunity to set up a Jenkins
job to check dependency conflicts.

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you run the basic precommit checks for
https://github.com/apache/beam/pull/11674 ?

Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you trigger the precommit checks for this PR?
https://github.com/apache/beam/pull/11586

Regards,
Tomo

Re: BEAM-8751: Code Review Wanted for PR 11208

[Tomo Suzuki]

Gentle reminder. Can somebody approve and merge this dependency upgrade
<https://github.com/apache/beam/pull/11208> "google-api-client 1.30.9"
https://github.com/apache/beam/pull/11208?

On Fri, Apr 3, 2020 at 11:38 AM Luke Cwik  wrote:

> Still too busy to help with this. Does anyone else have time?
>
> On Thu, Apr 2, 2020 at 7:54 PM Tomo Suzuki  wrote:
>
>> Hi Luke and Beam Committers,
>>
>> Would you review/merge this google-api-client dependency upgrade
>> https://github.com/apache/beam/pull/11208
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

BEAM-8751: Code Review Wanted for PR 11208

[Tomo Suzuki]

Hi Luke and Beam Committers,

Would you review/merge this google-api-client dependency upgrade
https://github.com/apache/beam/pull/11208

-- 
Regards,
Tomo

Re: [Proposal] Requesting PMC approval to start planning for Beam Summits 2019

[Tomo Suzuki]

2020 in email subject?

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

Hi Chamikara,

Ahmet is asking for your input in
https://github.com/apache/beam/pull/11156#issuecomment-602716275 . Would
you check this?

On Mon, Mar 23, 2020 at 3:05 PM Tomo Suzuki  wrote:

> (Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)
>
> Hi Luke,
>
> > the pom files that are produced
>
> Build.gradle that uses the BOM will produce pom files that have
> corresponding dependencies without versions, and have a
> "dependencyManagement" section through which they import the GCP Libraries
> BOM. This fills the versions of the versionless dependencies when build
> systems evaluates the pom files.
>
> > how the GCP BOM impacts the release process
>
> I'm afraid I don't have knowledge on how dependencies affect the Beam
> release process. Would you be willing to break down this concern into some
> questions I can answer?
>
>
>
>
> On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik  wrote:
>
>> My concern would be to validate how the GCP BOM impacts the release
>> process and the pom files that are produced otherwise the next person
>> running a release may run into trouble.
>>
>> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada  wrote:
>>
>>> +1
>>>
>>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía  wrote:
>>>
>>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>>> confident to LGTM it. but given that 22 checks pass and given that
>>>> Tomo addressed most comments and he has already a consistent track of
>>>> good work on dependency improvements I think it is worth to merge it
>>>> as it is. We still have some time to fix stuff if we find any
>>>> regression. WDYT?
>>>>
>>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik  wrote:
>>>> >
>>>> > As much as I would like to spend time on these reviews, I believe
>>>> I'll be delayed from reviewing them thoroughly till I finish other work
>>>> that I'm targeting for the 2.21 release related to portability. It would be
>>>> greatly appreciated if there are others that could review this in the
>>>> meantime.
>>>> >
>>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki 
>>>> wrote:
>>>> >>
>>>> >> PR is ready (22 successful check)
>>>> >> https://github.com/apache/beam/pull/11156
>>>> >> (Luke assigned himself as a reviewer)
>>>> >>
>>>> >> Regards,
>>>> >> Tomo
>>>> >>
>>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki 
>>>> wrote:
>>>> >>>
>>>> >>> Thank you for favorable responses. I'll start implementation.
>>>> >>>
>>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki 
>>>> wrote:
>>>> >>>>
>>>> >>>> > Do Spark or Flink have BOMs?
>>>> >>>>
>>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1,
>>>> 2].
>>>> >>>>
>>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles 
>>>> wrote:
>>>> >>>>>
>>>> >>>>> +1 and you have phrased the benefits and limitations well. We
>>>> have plenty of not-Google-related dependencies that use Guava and protobuf
>>>> (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>>> tightly coupled libraries that Google ships from their head.
>>>> >>>>>
>>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity
>>>> to catch incompatible deps at a larger scale by comparing and merging a
>>>> half dozen BOMs (although in the limit it approximately expands to one per
>>>> runner and one per IO and projects mature and become independent)
>>>> >>>>>
>>>> >>>>> Kenn
>>>> >>>>>
>>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik 
>>>> wrote:
>>>> >>>>>>
>

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,
(Thanks Ahmet)

Would you retrigger the following 5 jobs for
https://github.com/apache/beam/pull/11156 ?
Run Dataflow ValidatesRunner
Run Java PreCommit
Run Portable_Python PreCommit
Run PythonFormatter PreCommit
Run Website PreCommit


On Thu, Mar 26, 2020 at 9:45 PM Ahmet Altay  wrote:

> Done.
>
> On Thu, Mar 26, 2020 at 2:37 PM Tomo Suzuki  wrote:
>
>> Hi Beam committers,
>> (Thanks Ahmet for previous attempt)
>>
>> Would you trigger
>> - The failed "Run Java Precommit" for
>> https://github.com/apache/beam/pull/11208,
>> - and the precommit checks for https://github.com/apache/beam/pull/11156  
>> with
>> the following 6 checks?
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>> On Tue, Mar 24, 2020 at 6:37 PM Ahmet Altay  wrote:
>>
>>> KInd of done. 11208 is running the tests. 11156 did not start all the
>>> tests. Someone could retry that again in a bit.
>>>
>>> On Tue, Mar 24, 2020 at 12:11 PM Tomo Suzuki  wrote:
>>>
>>>> Hi Beam committers,
>>>>
>>>> Would you trigger precommit checks for
>>>> https://github.com/apache/beam/pull/11156 and
>>>> https://github.com/apache/beam/pull/11208 with the following 6 checks?
>>>> Run Java PostCommit
>>>> Run Java HadoopFormatIO Performance Test
>>>> Run BigQueryIO Streaming Performance Test Java
>>>> Run Dataflow ValidatesRunner
>>>> Run Spark ValidatesRunner
>>>> Run SQL Postcommit
>>>>
>>>>
>>>> On Fri, Mar 20, 2020 at 11:13 AM Jan Lukavský  wrote:
>>>>
>>>>> Hi, done.
>>>>> On 3/20/20 3:59 PM, Tomo Suzuki wrote:
>>>>>
>>>>> HI Beam committers,
>>>>> (Thanks Ahmet)
>>>>>
>>>>> Would you re-run the presubmit checks for
>>>>> https://github.com/apache/beam/pull/11168 with the followings?
>>>>> Run Java PostCommit
>>>>> Run Java HadoopFormatIO Performance Test
>>>>> Run BigQueryIO Streaming Performance Test Java
>>>>> Run Dataflow ValidatesRunner
>>>>> Run Spark ValidatesRunner
>>>>> Run SQL Postcommit
>>>>>
>>>>>
>>>>> On Wed, Mar 18, 2020 at 9:09 PM Ahmet Altay  wrote:
>>>>>
>>>>>> Done.
>>>>>>
>>>>>> On Wed, Mar 18, 2020 at 5:57 PM Tomo Suzuki 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Beam committers,
>>>>>>> (Alexey, thank you!)
>>>>>>>
>>>>>>> 1. Would you run the 2 failed checks Java PreCommit and Python
>>>>>>> PreCommit for https://github.com/apache/beam/pull/11156
>>>>>>>
>>>>>>> 2. Would you run the precommit checks for
>>>>>>> https://github.com/apache/beam/pull/11168 with the following 6
>>>>>>> commands?
>>>>>>> Run Java PostCommit
>>>>>>> Run Java HadoopFormatIO Performance Test
>>>>>>> Run BigQueryIO Streaming Performance Test Java
>>>>>>> Run Dataflow ValidatesRunner
>>>>>>> Run Spark ValidatesRunner
>>>>>>> Run SQL Postcommit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 18, 2020 at 1:08 PM Alexey Romanenko <
>>>>>>> aromanenko@gmail.com> wrote:
>>>>>>>
>>>>>>>> Done
>>>>>>>>
>>>>>>>> On 18 Mar 2020, at 15:25, Tomo Suzuki  wrote:
>>>>>>>>
>>>>>>>> Hi Beam committers,
>>>>>>>>
>>>>>>>> Would you trigger the precommit checks for
>>>>>>>> https://github.com/apache/beam/pull/11156 with the following 6
>>>>>>>> commands ?
>>>>>>>>
>>>>>>>> Run Java PostCommit
>>>>>>>> Run Java HadoopFormatIO Performance Test
>>>>>>>> Run BigQueryIO Streaming Performance Test Java
>>>>>>>> Run Dataflow ValidatesRunner
>>>>>>>> Run Spark ValidatesRunner
>>>>>>>> Run SQL Postcommit
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Tomo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Tomo
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Tomo
>>>>>
>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,
(Thanks Ahmet for previous attempt)

Would you trigger
- The failed "Run Java Precommit" for
https://github.com/apache/beam/pull/11208,
- and the precommit checks for https://github.com/apache/beam/pull/11156  with
the following 6 checks?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

On Tue, Mar 24, 2020 at 6:37 PM Ahmet Altay  wrote:

> KInd of done. 11208 is running the tests. 11156 did not start all the
> tests. Someone could retry that again in a bit.
>
> On Tue, Mar 24, 2020 at 12:11 PM Tomo Suzuki  wrote:
>
>> Hi Beam committers,
>>
>> Would you trigger precommit checks for
>> https://github.com/apache/beam/pull/11156 and
>> https://github.com/apache/beam/pull/11208 with the following 6 checks?
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>>
>> On Fri, Mar 20, 2020 at 11:13 AM Jan Lukavský  wrote:
>>
>>> Hi, done.
>>> On 3/20/20 3:59 PM, Tomo Suzuki wrote:
>>>
>>> HI Beam committers,
>>> (Thanks Ahmet)
>>>
>>> Would you re-run the presubmit checks for
>>> https://github.com/apache/beam/pull/11168 with the followings?
>>> Run Java PostCommit
>>> Run Java HadoopFormatIO Performance Test
>>> Run BigQueryIO Streaming Performance Test Java
>>> Run Dataflow ValidatesRunner
>>> Run Spark ValidatesRunner
>>> Run SQL Postcommit
>>>
>>>
>>> On Wed, Mar 18, 2020 at 9:09 PM Ahmet Altay  wrote:
>>>
>>>> Done.
>>>>
>>>> On Wed, Mar 18, 2020 at 5:57 PM Tomo Suzuki  wrote:
>>>>
>>>>> Hi Beam committers,
>>>>> (Alexey, thank you!)
>>>>>
>>>>> 1. Would you run the 2 failed checks Java PreCommit and Python
>>>>> PreCommit for https://github.com/apache/beam/pull/11156
>>>>>
>>>>> 2. Would you run the precommit checks for
>>>>> https://github.com/apache/beam/pull/11168 with the following 6
>>>>> commands?
>>>>> Run Java PostCommit
>>>>> Run Java HadoopFormatIO Performance Test
>>>>> Run BigQueryIO Streaming Performance Test Java
>>>>> Run Dataflow ValidatesRunner
>>>>> Run Spark ValidatesRunner
>>>>> Run SQL Postcommit
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 18, 2020 at 1:08 PM Alexey Romanenko <
>>>>> aromanenko@gmail.com> wrote:
>>>>>
>>>>>> Done
>>>>>>
>>>>>> On 18 Mar 2020, at 15:25, Tomo Suzuki  wrote:
>>>>>>
>>>>>> Hi Beam committers,
>>>>>>
>>>>>> Would you trigger the precommit checks for
>>>>>> https://github.com/apache/beam/pull/11156 with the following 6
>>>>>> commands ?
>>>>>>
>>>>>> Run Java PostCommit
>>>>>> Run Java HadoopFormatIO Performance Test
>>>>>> Run BigQueryIO Streaming Performance Test Java
>>>>>> Run Dataflow ValidatesRunner
>>>>>> Run Spark ValidatesRunner
>>>>>> Run SQL Postcommit
>>>>>>
>>>>>> Regards,
>>>>>> Tomo
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Tomo
>>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Beam dependency check failing

[Tomo Suzuki]

Thank you.

On Wed, Mar 25, 2020 at 12:42 PM Piotr Szuberski <
piotr.szuber...@polidea.com> wrote:

> It's been solved in commit #11194
>
> On 2020/03/16 12:26:10, Michał Walenia 
> wrote:
> > Hi,
> > two last builds of Beam Dependency Check job failed with errors related
> to
> > Python grpc_tools dependency.
> > https://issues.apache.org/jira/browse/BEAM-9507 here's the issue I
> created
> > to track this.
> >
> > --
> >
> > Michał Walenia
> > Polidea  | Software Engineer
> >
> > M: +48 791 432 002 <+48%20791%20432%20002> <+48791432002
> <+48%20791%20432%20002>>
> > E: michal.wale...@polidea.com
> >
> > Unique Tech
> > Check out our projects! 
> >
>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you trigger precommit checks for
https://github.com/apache/beam/pull/11156 and
https://github.com/apache/beam/pull/11208 with the following 6 checks?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit


On Fri, Mar 20, 2020 at 11:13 AM Jan Lukavský  wrote:

> Hi, done.
> On 3/20/20 3:59 PM, Tomo Suzuki wrote:
>
> HI Beam committers,
> (Thanks Ahmet)
>
> Would you re-run the presubmit checks for
> https://github.com/apache/beam/pull/11168 with the followings?
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
>
> On Wed, Mar 18, 2020 at 9:09 PM Ahmet Altay  wrote:
>
>> Done.
>>
>> On Wed, Mar 18, 2020 at 5:57 PM Tomo Suzuki  wrote:
>>
>>> Hi Beam committers,
>>> (Alexey, thank you!)
>>>
>>> 1. Would you run the 2 failed checks Java PreCommit and Python PreCommit
>>> for https://github.com/apache/beam/pull/11156
>>>
>>> 2. Would you run the precommit checks for
>>> https://github.com/apache/beam/pull/11168 with the following 6 commands?
>>> Run Java PostCommit
>>> Run Java HadoopFormatIO Performance Test
>>> Run BigQueryIO Streaming Performance Test Java
>>> Run Dataflow ValidatesRunner
>>> Run Spark ValidatesRunner
>>> Run SQL Postcommit
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Mar 18, 2020 at 1:08 PM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
>>>> Done
>>>>
>>>> On 18 Mar 2020, at 15:25, Tomo Suzuki  wrote:
>>>>
>>>> Hi Beam committers,
>>>>
>>>> Would you trigger the precommit checks for
>>>> https://github.com/apache/beam/pull/11156 with the following 6
>>>> commands ?
>>>>
>>>> Run Java PostCommit
>>>> Run Java HadoopFormatIO Performance Test
>>>> Run BigQueryIO Streaming Performance Test Java
>>>> Run Dataflow ValidatesRunner
>>>> Run Spark ValidatesRunner
>>>> Run SQL Postcommit
>>>>
>>>> Regards,
>>>> Tomo
>>>>
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>
> --
> Regards,
> Tomo
>
>

-- 
Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

(Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)

Hi Luke,

> the pom files that are produced

Build.gradle that uses the BOM will produce pom files that have
corresponding dependencies without versions, and have a
"dependencyManagement" section through which they import the GCP Libraries
BOM. This fills the versions of the versionless dependencies when build
systems evaluates the pom files.

> how the GCP BOM impacts the release process

I'm afraid I don't have knowledge on how dependencies affect the Beam
release process. Would you be willing to break down this concern into some
questions I can answer?




On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik  wrote:

> My concern would be to validate how the GCP BOM impacts the release
> process and the pom files that are produced otherwise the next person
> running a release may run into trouble.
>
> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada  wrote:
>
>> +1
>>
>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía  wrote:
>>
>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>> confident to LGTM it. but given that 22 checks pass and given that
>>> Tomo addressed most comments and he has already a consistent track of
>>> good work on dependency improvements I think it is worth to merge it
>>> as it is. We still have some time to fix stuff if we find any
>>> regression. WDYT?
>>>
>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik  wrote:
>>> >
>>> > As much as I would like to spend time on these reviews, I believe I'll
>>> be delayed from reviewing them thoroughly till I finish other work that I'm
>>> targeting for the 2.21 release related to portability. It would be greatly
>>> appreciated if there are others that could review this in the meantime.
>>> >
>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki 
>>> wrote:
>>> >>
>>> >> PR is ready (22 successful check)
>>> >> https://github.com/apache/beam/pull/11156
>>> >> (Luke assigned himself as a reviewer)
>>> >>
>>> >> Regards,
>>> >> Tomo
>>> >>
>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki 
>>> wrote:
>>> >>>
>>> >>> Thank you for favorable responses. I'll start implementation.
>>> >>>
>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki 
>>> wrote:
>>> >>>>
>>> >>>> > Do Spark or Flink have BOMs?
>>> >>>>
>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>> >>>>
>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles 
>>> wrote:
>>> >>>>>
>>> >>>>> +1 and you have phrased the benefits and limitations well. We have
>>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>> tightly coupled libraries that Google ships from their head.
>>> >>>>>
>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to
>>> catch incompatible deps at a larger scale by comparing and merging a half
>>> dozen BOMs (although in the limit it approximately expands to one per
>>> runner and one per IO and projects mature and become independent)
>>> >>>>>
>>> >>>>> Kenn
>>> >>>>>
>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik 
>>> wrote:
>>> >>>>>>
>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>> >>>>>>
>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
>>> filiperega...@gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Big +1, this is a step in the right direction and checking with
>>> other Beam's direct and transitive deps is crucial since the referred bom
>>> only convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>> apis, cloud}`, slf4j comes to mi

Re: Issue in testing doing DirectRunner Apache Beam Local Machine

[Tomo Suzuki]

Yi Yogesh,

It seems you have incompatible dependencies. Would you share your pom.xml?
I'm interested in "dependencies" and "dependencyManagement" sections.

Regards,
Tomo

On Fri, Mar 20, 2020 at 12:43 PM Austin Bennett 
wrote:

> Hi Yogesh,
>
> Many of the examples, use the direct runner:
>
> https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples
>
> As does the word count example:
> https://beam.apache.org/get-started/quickstart-java/#get-the-wordcount-code
>
> I'm not well versed enough with our Java SDK (and dependencies for it), to
> recognize the error you shared.
>
> Good luck!
> Austin
>
>
> On Fri, Mar 20, 2020 at 9:25 AM Kumbhare, Yogesh <
> yogesh_kumbh...@intuit.com> wrote:
>
>> Hi Team,
>>
>>
>>
>> Please find the below command to build .
>>
>>
>>
>> mvn compile exec:java
>> -Dexec.mainClass=com.intuit.dedupe.beam.poc.StreamingDedupe -Pdirect-runner
>>
>>
>>
>> is there any DirectRunner example you have please do share with us.
>>
>>
>>
>>
>>
>> Thanks ,
>>
>> Yogesh K
>>
>> 8050397434
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *"Kumbhare, Yogesh" 
>> *Date: *Friday, 20 March 2020 at 3:03 PM
>> *To: *"dev@beam.apache.org" 
>> *Subject: *Issue in testing doing DirectRunner Apache Beam Local Machine
>>
>>
>>
>> Hi Team ,
>>
>>
>>
>> I am facing issue to run the simple project on Direct Runner Apache beam
>> , to test the example locally in our machine .
>>
>>
>>
>> I added in pom.xml dependency ,
>>
>>
>>
>> 
>>
>>org.apache.beam
>>
>>beam-runners-direct-java
>>
>>2.19.0
>>
>>runtime
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> Please find the below error , let me know any update .
>>
>>
>>
>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Class
>> org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives
>> does not implement the requested interface
>> org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ProtocolMessageEnum
>>
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.construction.BeamUrns.getUrn(
>> BeamUrns.java:27)
>>
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.construction.PTransformTranslation.(
>> PTransformTranslation.java:129)
>>
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.construction.PTransformMatchers.lambda$writeWithRunnerDeterminedSharding$1(
>> PTransformMatchers.java:483)
>>
>> at
>> org.apache.beam.sdk.Pipeline$2.enterCompositeTransform(Pipeline.java:268)
>>
>> at
>> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(
>> TransformHierarchy.java:645)
>>
>> at
>> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(
>> TransformHierarchy.java:649)
>>
>> at
>> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(
>> TransformHierarchy.java:311)
>>
>> at org.apache.beam.sdk.runners.TransformHierarchy.visit(
>> TransformHierarchy.java:245)
>>
>> at org.apache.beam.sdk.Pipeline.traverseTopologically(
>> Pipeline.java:458)
>>
>> at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:258
>> )
>>
>> at org.apache.beam.sdk.Pipeline.replaceAll(
>> Pipeline.java:208)
>>
>> at org.apache.beam.runners.direct.DirectRunner.run(
>> DirectRunner.java:170)
>>
>> at org.apache.beam.runners.direct.DirectRunner.run(
>> DirectRunner.java:67)
>>
>> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
>>
>> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
>>
>> at com.intuit.dedupe.beam.poc.StreamingDedupe.main(
>> StreamingDedupe.java:144)
>>
>>
>>
>>
>>
>>
>>
>

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

HI Beam committers,
(Thanks Ahmet)

Would you re-run the presubmit checks for
https://github.com/apache/beam/pull/11168 with the followings?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit


On Wed, Mar 18, 2020 at 9:09 PM Ahmet Altay  wrote:

> Done.
>
> On Wed, Mar 18, 2020 at 5:57 PM Tomo Suzuki  wrote:
>
>> Hi Beam committers,
>> (Alexey, thank you!)
>>
>> 1. Would you run the 2 failed checks Java PreCommit and Python PreCommit
>> for https://github.com/apache/beam/pull/11156
>>
>> 2. Would you run the precommit checks for
>> https://github.com/apache/beam/pull/11168 with the following 6 commands?
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>>
>>
>>
>>
>> On Wed, Mar 18, 2020 at 1:08 PM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Done
>>>
>>> On 18 Mar 2020, at 15:25, Tomo Suzuki  wrote:
>>>
>>> Hi Beam committers,
>>>
>>> Would you trigger the precommit checks for
>>> https://github.com/apache/beam/pull/11156 with the following 6 commands
>>> ?
>>>
>>> Run Java PostCommit
>>> Run Java HadoopFormatIO Performance Test
>>> Run BigQueryIO Streaming Performance Test Java
>>> Run Dataflow ValidatesRunner
>>> Run Spark ValidatesRunner
>>> Run SQL Postcommit
>>>
>>> Regards,
>>> Tomo
>>>
>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

PR is ready (22 successful check)
https://github.com/apache/beam/pull/11156
(Luke assigned himself as a reviewer)

Regards,
Tomo

On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki  wrote:

> Thank you for favorable responses. I'll start implementation.
>
> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki  wrote:
>
>> > Do Spark or Flink have BOMs?
>>
>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>
>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>
>>
>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles  wrote:
>>
>>> +1 and you have phrased the benefits and limitations well. We have
>>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>> tightly coupled libraries that Google ships from their head.
>>>
>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
>>> incompatible deps at a larger scale by comparing and merging a half dozen
>>> BOMs (although in the limit it approximately expands to one per runner and
>>> one per IO and projects mature and become independent)
>>>
>>> Kenn
>>>
>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik  wrote:
>>>
>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>>
>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas 
>>>> wrote:
>>>>
>>>>> Big +1, this is a step in the right direction and checking with other
>>>>> Beam's direct and transitive deps is crucial since the referred bom only
>>>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>>> apis, cloud}`, slf4j comes to mind.
>>>>>
>>>>> Filipe Regadas
>>>>>
>>>>>
>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía  wrote:
>>>>>
>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel 
>>>>>> wrote:
>>>>>> >
>>>>>> > +1, I can remember the countless hours that we fought with Google
>>>>>> dependencies.
>>>>>> >
>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath 
>>>>>> wrote:
>>>>>> >>
>>>>>> >> +1 for this.
>>>>>> >>
>>>>>> >> This will make life easy for many of our users and will help us
>>>>>> keep GCP related dependencies compatible (which has not been easy in the
>>>>>> past).
>>>>>> >>
>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki 
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi Beam developers,
>>>>>> >>>
>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>>>> library versions in Beam?
>>>>>> >>>
>>>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>>>> months. It's time to consider a long-term solution to keep the libraries
>>>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>>>> to use GCP Libraries BOM to set the Google-related library versions, 
>>>>>> rather
>>>>>> than the current way of making changes in each of ~30 Google libraries 
>>>>>> with
>>>>>> individual PRs [2].
>>>>>> >>>
>>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>>>> version to upgrade these Google-related libraries. This still needs to
>>>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which 
>>>>>> we
>>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort 
>>>>>> in
>>>>>> Beam's developers.
>>>>>> >>>
>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if
>>>>>> you can share questions or feedback (thumbs-up / concerns).
>>>>>> >>>
>>>>>> >>> [1]:
>>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>>> >>> [2]:
>>>>>> https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Asuztomo
>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Regards,
>>>>>> >>> Tomo
>>>>>>
>>>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,
(Alexey, thank you!)

1. Would you run the 2 failed checks Java PreCommit and Python PreCommit
for https://github.com/apache/beam/pull/11156

2. Would you run the precommit checks for
https://github.com/apache/beam/pull/11168 with the following 6 commands?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit





On Wed, Mar 18, 2020 at 1:08 PM Alexey Romanenko 
wrote:

> Done
>
> On 18 Mar 2020, at 15:25, Tomo Suzuki  wrote:
>
> Hi Beam committers,
>
> Would you trigger the precommit checks for
> https://github.com/apache/beam/pull/11156 with the following 6 commands ?
>
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
> Regards,
> Tomo
>
>
>

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you trigger the precommit checks for
https://github.com/apache/beam/pull/11156 with the following 6 commands ?

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: /zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR

[Tomo Suzuki]

I see. Thank you. I also found Alex's ticket for zetasql.
https://github.com/google/zetasql/issues/25

Closing thread.

Regards,
Tomo

On Mon, Mar 16, 2020 at 3:21 PM Andrew Pilloud  wrote:

> That error is expected unless you've built your own jar with
> liblocal_service_jni.jnilib for OS X. A few have tried but no one has
> succeeded (as far as I know), it is on the ZetaSQL team's todo list. You'll
> need to run that module on Linux for now.
>
> See: https://github.com/google/zetasql/pull/3
>
> Andrew
>
> On Mon, Mar 16, 2020 at 12:09 PM Tomo Suzuki  wrote:
>
>> Hi Beam developers,
>>
>> I started getting test failures when building Beam in my MacBook Pro.
>> Module: sdks/java/extensions/sql/zetasql. The NoClassDefFoundError occurs
>> because of jnilib file is missing.
>>
>> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
>> File /zetasql/local_service/liblocal_service_jni.jnilib was not found
>> inside JAR.
>> at
>> com.google.zetasql.JniChannelProvider.(JniChannelProvider.java:68)
>> ... 69 more
>> Caused by: java.io.FileNotFoundException: File
>> /zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR.
>> at
>> com.google.zetasql.cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:105)
>> at
>> com.google.zetasql.JniChannelProvider.(JniChannelProvider.java:66)
>> ... 69 more
>>
>> Full log:
>> https://gist.github.com/suztomo/f3d8815e8f48aeabd0288de34c1488f0
>>
>> Has anyone encountered a similar problem?
>>
>> --
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

/zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR

[Tomo Suzuki]

Hi Beam developers,

I started getting test failures when building Beam in my MacBook Pro.
Module: sdks/java/extensions/sql/zetasql. The NoClassDefFoundError occurs
because of jnilib file is missing.

Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
/zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR.
at
com.google.zetasql.JniChannelProvider.(JniChannelProvider.java:68)
... 69 more
Caused by: java.io.FileNotFoundException: File
/zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR.
at
com.google.zetasql.cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:105)
at
com.google.zetasql.JniChannelProvider.(JniChannelProvider.java:66)
... 69 more

Full log: https://gist.github.com/suztomo/f3d8815e8f48aeabd0288de34c1488f0

Has anyone encountered a similar problem?

-- 
Regards,
Tomo

Re: Hello Beam Community!

[Tomo Suzuki]

Welcome.

On Fri, Mar 13, 2020 at 1:20 PM Udi Meiri  wrote:

> Welcome!
>
>
> On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang  wrote:
>
>> Welcome!
>>
>> On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay  wrote:
>>
>>> Welcome Brittany!
>>>
>>> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
>>> wrote:
>>>
 Hello Beam Community!

 My name is Brittany Hermann and I recently joined the Open Source team
 in Data Analytics at Google. As a Program Manager, I will be focusing on
 community engagement while getting to work on Apache Beam and Airflow
 projects! I have always thrived on creating healthy, diverse, and overall
 happy communities and am excited to bring that to the team. For a fun fact,
 I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
 named Ollie!

 I look forward to collaborating with you all!

 Kind regards,

 Brittany Hermann




-- 
Regards,
Tomo

Re: [Newbie Contributing Question] How to add a dependency?

[Tomo Suzuki]

Jacob,

It works fine for me.
https://gist.github.com/suztomo/d90951a2c028c7406d87b13bd43e6eaa
Do you want to share the output?


On Wed, Mar 11, 2020 at 8:49 PM Jacob Ferriero  wrote:

> Thanks that was helpful in getting my dependencies added!
> I am not having a an issue where the testCompile dependencies such as
> junit aren't getting resolved when running
> ```
>  ./gradlew -p sdks/java/io/google-cloud-platform/ test
> ```
>  I didn't touch these. Is there some place I have to add new test
> directory?
>
> On 2020/03/11 04:17:50, Tomo Suzuki  wrote:
> > Hi Jacob,>
> >
> > You'll need to modify BeamModulePlugin.groovy, which defines a big map
> of>
> > Maven artifacts. The map is referenced by each module such as the>
> > google-cloud-platform/build.gradle.>
> > Example PR to touch dependencies:>
> > https://github.com/apache/beam/pull/11063/files>
> >
> > On Tue, Mar 10, 2020 at 11:22 PM Jacob Ferriero >
> > wrote:>
> >
> > > Hi beam dev list,>
> > >>
> > > Hoping to find some pointers on how to best add a dependency for a new
> GCP>
> > > IO connector.>
> > > Specifically I want to add a dependency on the Cloud Healthcare API or
> the>
> > > equivalent of this section of a maven pom.xml:>
> > > ```xml>
> > > >
> > >
> v1alpha2-rev20190901-1.30.1>
> > >   1.30.1>
> > > >
> > > >
> > >   com.google.http-client>
> > >   google-http-client-gson>
> > >   ${http.client.version}>
> > > >
> > > >
> > >   com.google.apis>
> > >   google-api-services-healthcare>
> > >   ${healthcare.version}>
> > > 
> > > ```>
> > > I know I'm one step away looking at this build.gradle>
> > >
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/build.gradle#L30>
>
> > >>
> > > *Background:*>
> > > I'm a first time contributor working on a PR for BEAM-9468>
> > > <https://issues.apache.org/jira/projects/BEAM/issues/BEAM-9468> an
> issue>
> > > I opened to create a new GCP IO connector for a series of use cases we
> have>
> > > in the healthcare space.>
> > > I'm completely new to gradle. I started my development in a fork of>
> > > https://github.com/GoogleCloudPlatform/DataflowTemplates which uses>
> > > maven but then decided these would better be to contribute these IO>
> > > connectors upstream.>
> > >>
> > > -->
> > >>
> > > *Jacob Ferriero*>
> > >>
> > > Strategic Cloud Engineer: Data Engineering>
> > >>
> > > jferri...@google.com>
> > >>
> > > 617-714-2509 <(617)%20714-2509> <(617)%20714-2509>>
> > >>
> >
> >
> > -- >
> > Regards,>
> > Tomo>
> >
>


-- 
Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

Thank you for favorable responses. I'll start implementation.

On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki  wrote:

> > Do Spark or Flink have BOMs?
>
> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>
> [1]: https://search.maven.org/search?q=g:org.apache.flink
> [2]: https://search.maven.org/search?q=g:org.apache.spark
>
>
> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles  wrote:
>
>> +1 and you have phrased the benefits and limitations well. We have plenty
>> of not-Google-related dependencies that use Guava and protobuf (I know of
>> Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing
>> deps, but the BOM should make it a lot easier to upgrade all these tightly
>> coupled libraries that Google ships from their head.
>>
>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
>> incompatible deps at a larger scale by comparing and merging a half dozen
>> BOMs (although in the limit it approximately expands to one per runner and
>> one per IO and projects mature and become independent)
>>
>> Kenn
>>
>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik  wrote:
>>
>>> How would the Apache Beam BOM and GCP BOM work together?
>>>
>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas 
>>> wrote:
>>>
>>>> Big +1, this is a step in the right direction and checking with other
>>>> Beam's direct and transitive deps is crucial since the referred bom only
>>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>> apis, cloud}`, slf4j comes to mind.
>>>>
>>>> Filipe Regadas
>>>>
>>>>
>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía  wrote:
>>>>
>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>
>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel 
>>>>> wrote:
>>>>> >
>>>>> > +1, I can remember the countless hours that we fought with Google
>>>>> dependencies.
>>>>> >
>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath 
>>>>> wrote:
>>>>> >>
>>>>> >> +1 for this.
>>>>> >>
>>>>> >> This will make life easy for many of our users and will help us
>>>>> keep GCP related dependencies compatible (which has not been easy in the
>>>>> past).
>>>>> >>
>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki 
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi Beam developers,
>>>>> >>>
>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>>> library versions in Beam?
>>>>> >>>
>>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>>> months. It's time to consider a long-term solution to keep the libraries
>>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>>> to use GCP Libraries BOM to set the Google-related library versions, 
>>>>> rather
>>>>> than the current way of making changes in each of ~30 Google libraries 
>>>>> with
>>>>> individual PRs [2].
>>>>> >>>
>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>>> version to upgrade these Google-related libraries. This still needs to
>>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort 
>>>>> in
>>>>> Beam's developers.
>>>>> >>>
>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>>>> can share questions or feedback (thumbs-up / concerns).
>>>>> >>>
>>>>> >>> [1]:
>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>> >>> [2]:
>>>>> https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Asuztomo
>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>> >>>
>>>>> >>> --
>>>>> >>> Regards,
>>>>> >>> Tomo
>>>>>
>>>>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: [Newbie Contributing Question] How to add a dependency?

[Tomo Suzuki]

Hi Jacob,

You'll need to modify BeamModulePlugin.groovy, which defines a big map of
Maven artifacts. The map is referenced by each module such as the
google-cloud-platform/build.gradle.
Example PR to touch dependencies:
https://github.com/apache/beam/pull/11063/files

On Tue, Mar 10, 2020 at 11:22 PM Jacob Ferriero 
wrote:

> Hi beam dev list,
>
> Hoping to find some pointers on how to best add a dependency for a new GCP
> IO connector.
> Specifically I want to add a dependency on the Cloud Healthcare API or the
> equivalent of this section of a maven pom.xml:
> ```xml
> 
>   v1alpha2-rev20190901-1.30.1
>   1.30.1
> 
> 
>   com.google.http-client
>   google-http-client-gson
>   ${http.client.version}
> 
> 
>   com.google.apis
>   google-api-services-healthcare
>   ${healthcare.version}
>  ```
> I know I'm one step away looking at this build.gradle
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/build.gradle#L30
>
> *Background:*
> I'm a first time contributor working on a PR for BEAM-9468
>  an issue
> I opened to create a new GCP IO connector for a series of use cases we have
> in the healthcare space.
> I'm completely new to gradle. I started my development in a fork of
> https://github.com/GoogleCloudPlatform/DataflowTemplates which uses
> maven but then decided these would better be to contribute these IO
> connectors upstream.
>
> --
>
> *Jacob Ferriero*
>
> Strategic Cloud Engineer: Data Engineering
>
> jferri...@google.com
>
> 617-714-2509 <(617)%20714-2509>
>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you trigger precomimt checks for
https://github.com/apache/beam/pull/11095 with the following 6 commands ?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Thank you, Luke. Now Java Precommit is green.

On Mon, Mar 9, 2020 at 2:21 PM Tomo Suzuki  wrote:

> Java PreCommit failed due to Flink's UnboundedSourceWrapperTest
> (BEAM-9164).
> Can somebody retrigger "Run Java PreCommit" for
> https://github.com/apache/beam/pull/11063 ?
>
>
> On Mon, Mar 9, 2020 at 9:58 AM Tomo Suzuki  wrote:
>
>> Hi Beam committers,
>>
>> 2 of the previous checks didn't succeed. Would you retrigger
>> Run Java PreCommit
>> Run Java_Examples_Dataflow_Java11 PreCommit
>> for https://github.com/apache/beam/pull/11063 ?
>>
>> Regards,
>> Tomo
>>
>> On Fri, Mar 6, 2020 at 9:31 PM Luke Cwik  wrote:
>>
>>> Done
>>>
>>> On Fri, Mar 6, 2020 at 10:35 AM Tomo Suzuki  wrote:
>>>
>>>> Hi Beam committers,
>>>> (Thank you, Michał!)
>>>>
>>>> Would you trigger the precommit checks for
>>>> https://github.com/apache/beam/pull/11063
>>>> with the following 6 commands ?
>>>> Run Java PostCommit
>>>> Run Java HadoopFormatIO Performance Test
>>>> Run BigQueryIO Streaming Performance Test Java
>>>> Run Dataflow ValidatesRunner
>>>> Run Spark ValidatesRunner
>>>> Run SQL Postcommit
>>>>
>>>> Regards,
>>>> Tomo
>>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Java PreCommit failed due to Flink's UnboundedSourceWrapperTest (BEAM-9164).
Can somebody retrigger "Run Java PreCommit" for
https://github.com/apache/beam/pull/11063 ?


On Mon, Mar 9, 2020 at 9:58 AM Tomo Suzuki  wrote:

> Hi Beam committers,
>
> 2 of the previous checks didn't succeed. Would you retrigger
> Run Java PreCommit
> Run Java_Examples_Dataflow_Java11 PreCommit
> for https://github.com/apache/beam/pull/11063 ?
>
> Regards,
> Tomo
>
> On Fri, Mar 6, 2020 at 9:31 PM Luke Cwik  wrote:
>
>> Done
>>
>> On Fri, Mar 6, 2020 at 10:35 AM Tomo Suzuki  wrote:
>>
>>> Hi Beam committers,
>>> (Thank you, Michał!)
>>>
>>> Would you trigger the precommit checks for
>>> https://github.com/apache/beam/pull/11063
>>> with the following 6 commands ?
>>> Run Java PostCommit
>>> Run Java HadoopFormatIO Performance Test
>>> Run BigQueryIO Streaming Performance Test Java
>>> Run Dataflow ValidatesRunner
>>> Run Spark ValidatesRunner
>>> Run SQL Postcommit
>>>
>>> Regards,
>>> Tomo
>>>
>>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

2 of the previous checks didn't succeed. Would you retrigger
Run Java PreCommit
Run Java_Examples_Dataflow_Java11 PreCommit
for https://github.com/apache/beam/pull/11063 ?

Regards,
Tomo

On Fri, Mar 6, 2020 at 9:31 PM Luke Cwik  wrote:

> Done
>
> On Fri, Mar 6, 2020 at 10:35 AM Tomo Suzuki  wrote:
>
>> Hi Beam committers,
>> (Thank you, Michał!)
>>
>> Would you trigger the precommit checks for
>> https://github.com/apache/beam/pull/11063
>> with the following 6 commands ?
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>> Regards,
>> Tomo
>>
>

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Thanks, Luke!

On Fri, Mar 6, 2020 at 13:34 Tomo Suzuki  wrote:

> Hi Beam committers,
> (Thank you, Michał!)
>
> Would you trigger the precommit checks for
> https://github.com/apache/beam/pull/11063
> with the following 6 commands ?
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
> Regards,
> Tomo
>
-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,
(Thank you, Michał!)

Would you trigger the precommit checks for
https://github.com/apache/beam/pull/11063
with the following 6 commands ?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

> Do Spark or Flink have BOMs?

Not that I know of. I couldn't find "bom" in their artifacts [1, 2].

[1]: https://search.maven.org/search?q=g:org.apache.flink
[2]: https://search.maven.org/search?q=g:org.apache.spark


On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles  wrote:

> +1 and you have phrased the benefits and limitations well. We have plenty
> of not-Google-related dependencies that use Guava and protobuf (I know of
> Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing
> deps, but the BOM should make it a lot easier to upgrade all these tightly
> coupled libraries that Google ships from their head.
>
> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
> incompatible deps at a larger scale by comparing and merging a half dozen
> BOMs (although in the limit it approximately expands to one per runner and
> one per IO and projects mature and become independent)
>
> Kenn
>
> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik  wrote:
>
>> How would the Apache Beam BOM and GCP BOM work together?
>>
>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas 
>> wrote:
>>
>>> Big +1, this is a step in the right direction and checking with other
>>> Beam's direct and transitive deps is crucial since the referred bom only
>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>> apis, cloud}`, slf4j comes to mind.
>>>
>>> Filipe Regadas
>>>
>>>
>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía  wrote:
>>>
>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>
>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel  wrote:
>>>> >
>>>> > +1, I can remember the countless hours that we fought with Google
>>>> dependencies.
>>>> >
>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath 
>>>> wrote:
>>>> >>
>>>> >> +1 for this.
>>>> >>
>>>> >> This will make life easy for many of our users and will help us keep
>>>> GCP related dependencies compatible (which has not been easy in the past).
>>>> >>
>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki 
>>>> wrote:
>>>> >>>
>>>> >>> Hi Beam developers,
>>>> >>>
>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>> library versions in Beam?
>>>> >>>
>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>> months. It's time to consider a long-term solution to keep the libraries
>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>>> than the current way of making changes in each of ~30 Google libraries with
>>>> individual PRs [2].
>>>> >>>
>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>> version to upgrade these Google-related libraries. This still needs to
>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>> Beam's developers.
>>>> >>>
>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>>> can share questions or feedback (thumbs-up / concerns).
>>>> >>>
>>>> >>> [1]:
>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>> >>> [2]:
>>>> https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Asuztomo
>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>> >>>
>>>> >>> --
>>>> >>> Regards,
>>>> >>> Tomo
>>>>
>>>

-- 
Regards,
Tomo

Re: Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

> How would the Apache Beam BOM and GCP BOM work together?

I envision there will be (new) "Beam GCP BOM" that imports (existing) Beam
BOM and GCP Libraries BOM with necessary overwrites (such as Guava version).
This clarifies which versions of Google libraries should be compatible with
Beam's version.


On Thu, Mar 5, 2020 at 1:05 PM Luke Cwik  wrote:

> How would the Apache Beam BOM and GCP BOM work together?
>
> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas 
> wrote:
>
>> Big +1, this is a step in the right direction and checking with other
>> Beam's direct and transitive deps is crucial since the referred bom only
>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>> apis, cloud}`, slf4j comes to mind.
>>
>> Filipe Regadas
>>
>>
>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía  wrote:
>>
>>> +1 Sounds like a good improvement for users and maintainers !
>>>
>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel  wrote:
>>> >
>>> > +1, I can remember the countless hours that we fought with Google
>>> dependencies.
>>> >
>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath 
>>> wrote:
>>> >>
>>> >> +1 for this.
>>> >>
>>> >> This will make life easy for many of our users and will help us keep
>>> GCP related dependencies compatible (which has not been easy in the past).
>>> >>
>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki 
>>> wrote:
>>> >>>
>>> >>> Hi Beam developers,
>>> >>>
>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>> library versions in Beam?
>>> >>>
>>> >>> I've been working on Beam's dependency upgrades in the past few
>>> months. It's time to consider a long-term solution to keep the libraries
>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>> than the current way of making changes in each of ~30 Google libraries with
>>> individual PRs [2].
>>> >>>
>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>> version to upgrade these Google-related libraries. This still needs to
>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>> dependencies. (Linkage Checker will help with this job.) I believe
>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>> Beam's developers.
>>> >>>
>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>> can share questions or feedback (thumbs-up / concerns).
>>> >>>
>>> >>> [1]:
>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>> >>> [2]:
>>> https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Asuztomo
>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>> >>>
>>> >>> --
>>> >>> Regards,
>>> >>> Tomo
>>>
>>

-- 
Regards,
Tomo

Proposal: Beam to use GCP Libraries BOM

[Tomo Suzuki]

Hi Beam developers,

Shall we use GCP Libraries BOM [1] to specify the Google-related library
versions in Beam?

I've been working on Beam's dependency upgrades in the past few months.
It's time to consider a long-term solution to keep the libraries up-to-date
with small maintenance effort. To achieve that, I propose Beam to use GCP
Libraries BOM to set the Google-related library versions, rather than the
current way of making changes in each of ~30 Google libraries with
individual PRs [2].

After the proposal is implemented, Beam project upgrades the BOM version to
upgrade these Google-related libraries. This still needs to ensure the
libraries in GCP Library BOM are compatible with Beam's other dependencies.
(Linkage Checker will help with this job.) I believe onboarding GCP
Libraries BOM will solve lots of incompatibilities which we have seen in
gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's
developers.

Created an issue to track this: BEAM-9444 [3]. I appreciate if you can
share questions or feedback (thumbs-up / concerns).

[1]:
https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
[2]: https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Asuztomo
[3]: https://issues.apache.org/jira/browse/BEAM-9444

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you trigger the precommit checks for
https://github.com/apache/beam/pull/11042
with the following 6 commands ?

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Ahmet, thanks. But it seems Jenkins is not reporting the status
correctly. Will check tomorrow.

On Thu, Feb 13, 2020 at 2:45 PM Tomo Suzuki  wrote:
>
> Hi Beam committers,
>
> Would you run precommit checks on https://github.com/apache/beam/pull/10765
> with the following 6 additional commands?
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
> Regards,
> Tomo



-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Would you run precommit checks on https://github.com/apache/beam/pull/10765
with the following 6 additional commands?
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Upgrades gcsio to 2.0.0

[Tomo Suzuki]

> What prevents the usage of the newer version of Guava?

Cassandra is referencing an old field of Guava's CharMatcher. The
field "DIGIT" is no longer available after Guava 26.
https://github.com/apache/beam/pull/10769#issuecomment-583698718

On Mon, Feb 10, 2020 at 5:39 PM Luke Cwik  wrote:
>
> What prevents the usage of the newer version of Guava?
>
> On Mon, Feb 10, 2020 at 2:28 PM Esun Kim  wrote:
>>
>> Hi Beam Developers,
>>
>> I'm working on pr/10769 which upgrades gcsio from 1.9.16 to 2.0.0 which is 
>> an intermediate step to get us to use gcsio 2.x which supports gRPC, which 
>> potentially gives us better performance. (FYI, gcsio is a driver for Google 
>> Cloud Storage.)
>>
>> Link-check was run over this PR (result) and it appears that it has a couple 
>> of linker warnings from following modules because this uses a newer version 
>> of guava.
>>
>> com.google.cloud.hadoop.gcsio.cooplock.CoopLockRecordsDao (gcsio-2.0.0.jar)
>> com.google.cloud.hadoop.gcsio.cooplock.CoopLockOperationDao (gcsio-2.0.0.jar)
>> com.google.cloud.hadoop.gcsio.testing.InMemoryObjectEntry (gcsio-2.0.0.jar)
>>
>> But I believe that none of these is not actually problematic because 
>> cooplock is only for Hadoop (not for Beam) and testing is just testing. So I 
>> think it's okay to get this merged but I want to get an opinion on this from 
>> you.
>>
>> Regards,
>> Esun.
>>


-- 
Regards,
Tomo

BEAM-8758: Code Review Wanted for PR 10765

[Tomo Suzuki]

Hi Udi, Luke, and Beam committers,

Would you review/merge this google-cloud-spanner dependency upgrade?
https://github.com/apache/beam/pull/10765

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

I appreciate if you can run precommit checks for
https://github.com/apache/beam/pull/10769
with the following 6 commands:

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

[Tomo Suzuki]

Ismaël,

The latest postcommit failure
https://builds.apache.org/job/beam_PostCommit_SQL/3951/ was 4:15:45 PM
Brian's successful case
https://builds.apache.org/job/beam_PostCommit_SQL_PR/243/ started
4:29:57 PM.
I hope the next SQL postcommit should succeed.

On Thu, Feb 6, 2020 at 11:57 AM Ismaël Mejía  wrote:
>
> This one is still broken, maybe there are two different data sources, one for 
> the '_PR' version and the normal one, can you please confirm?
> https://builds.apache.org/job/beam_PostCommit_SQL/
>
>
> On Thu, Feb 6, 2020 at 5:44 PM Brian Hulette  wrote:
>>
>> Sorry for the delay. I had some issues updating the schema, I ended up 
>> having to drop it and re-create for some reason. Looks like SQL PostCommit 
>> is green on https://github.com/apache/beam/pull/10765 now.
>>
>> > setting up from scratch is a good idea.
>> +1, I filed https://issues.apache.org/jira/browse/BEAM-9260 for this and 
>> assigned it to Kenn for now.
>>
>> > I’m still feeling it’s ok to read an Integer as Long.
>> In this case the issue is that we were reading data with a schema (id: 
>> INT64), and then passing it to a PAssert that checked against Rows with a 
>> different schema (id: INT32). So I think it's a legitimate error because we 
>> were asserting that a PCollection contained rows with one schema but it 
>> actually had a different schema. The message should definitely be better in 
>> this case though.
>>
>> Brian
>>
>> On Wed, Feb 5, 2020 at 9:28 PM Tomo Suzuki  wrote:
>>>
>>> (My understanding)
>>> The test ensures the CSV data stored in GCS should be readable through 
>>> Datacatalog. It fails because an Integer value in the CSV was read as Long 
>>> as per Datacatalog.
>>>
>>>
>>> > setting up from scratch is a good idea.
>>>
>>> I agree. Furthermore, it would be nice if it can test different type-cast 
>>> behaviors. I’m still feeling it’s ok to read an Integer as Long. (If this 
>>> is the case, how about Long to Integer? What if the long is small enough to 
>>> fit in 32 bits? and so on)
>>>
>>> On Wed, Feb 5, 2020 at 23:15 Kenneth Knowles  wrote:
>>>>
>>>> I think that was me... sorry!
>>>>
>>>> Is this a test where it is important that the data is pre-existing? 
>>>> Otherwise I would say that setting up from scratch is a good idea. Does 
>>>> anyone have context on it? I am happy to take on the small bit of coding, 
>>>> since I broke it.
>>>>
>>>> Kenn
>>>>
>>>> On Wed, Feb 5, 2020 at 1:22 PM Brian Hulette  wrote:
>>>>>
>>>>> So it looks like the schema for `integ_test_small_csv_test_1` was updated 
>>>>> yesterday around the same time that PR#10563 went in, and it no longer 
>>>>> matches the schema we expect in the test.
>>>>>
>>>>> I'm just going to change it back for now. I am curious who changed it and 
>>>>> why, if the perpetrator is on this list please let us know :)
>>>>>
>>>>>
>>>>> Note the updateTime:
>>>>> ```
>>>>> ❯ gcloud beta data-catalog entries lookup 
>>>>> '`datacatalog`.`entry`.`apache-beam-testing`.`us-central1`.`samples`.`integ_test_small_csv_test_1`'``
>>>>> gcsFilesetSpec:
>>>>>   filePatterns:
>>>>>   - gs://apache-beam-samples/integration_test_small_csv/test.csv
>>>>> linkedResource: 
>>>>> //datacatalog.googleapis.com/projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1
>>>>> name: 
>>>>> projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1
>>>>> schema:
>>>>>   columns:
>>>>>   - column: id
>>>>> mode: NULLABLE
>>>>> type: INT64
>>>>>   - column: name
>>>>> mode: NULLABLE
>>>>> type: STRING
>>>>>   - column: type
>>>>> mode: NULLABLE
>>>>> type: STRING
>>>>> sourceSystemTimestamps:
>>>>>   createTime: '2019-08-16T01:49:06.235Z'
>>>>>   updateTime: '2020-02-04T17:18:17.671Z'
>>>>> type: FILESET
>>>>> ```
>>>
>>> --
>>> Regards,
>>> Tomo



-- 
Regards,
Tomo

Re: SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

[Tomo Suzuki]

(My understanding)
The test ensures the CSV data stored in GCS should be readable through
Datacatalog. It fails because an Integer value in the CSV was read as Long
as per Datacatalog.


> setting up from scratch is a good idea.

I agree. Furthermore, it would be nice if it can test different type-cast
behaviors. I’m still feeling it’s ok to read an Integer as Long. (If this
is the case, how about Long to Integer? What if the long is small enough to
fit in 32 bits? and so on)

On Wed, Feb 5, 2020 at 23:15 Kenneth Knowles  wrote:

> I think that was me... sorry!
>
> Is this a test where it is important that the data is pre-existing?
> Otherwise I would say that setting up from scratch is a good idea. Does
> anyone have context on it? I am happy to take on the small bit of coding,
> since I broke it.
>
> Kenn
>
> On Wed, Feb 5, 2020 at 1:22 PM Brian Hulette  wrote:
>
>> So it looks like the schema for `integ_test_small_csv_test_1` was updated
>> yesterday around the same time that PR#10563 went in, and it no longer
>> matches the schema we expect in the test.
>>
>> I'm just going to change it back for now. I am curious who changed it and
>> why, if the perpetrator is on this list please let us know :)
>>
>>
>> Note the updateTime:
>> ```
>> ❯ gcloud beta data-catalog entries lookup
>> '`datacatalog`.`entry`.`apache-beam-testing`.`us-central1`.`samples`.`integ_test_small_csv_test_1`'``
>> gcsFilesetSpec:
>>   filePatterns:
>>   - gs://apache-beam-samples/integration_test_small_csv/test.csv
>> linkedResource: //
>> datacatalog.googleapis.com/projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1
>> name:
>> projects/apache-beam-testing/locations/us-central1/entryGroups/samples/entries/integ_test_small_csv_test_1
>> schema:
>>   columns:
>>   - column: id
>> mode: NULLABLE
>> type: INT64
>>   - column: name
>> mode: NULLABLE
>> type: STRING
>>   - column: type
>> mode: NULLABLE
>> type: STRING
>> sourceSystemTimestamps:
>>   createTime: '2019-08-16T01:49:06.235Z'
>>   updateTime: '2020-02-04T17:18:17.671Z'
>> type: FILESET
>> ```
>>
> --
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Committers,

Would you run the precommit checks for https://github.com/apache/beam/pull/10769
with the following 6 additional commands (one command per comment) ?

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

-- 
Regards,
Tomo

Re: SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

[Tomo Suzuki]

Brian,

Thank you! (I don't need access as long as it's resolved)


On Wed, Feb 5, 2020 at 4:05 PM Brian Hulette  wrote:
>
> I can take a look at this.
>
> I think the access issue (and your problem running locally) is because you 
> need access to apache-beam-testing. I'm not sure if we have any formal 
> process for that.
>
> Brian
>
> On Wed, Feb 5, 2020 at 5:31 AM Tomo Suzuki  wrote:
>>
>> HI Beam developers,
>>
>> Can somebody help this SQL PostCommit integration test failure?
>> https://issues.apache.org/jira/browse/BEAM-9253
>> (since https://github.com/apache/beam/pull/10563)
>>
>> SQL PostCommit failure: ClassCastException: java.lang.Integer cannot
>> be cast to java.lang.Long
>>
>> (Why not?)
>>
>> The difficulty is that the integration test requires access to Google
>> Cloud Storage objects and thus Alexey (PR#10563 author) or I cannot
>> test locally.
>>
>> --
>> Regards,
>> Tomo



-- 
Regards,
Tomo

SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

[Tomo Suzuki]

HI Beam developers,

Can somebody help this SQL PostCommit integration test failure?
https://issues.apache.org/jira/browse/BEAM-9253
(since https://github.com/apache/beam/pull/10563)

SQL PostCommit failure: ClassCastException: java.lang.Integer cannot
be cast to java.lang.Long

(Why not?)

The difficulty is that the integration test requires access to Google
Cloud Storage objects and thus Alexey (PR#10563 author) or I cannot
test locally.

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Thank you, Ahmet!

On Tue, Feb 4, 2020 at 12:36 PM Tomo Suzuki  wrote:
>
> Hi Beam Committers,
>
> Would you run the precommit checks for 
> https://github.com/apache/beam/pull/10765
> with following 6 additional commands (one command per comment) ?:
>
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
> Regards,
> Tomo



-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Committers,

Would you run the precommit checks for https://github.com/apache/beam/pull/10765
with following 6 additional commands (one command per comment) ?:

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

I appreciate if you can trigger precommit checks for
https://github.com/apache/beam/pull/10756 .

Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

HI Beam committers,

Would you re-trigger the 2 failed checks in
https://github.com/apache/beam/pull/10714 ?
Run Java PreCommit
Run Java_Examples_Dataflow PreCommit


On Fri, Jan 31, 2020 at 7:51 AM Rehman Murad Ali <
rehman.murad...@venturedive.com> wrote:

> Hi,
>
> I appreciate if someone could trigger the jobs for this PR:
>
> https://github.com/apache/beam/pull/10627
>
> *Thanks*
>
> *Rehman Murad Ali*
> Software Engineer
> Mobile: +92 3452076766 <+92%20345%202076766>
> Skype: rehman.muradali
>
>
> On Thu, Jan 30, 2020 at 10:19 PM Luke Cwik  wrote:
>
>> done
>>
>> On Thu, Jan 30, 2020 at 12:28 AM Shoaib Zafar <
>> shoaib.za...@venturedive.com> wrote:
>>
>>> Hi Beam Committer,
>>>
>>> I appreciate if someone could trigger jobs for
>>> https://github.com/apache/beam/pull/10712.
>>>
>>> Thanks!
>>>
>>> *Shoaib Zafar*
>>>
>>> Software Engineering Lead
>>> Mobile: +92 333 274 6242
>>> Skype: live:shoaibzafar_1
>>>
>>> <http://venturedive.com/>
>>>
>>>
>>> On Thu, Jan 30, 2020 at 9:09 AM Boyuan Zhang  wrote:
>>>
>>>> Done : )
>>>>
>>>> On Wed, Jan 29, 2020 at 7:52 PM Tomo Suzuki  wrote:
>>>>
>>>>> HI Beam committers:
>>>>> (Thanks, Luke!)
>>>>>
>>>>> Can somebody retrigger the following 2 failed checks for
>>>>> https://github.com/apache/beam/pull/10714 ?
>>>>> Run Java PreCommit
>>>>> Run Java_Examples_Dataflow PreCommit
>>>>>
>>>>> On Wed, Jan 29, 2020 at 4:48 PM Luke Cwik  wrote:
>>>>> >
>>>>> > done
>>>>> >
>>>>> > On Wed, Jan 29, 2020 at 11:07 AM Tomo Suzuki 
>>>>> wrote:
>>>>> >>
>>>>> >> Hi Beam committers,
>>>>> >>
>>>>> >> I appreciate if you can trigger the precommit checks for
>>>>> >> https://github.com/apache/beam/pull/10714
>>>>> >>
>>>>> >> with following 6 additional commands (one command per comment):
>>>>> >>
>>>>> >> Run Java PostCommit
>>>>> >> Run Java HadoopFormatIO Performance Test
>>>>> >> Run BigQueryIO Streaming Performance Test Java
>>>>> >> Run Dataflow ValidatesRunner
>>>>> >> Run Spark ValidatesRunner
>>>>> >> Run SQL Postcommit
>>>>> >>
>>>>> >> Regards,
>>>>> >> Tomo
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Tomo
>>>>>
>>>>

-- 
Regards,
Tomo

Re: flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job

[Tomo Suzuki]

91720f2cd/netty-buffer-4.1.42.Final.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.14/3cb1181b2141a7e752f5bdc998b7ef1849f726cf/commons-codec-1.14.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.9/122c7cee69b53ed4a7681c03d4ee4c0e2765da5/commons-lang3-3.9.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/org.conscrypt/conscrypt-openjdk-uber/2.2.1/59a346d64c0ddca750c5c877e274d5e6278e53ce/conscrypt-openjdk-uber-2.2.1.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/com.squareup.okio/okio/1.13.0/a9283170b7305c8d92d25aff02a6ab7e45d06cbe/okio-1.13.0.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/com.squareup.okhttp/okhttp/2.5.0/4de2b4ed3445c37ec1720a7d214712e845a24636/okhttp-2.5.0.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.42.Final/ccaacf418a9e486b65e82c47bed66439119c5fdb/netty-resolver-4.1.42.Final.jar","/home/jenkins/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.42.Final/e02700b574d3a0e2100308f971f0753ac8700e7c/netty-common-4.1.42.Final.jar"],"gcpTempLocation":"gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0128181056-28c844f3/output/results","gcsPerformanceMetrics":false,"gcsUploadBufferSizeBytes":1048576,"inputFile":"gs://apache-beam-samples/shakespeare/winterstale-personae","jobName":"testpipeline-jenkins-0128181056-28c844f3","maxNumWorkers":0,"numWorkers":1,"numberOfWorkerHarnessThreads":0,"optionsId":0,"output":"gs://temp-storage-for-end-to-end-tests/WordCountIT-2020-01-28-18-10-56-846/output/results","pathValidatorClass":"org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator","pipelineUrl":"gs://temp-storage-for-end-to-end-tests//testpipeline-jenkins-0128181056-28c844f3/output/results/staging/pipeline--O-4jZ6_m4-7kbkqc1gcAw.pb","project":"apache-beam-testing","runner":"org.apache.beam.runners.dataflow.TestDataflowRunner","stableUniqueNames":"ERROR","stagerClass":"org.apache.beam.runners.dataflow.util.GcsStager","stagingLocation":"gs://temp-storage-for-end-to-end-tests//testpipeline-jenkins-0128181056-28c844f3/output/results/staging/","streaming":true,"tempLocation":"gs://temp-storage-for-end-to-end-tests//testpipeline-jenkins-0128181056-28c844f3/output/results","tempRoot":"gs://temp-storage-for-end-to-end-tests/","userAgent":"Apache_Beam_SDK_for_Java/2.20.0-SNAPSHOT(JRE_8_environment)","workerHarnessContainerImage":""}}
> I 2020-01-28T18:12:07.231Z Not using conscrypt SSL. Note this is the default 
> Java behavior, but may have reduced performance. To use conscrypt SSL pass 
> pipeline option --experiments=enable_conscrypt_security_provider
> I 2020-01-28T18:12:08.112Z Logging initialized @4180ms
> W 2020-01-28T18:12:08.186Z Region will default to us-central1. Future 
> releases of Beam will require the user to set the region explicitly. 
> https://cloud.google.com/compute/docs/regions-zones/regions-zones
> I 2020-01-28T18:12:08.250Z jetty-9.2.z-SNAPSHOT
> I 2020-01-28T18:12:08.345Z Started 
> ServerConnector@52960f75{HTTP/1.1}{0.0.0.0:8081}
> I 2020-01-28T18:12:08.346Z Started @4414ms
> I 2020-01-28T18:12:08.346Z Status server started on http://10.128.0.32:8081/
> I 2020-01-28T18:12:08.347Z Sending request to get global configuration for 
> this worker
> I 2020-01-28T18:12:09.660Z Setting maxWorkItemCommitBytes to 188743680
> I 2020-01-28T18:12:09.672Z Creating a new windmill stub, endpoints: 
> [us-central1-dataflowstreaming-pa.googleapis.com:443]
> I 2020-01-28T18:12:09.672Z Previous windmill stub endpoints: []
> I 2020-01-28T18:12:09.673Z Initializing Streaming Engine GRPC client for 
> endpoints: [us-central1-dataflowstreaming-pa.googleapis.com:443]
> I 2020-01-28T18:12:10.090Z windmillServerStub is now ready
> I 2020-01-28T18:12:10.101Z Dispatch starting
> I 2020-01-28T18:12:25.114Z Memory is used/total/max = 189/312/5052 MB, GC 
> last/max = 1.45/1.45 %, #pushbacks=0, gc thrashing=false
> I 2020-01-28T18:17:25.118Z Memory is used/total/max = 134/442/5052 MB, GC 
> last/max = 0.00/1.45 %, #pushbacks=0, gc thrashing=false
>
> On Wed, Jan 29, 2020 at 8:18 PM Tomo Suzuki  wrote:
>>
>> Hi Beam developers,
>>
>> I filed a ticket BEAM-9224 "flaky WordCountIT.testE2EWordCount in
>> Jenkins Java_Examples_Dataflow Job". The trend [1] shows the flakiness
>> started around Jan 28 13:00 ET (Build #6538 [2]).
>>
>>  Example job failure is in [3]. This Jenkins log does not provide why it 
>> failed:
>> java.lang.RuntimeException: Dataflow job
>> 2020-01-28_10_11_07-12281678733591188188 terminated in state
>> UNRECOGNIZED but did not return a failure reason.
>>
>> I appreciate if someone with entitlement can check the Stackdriver
>> logs for the failed E2E tests.
>>
>> [1]: 
>> https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/
>> [2]: 
>> https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6538/
>> [3]: 
>> https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6554/testReport/junit/org.apache.beam.examples/WordCountIT/
>>
>> --
>> Regards,
>> Tomo



-- 
Regards,
Tomo

flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job

[Tomo Suzuki]

Hi Beam developers,

I filed a ticket BEAM-9224 "flaky WordCountIT.testE2EWordCount in
Jenkins Java_Examples_Dataflow Job". The trend [1] shows the flakiness
started around Jan 28 13:00 ET (Build #6538 [2]).

 Example job failure is in [3]. This Jenkins log does not provide why it failed:
java.lang.RuntimeException: Dataflow job
2020-01-28_10_11_07-12281678733591188188 terminated in state
UNRECOGNIZED but did not return a failure reason.

I appreciate if someone with entitlement can check the Stackdriver
logs for the failed E2E tests.

[1]: https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/
[2]: 
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6538/
[3]: 
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6554/testReport/junit/org.apache.beam.examples/WordCountIT/

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

HI Beam committers:
(Thanks, Luke!)

Can somebody retrigger the following 2 failed checks for
https://github.com/apache/beam/pull/10714 ?
Run Java PreCommit
Run Java_Examples_Dataflow PreCommit

On Wed, Jan 29, 2020 at 4:48 PM Luke Cwik  wrote:
>
> done
>
> On Wed, Jan 29, 2020 at 11:07 AM Tomo Suzuki  wrote:
>>
>> Hi Beam committers,
>>
>> I appreciate if you can trigger the precommit checks for
>> https://github.com/apache/beam/pull/10714
>>
>> with following 6 additional commands (one command per comment):
>>
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>> Regards,
>> Tomo
>>
>>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

I appreciate if you can trigger the precommit checks for
https://github.com/apache/beam/pull/10714

with following 6 additional commands (one command per comment):

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: anyone working on updating com.google.cloud:google-cloud-spanner?

[Tomo Suzuki]

I'm going to take that in the coming weeks. Currently I'm focusing on
updating the bigtable-client-core version.

On Tue, Jan 28, 2020 at 4:32 PM Luke Cwik  wrote:
>
> +Tomo Suzuki has been working on plenty of dependency updates related to GCP 
> libraries and could give guidance (e.g. [1]). I have reviewed several.
>
> Tomo has updated many of the core libraries so a version bump for the core 
> spanner library should be much easier now and can be validated using the 
> linkage checker[2].
>
> 1: https://github.com/apache/beam/pull/10674
> 2: 
> https://github.com/apache/beam/blob/f9b1572da636910d01ac2245592206801be3bb70/build.gradle#L290
>
> On Tue, Jan 28, 2020 at 12:00 PM Udi Meiri  wrote:
>>
>> It's currently at 1.6.0, which a year old.
>> https://github.com/googleapis/java-spanner/releases?after=1.9.0
>>
>> I would appreciate any help with this.
>>
>> Tracking issues:
>> https://issues.apache.org/jira/browse/BEAM-8758
>> https://issues.apache.org/jira/browse/BEAM-8682
>>


-- 
Regards,
Tomo

Re: PreCommit Java Portability is failing

[Tomo Suzuki]

Hannah,

Thank you for the fix (#10676)!

On Thu, Jan 23, 2020 at 2:55 PM Hannah Jiang  wrote:
>
> I will submit a small patch this afternoon, the original PR to fix this issue 
> needs some time to merge.
>
>
> On Thu, Jan 23, 2020 at 11:33 AM Andrew Pilloud  wrote:
>>
>> This is still hard failing on all Java PRs. Is there anything you can do to 
>> mitigate this (disable the test) until it is fixed?
>>
>> Andrew
>>
>> On Tue, Jan 21, 2020 at 1:47 PM Hannah Jiang  wrote:
>>>
>>> This is caused by PR 10557, I will fix it soon.
>>>
>>> On Tue, Jan 21, 2020 at 1:24 PM Kirill Kozlov  
>>> wrote:

 Hello beam developers!

 Jenkins job page: 
 https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/
 The following tasks fails: 
 `:runners:google-cloud-dataflow-java:buildAndPushDockerContainer`
 with error:
 'command 'docker'' finished with non-zero exit value 1

 Is it possible that this task is failing due to recent INFRA changes?

 -
 Kirill



-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Comitters,

Can somebody trigger 2 failed checks below for
https://github.com/apache/beam/pull/10674 ?
Run Java PreCommit
Run JavaPortabilityApi PreCommit


On Thu, Jan 23, 2020 at 14:32 Tomo Suzuki  wrote:

> Hi Rui,
> (Thank you for quick response)
>
> Would you run one command per one comment? I don't think Jenkins
> recognizes multiple at once.
>
> On Thu, Jan 23, 2020 at 2:25 PM Rui Wang  wrote:
> >
> > Done
> >
> > On Thu, Jan 23, 2020 at 11:20 AM Tomo Suzuki  wrote:
> >>
> >> Hi Beam Committers,
> >>
> >> I appreciate if you can run precommit checks for
> >> https://github.com/apache/beam/pull/10674
> >> plus the following 6 extra commands:
> >>
> >> Run Java PostCommit
> >> Run Java HadoopFormatIO Performance Test
> >> Run BigQueryIO Streaming Performance Test Java
> >> Run Dataflow ValidatesRunner
> >> Run Spark ValidatesRunner
> >> Run SQL Postcommit
> >>
> >> Regards,
> >> Tomo
>
>
>
> --
> Regards,
> Tomo
>

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Rui,
(Thank you for quick response)

Would you run one command per one comment? I don't think Jenkins
recognizes multiple at once.

On Thu, Jan 23, 2020 at 2:25 PM Rui Wang  wrote:
>
> Done
>
> On Thu, Jan 23, 2020 at 11:20 AM Tomo Suzuki  wrote:
>>
>> Hi Beam Committers,
>>
>> I appreciate if you can run precommit checks for
>> https://github.com/apache/beam/pull/10674
>> plus the following 6 extra commands:
>>
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>> Regards,
>> Tomo



-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Committers,

I appreciate if you can run precommit checks for
https://github.com/apache/beam/pull/10674
plus the following 6 extra commands:

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: help with this error, please

[Tomo Suzuki]

Hi Vasu,
(Ignore my message if Luke's advice resolves the issue already)

Would you add the entire error message, if any? A NoClassDefFoundError
is usually caused by a ClassNotFoundError saying a class is not found.
I don't see the missing class name in your stacktrace. I'll need (1)
the entire error message and (2) your branch in GitHub and a command
to reproduce the error.

Regards,
Tomo


On Wed, Jan 22, 2020 at 7:53 PM Luke Cwik  wrote:
>
> boolean properties only allow for getYYY, isYYY and setYYY, you can't use 
> "should". I think you should have gotten a better error message though so 
> it's likely something else is not working for you. How are you trying to run 
> the test?
>
> All pipeline options use a global namespace so UseGrpc will "reserve" that 
> name from it being used anywhere else. Do you want to define a new property 
> UseGrpc for all GCP IOs or use a name that is specific to GCS?
>
>
> On Wed, Jan 22, 2020 at 3:59 PM Vasu Nori  wrote:
>>
>> sorry I didn't realize I posted a screenshot link that wasn't visible 
>> outside google.com.
>> here is the code I was trying to add to GcsOptions.java
>>
>>   @Description("Whether to use gRPC or not, as transport mechanism.")
>>   @Default.Boolean(false)
>>   Boolean shouldUseGrpc();
>>
>>   void setUseGrpc(Boolean useGrpc);
>>
>>
>> On Wed, Jan 22, 2020 at 2:37 PM Vasu Nori  wrote:
>>>
>>> Hello
>>>
>>> I am trying to add a new property to this file like this
>>> but this results in some error I can't understand the origins of.
>>> Stacktrace is below.
>>> any pointers would be appreciated.
>>>
>>> java.lang.NoClassDefFoundError: Could not initialize class 
>>> org.apache.beam.sdk.options.PipelineOptionsFactory
>>> at 
>>> org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemTest.setUp(GcsFileSystemTest.java:65)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at 
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at 
>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>>> at 
>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>>> at 
>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>>> at 
>>> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
>>> at 
>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>>> at 
>>> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
>>> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>>> at 
>>> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
>>> at 
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>>> at 
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>>> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
>>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
>>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
>>> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
>>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
>>> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>>> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
>>> at 
>>> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>>> at 
>>> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>>> at 
>>> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>>> at 
>>> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>>> at 
>>> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at 
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at 
>>> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>>> at 
>>> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>>> at 
>>> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>>> at 
>>> 

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Thank you, Ahmet.


On Fri, Jan 17, 2020 at 7:22 PM Tomo Suzuki  wrote:
>
> Hi Beam committer,
>
> I appreciate if somebody can trigger the following checks for 
> https://github.com/apache/beam/pull/10631
>
> Run JavaPortabilityApi PreCommit
> Run Java PostCommit
> Run Java HadoopFormatIO Performance Test
> Run BigQueryIO Streaming Performance Test Java
> Run Dataflow ValidatesRunner
> Run Spark ValidatesRunner
> Run SQL Postcommit
>
> Regards,
> Tomo



--
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committer,

I appreciate if somebody can trigger the following checks for
https://github.com/apache/beam/pull/10631

Run JavaPortabilityApi PreCommit
Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Beam's Avro 1.8.x dependency

[Tomo Suzuki]

Ismaël and I think the short-term solution for Aaron's issue is to
define Beam's own class for Avro's TimestampConversion. Luckily it's
the only blocker for Avro 1.9 I found.
I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
PR https://github.com/apache/beam/pull/10628 .

Regards,
Tomo

On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
 wrote:
>
> On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles  wrote:
> >
> > Regarding Google's advice about shading: don't go to a "one version rule" 
> > monorepo for advice about solving diamond dependencies in the wild.
>
> FWIW these docs came out of the open source, Github, multirepo, many
> versions part of Google Cloud, not the mono repo world of google3.
> Though I will say the more time I spend trying to deal with these
> issues the more I love the monorepo.
>
> > It is a useful description of the pitfalls. We (and Flink before us, and 
> > likely many more) are already doing something that avoids many of them or 
> > makes them less likely. Building a separate vendored library is simpler and 
> > more robust than the "shade during build".
> >
>
> I've only encountered "vendoring" as distinct from shading in Beam.
> Are there more details about this anywhere? A quick Google search only
> turned up one Beam doc that pointed to Go docs and a cloud endpoints
> doc that seemed to use it as a synonym for shading.
>
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org



-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Committers,
(Ismaël, thanks!)

I appreciate if somebody can re-trigger 2 failed precommit checks on
https://github.com/apache/beam/pull/10614 :
Run JavaPortabilityApi PreCommit
Run Python PreCommit


On Fri, Jan 17, 2020 at 7:51 AM Shoaib Zafar 
wrote:

> Hi Beam Committer,
>
> I appreciate if someone could trigger jobs for
> https://github.com/apache/beam/pull/9606
>
> Thanks!
>
> *Shoaib Zafar*
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
>
> <http://venturedive.com/>
>
>
> On Fri, Jan 17, 2020 at 1:35 PM Ismaël Mejía  wrote:
>
>> done
>>
>> On Fri, Jan 17, 2020 at 5:51 AM Tomo Suzuki  wrote:
>>
>>> Hi Beam Committers,
>>> (Andrew, thanks! but I needed to fix tests)
>>>
>>> I appreciate if somebody can re-trigger precommit checks for
>>> https://github.com/apache/beam/pull/10614 with the following
>>> additional checks:
>>>
>>> Run Java PostCommit
>>> Run Java HadoopFormatIO Performance Test
>>> Run BigQueryIO Streaming Performance Test Java
>>> Run Dataflow ValidatesRunner
>>> Run Spark ValidatesRunner
>>> Run SQL Postcommit
>>>
>>> On Thu, Jan 16, 2020 at 4:11 PM Andrew Pilloud 
>>> wrote:
>>> >
>>> > done.
>>> >
>>> > On Thu, Jan 16, 2020 at 1:07 PM Tomo Suzuki 
>>> wrote:
>>> >>
>>> >> Hi Beam committers,
>>> >>
>>> >> I appreciate if somebody can trigger precommit checks for
>>> https://github.com/apache/beam/pull/10614 with the following additional
>>> checks:
>>> >>
>>> >> Run Java PostCommit
>>> >> Run Java HadoopFormatIO Performance Test
>>> >> Run BigQueryIO Streaming Performance Test Java
>>> >> Run Dataflow ValidatesRunner
>>> >> Run Spark ValidatesRunner
>>> >> Run SQL Postcommit
>>> >>
>>> >> Regards,
>>> >> Tomo
>>> >>
>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>

-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam Committers,
(Andrew, thanks! but I needed to fix tests)

I appreciate if somebody can re-trigger precommit checks for
https://github.com/apache/beam/pull/10614 with the following
additional checks:

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

On Thu, Jan 16, 2020 at 4:11 PM Andrew Pilloud  wrote:
>
> done.
>
> On Thu, Jan 16, 2020 at 1:07 PM Tomo Suzuki  wrote:
>>
>> Hi Beam committers,
>>
>> I appreciate if somebody can trigger precommit checks for 
>> https://github.com/apache/beam/pull/10614 with the following additional 
>> checks:
>>
>> Run Java PostCommit
>> Run Java HadoopFormatIO Performance Test
>> Run BigQueryIO Streaming Performance Test Java
>> Run Dataflow ValidatesRunner
>> Run Spark ValidatesRunner
>> Run SQL Postcommit
>>
>> Regards,
>> Tomo
>>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

I appreciate if somebody can trigger precommit checks for
https://github.com/apache/beam/pull/10614 with the following additional
checks:

Run Java PostCommit
Run Java HadoopFormatIO Performance Test
Run BigQueryIO Streaming Performance Test Java
Run Dataflow ValidatesRunner
Run Spark ValidatesRunner
Run SQL Postcommit

Regards,
Tomo

Re: Beam's Avro 1.8.x dependency

[Tomo Suzuki]

I've been upgrading dependencies around gRPC. This Avro-problem is
interesting to me.
I'll study BEAM-8388 more tomorrow.

On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik  wrote:
>
> +Tomo Suzuki +jincheng sun
> There have been a few contributors upgrading the dependencies and validating 
> things not breaking by running the majority of the post commit integration 
> tests and also using the linkage checker to show that we aren't worse off 
> with respect to our dependency tree. Reaching out to them to help your is 
> your best bet of getting these upgrades through.
>
> On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon  wrote:
>>
>> I meant to mention that we must use Avro 1.9.x as we rely on some schema 
>> resolution fixes not present in 1.8.x - so am indeed blocked.
>>
>> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon  wrote:
>>>
>>> It looks like Avro version dependency from Beam has come up in the past [1, 
>>> 2].
>>>
>>> I'm currently on Beam 2.16.0, which has been compatible with my usage of 
>>> Avro 1.9.x.
>>>
>>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has 
>>> some dependencies on Avro classes only available in 1.8.x.
>>>
>>> Wondering if anyone else is similar blocked and what it would take to 
>>> prioritize Beam upgrading to 1.9.x or better using a shaded version so that 
>>> clients can use their own Avro version for their own coding purposes. (Eg, 
>>> I parse Avro messages from a KafkaIO source and need 1.9.x for this but am 
>>> perfectly happy if Beam's Avro coding facilities used a shaded other 
>>> version.)
>>>
>>> I've made a comment on BEAM-8388 [1] to this effect. But polling community 
>>> for discussion.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-8388
>>> [2] https://github.com/apache/beam/pull/9779
>>>


-- 
Regards,
Tomo

Re: Jenkins jobs not running for my PR 10438

[Tomo Suzuki]

Hi Beam committers,

Can somebody trigger the precommit cheeks for my new PR
https://github.com/apache/beam/pull/10603 ?

This PR still does not trigger the checks. I confirmed that my account
is in the .adf.yaml.

On Tue, Jan 14, 2020 at 9:48 PM Ahmet Altay  wrote:
>
> Done.
>
> +Kenneth Knowles, any updates from INFRA on this?
>
> On Tue, Jan 14, 2020 at 6:43 PM Tomo Suzuki  wrote:
>>
>> It hit Dataflow quota error again. Can somebody run
>> Run Dataflow ValidatesRunner
>> for https://github.com/apache/beam/pull/10554 ?
>>
>> On Tue, Jan 14, 2020 at 12:14 PM Tomo Suzuki  wrote:
>> >
>> > Valentyn, thank you.
>> >
>> > On Tue, Jan 14, 2020 at 12:05 PM Valentyn Tymofieiev
>> >  wrote:
>> > >
>> > > Done. If tests still don't trigger, you could try to make a push to the 
>> > > branch to reset the test status.
>> > >
>> > > On Tue, Jan 14, 2020 at 8:38 AM Tomo Suzuki  wrote:
>> > >>
>> > >> Hi Beam developers,
>> > >>
>> > >> Can somebody run the following to 
>> > >> https://github.com/apache/beam/pull/10554 ?
>> > >> Run Dataflow ValidatesRunner
>> > >> Run Java PreCommit
>> > >>
>> > >> On Mon, Jan 13, 2020 at 2:35 PM Tomo Suzuki  wrote:
>> > >> >
>> > >> > Thank you, Mark and Ismaël.
>> > >> >
>> > >> > On Mon, Jan 13, 2020 at 2:34 PM Mark Liu  wrote:
>> > >> > >
>> > >> > > done
>> > >> > >
>> > >> > > On Mon, Jan 13, 2020 at 8:03 AM Tomo Suzuki  
>> > >> > > wrote:
>> > >> > >>
>> > >> > >> Thanks Yifan (but Java Precommit is still missing).
>> > >> > >> Can somebody run "Run Java PreCommit" on
>> > >> > >> https://github.com/apache/beam/pull/10554?
>> > >> > >>
>> > >> > >>
>> > >> > >> On Mon, Jan 13, 2020 at 2:59 AM Yifan Zou  
>> > >> > >> wrote:
>> > >> > >> >
>> > >> > >> > done.
>> > >> > >> >
>> > >> > >> > On Sun, Jan 12, 2020 at 6:27 PM Tomo Suzuki  
>> > >> > >> > wrote:
>> > >> > >> >>
>> > >> > >> >> Hi Beam committers,
>> > >> > >> >>
>> > >> > >> >> Four Jenkins jobs did not report back for this PR
>> > >> > >> >> https://github.com/apache/beam/pull/10554 .
>> > >> > >> >> Can somebody trigger them?
>> > >> > >> >>
>> > >> > >> >> On Fri, Jan 10, 2020 at 4:51 PM Andrew Pilloud 
>> > >> > >> >>  wrote:
>> > >> > >> >> >
>> > >> > >> >> > Done.
>> > >> > >> >> >
>> > >> > >> >> > On Fri, Jan 10, 2020 at 12:59 PM Tomo Suzuki 
>> > >> > >> >> >  wrote:
>> > >> > >> >> >>
>> > >> > >> >> >> Hi Bean developers,
>> > >> > >> >> >>
>> > >> > >> >> >> I appreciate a committer can trigger precommit build for
>> > >> > >> >> >> https://github.com/apache/beam/pull/10554.
>> > >> > >> >> >>
>> > >> > >> >> >> In addition to normal precommit checks, I want the 
>> > >> > >> >> >> followings:
>> > >> > >> >> >> Run Java PostCommit
>> > >> > >> >> >> Run Java HadoopFormatIO Performance Test
>> > >> > >> >> >> Run BigQueryIO Streaming Performance Test Java
>> > >> > >> >> >> Run Dataflow ValidatesRunner
>> > >> > >> >> >> Run Spark ValidatesRunner
>> > >> > >> >> >> Run SQL Postcommit
>> > >> > >> >> >>
>> > >> > >> >> >> Regards,
>> > >> > >> >> >> Tomo
>> > >> > >> >>
>> > >> > >> >>
>> > >> > >> >>
>> > >> > >> >> --
>> > >> > >> >> Regards,
>> > >> > >> >> Tomo
>> > >> > >>
>> > >> > >>
>> > >> > >>
>> > >> > >> --
>> > >> > >> Regards,
>> > >> > >> Tomo
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Regards,
>> > >> > Tomo
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Regards,
>> > >> Tomo
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Tomo
>>
>>
>>
>> --
>> Regards,
>> Tomo



-- 
Regards,
Tomo

Re: Quota limitation for Java tests

[Tomo Suzuki]

Hi Beam committers,

I encountered a similar problem today for "Run Dataflow ValidatesRunner":
  Dataflow quota error for jobs-per-project quota. Project
apache-beam-testing is running 303 jobs.
  
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/190/testReport/junit/org.apache.beam.sdk/PipelineTest/testTupleProjectionTransform/
via https://github.com/apache/beam/pull/10554 .

Can somebody with permission check any unexpected long-running jobs?

Regards,
Tomo

On Tue, Dec 10, 2019 at 10:37 AM Łukasz Gajowy  wrote:
>
> Of course, fixing https://issues.apache.org/jira/browse/BEAM-8939 is also 
> crucial to avoid resource exhaustion but I didn't have time to do this. 
> Anyone, feel free to resolve it.
>
> Thanks!
>
> wt., 10 gru 2019 o 16:25 Łukasz Gajowy  napisał(a):
>>
>> https://github.com/apache/beam/pull/10342 - pr that skips the tests listed 
>> above - looking for reviewers
>>
>> Thanks!
>>
>> wt., 10 gru 2019 o 13:30 Łukasz Gajowy  napisał(a):
>>>
>>> What I invoked in the apache-beam-testing project:
>>>
>>> gcloud dataflow jobs list --created-before=-P5H --status=active 
>>> --format="value(JOB_ID)" --region=us-central|xargs gcloud dataflow jobs 
>>> cancel
>>>
>>> wt., 10 gru 2019 o 13:28 Łukasz Gajowy  napisał(a):

 Hi Kirill,

 We (along with Michał and Kamil) noticed the problem as well in Dataflow 
 ValidatesRunner suites yesterday. I started investigating the problem and 
 I noticed that there are jobs running for 5 days and counting. It seems 
 that those are not stopped by "beam_CancelStaleDataflowJobs" job that runs 
 randomly each day. After investigating deeper, it seems that lots of the 
 jobs that are stale are from 
 "https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/; job that 
 is currently being ABORTED due to timeout.

 Some tests (I'm not sure if this is the exhaustive list but they seem to 
 appear in the dataflow console repeatedly) that seem to not be killed and 
 eat our resources:
  - test_reshuffle_preserves_timestamps (spotted multiple times in the 
 dataflow console) (Python SDK)
  - test_flatten_same_pcollections (Python SDK)
  - testPairWithIndexWindowedTimestampedBounded (Java SDK)
  - testPairWithIndexBasicBounded

 I created https://issues.apache.org/jira/browse/BEAM-8938 to track tests 
 like this. Right now I'm going to kill all jobs that hang like this and 
 ignore the tests that I tracked down in a pr for the issue I created.

 I think it's good that job_CancelStaleDataflowJobs didn't catch them - I 
 think that if it did, we would not spot the problem. Is it possible to set 
 up some alerting on Dataflow instead of automatically cleaning the jobs? 
 IMO we should fix the tests rather than cancel them.

 Thanks,
 Łukasz


 wt., 10 gru 2019 o 00:09 Kirill Kozlov  
 napisał(a):
>
> Hello everyone!
>
> It looks like JavaPostCommit Jenkins tests [1] are failing due to CPU 
> quota limitations.
> Could someone please look into this?
>
> [1] 
> https://builds.apache.org/job/beam_PostCommit_Java/4838/testReport/junit/org.apache.beam.examples.complete/TrafficMaxLaneFlowIT/testE2ETrafficMaxLaneFlow/
>
> --
> Kirill



-- 
Regards,
Tomo