Re: Jenkins jobs not running for my PR 10438

Done. On Thu, Mar 26, 2020 at 2:37 PM Tomo Suzuki wrote: > Hi Beam committers, > (Thanks Ahmet for previous attempt) > > Would you trigger > - The failed "Run Java Precommit" for > https://github.com/apache/beam/pull/11208, > - and the precommit checks for

Re: [VOTE + INPUT] Beam Mascot Designs, 2nd iteration - Deadline Friday, March 27

Agree with Max. When I first saw the new image on slide 5, I could not perceive the end of the tail. So to me it looked less like a sting and more like it got brighter and brighter so that the shape of the end of the tail could not be perceived through the glow. On Thu, Mar 26, 2020 at 1:57 PM

[DISCUSS] [PROPOSAL] Redesigning the structure for Beam docs

Hi there, As part of the documentation revamping process [1] and ongoing website migration [2] [3], Aizhamal, David, and I would like to propose this website redesign [4]. It lists the rationale and

Re: Jenkins jobs not running for my PR 10438

Hi Beam committers, (Thanks Ahmet for previous attempt) Would you trigger - The failed "Run Java Precommit" for https://github.com/apache/beam/pull/11208, - and the precommit checks for https://github.com/apache/beam/pull/11156 with the following 6 checks? Run Java PostCommit Run Java

Re: ZetaSQL to Calcite translation layer

It doesn't, I want to use the translation layer outside of beam. A good chunk of the code in the library is beam agnostic, but it also contains a lot of beam dependencies that I don't want to pull in. I think if ZetaSQLPlannerImpl and its dependency graph were separate, that'd be all that's

Re: /zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR

Confirmed on 10.15.3. Nice! Kenn On Thu, Mar 26, 2020 at 5:37 AM Gleb Kanterov wrote: > Great work. I've tried these jars, and it works on my end. > > /Gleb > > On Wed, Mar 25, 2020 at 7:58 PM Andrew Pilloud > wrote: > >> This should be fixed in ZetaSQL 2020.03.2, which will be coming out

Re: ZetaSQL to Calcite translation layer

Hi Steve, Could you clarify a bit: could you use [1] directly to solve your case? If not, why? [1]: https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-extensions-sql-zetasql/2.17.0 -Rui On Thu, Mar 26, 2020 at 1:23 PM Steve Niemitz wrote: > Oh I think I actually remember

Re: Are docker image tags shared within a jenkins worker?

My thinking was that anything created in the file system (like the python environment) should already be properly isolated within each job's workspace, but docker image tags are shared. Brian On Thu, Mar 26, 2020 at 1:44 PM Kyle Weaver wrote: > Is it possible the Python environment the Docker

Re: [VOTE + INPUT] Beam Mascot Designs, 2nd iteration - Deadline Friday, March 27

Thanks for the update Daniel! > 1. Do you prefer red or black colored line art? Black. > 2. Do you have any additional feedback about the mascot's shape or features? Great improvement! The tail looks a bit sharp to me. I like the new shape but I'd prefer if it looked less like a sting.

Re: Are docker image tags shared within a jenkins worker?

Is it possible the Python environment the Docker image built from was contaminated to begin with? On Thu, Mar 26, 2020 at 11:06 AM Udi Meiri wrote: > The Python HDFS IT uses the jenkins BUILD_TAG to create unique names: > PROJECT_NAME=$(echo hdfs_IT-${BUILD_TAG:-non-jenkins}) > > The BUILD_TAG

Re: Tests not getting triggered

I see queueing time up to one hour: http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1 So estimate slow pre-commits reaction. On Thu, Mar 26, 2020 at 8:03 PM Ankur Goenka wrote: > Seems to be running now with some delay. > > On Thu, Mar 26, 2020 at 12:33 PM Luke Cwik

Re: ZetaSQL to Calcite translation layer

Oh I think I actually remember seeing that email on the calcite list. :) I agree that it being an alternate parser implementation in calcite itself would be ideal, but also agree (sadly) that that'll probably be a very slow process. Splitting it into its own library in beam seems ideal, the only

Re: ZetaSQL to Calcite translation layer

I think it makes sense for the ZetaSQL to Calcite translation layer to live in Calcite itself, and did suggest it at one point on their dev list (See: https://lists.apache.org/thread.html/38942fcb4775ed71f9b2ab8880ab68a4238166ea5e904111ca184a12%40%3Cdev.calcite.apache.org%3E). I don't think there

Re: Tests not getting triggered

Seems to be running now with some delay. On Thu, Mar 26, 2020 at 12:33 PM Luke Cwik wrote: > I saw upwards of a 20 min delay before anything was triggered on a couple > of PRs. > > On Thu, Mar 26, 2020 at 12:32 PM Ankur Goenka wrote: > >> Hi, >> >> I think the tests for PRs are not getting

Re: [BEAM-9322] Python SDK discussion on correct output tag names

On Thu, Mar 26, 2020 at 10:13 AM Luke Cwik wrote: > The issue seems to be that a PCollection can have a "tag" associated with > it and PTransform expansion can return an arbitrary nested dictionary/tuple > yet we need to figure out what the user wanted as the local name for the > PCollection

ZetaSQL to Calcite translation layer

The ZetaSQL to calcite translation layer that is bundled with beam seems generally useful in cases other than for beam. In fact, we're using (essentially a fork of) it internally outside of beam right now (and I've fixed a bunch of bugs in it). Has there ever been any thought about splitting

Re: Tests not getting triggered

I saw upwards of a 20 min delay before anything was triggered on a couple of PRs. On Thu, Mar 26, 2020 at 12:32 PM Ankur Goenka wrote: > Hi, > > I think the tests for PRs are not getting triggered example: > https://github.com/apache/beam/pull/10870 > > Can someone take a look. > > Thanks, >

Tests not getting triggered

Hi, I think the tests for PRs are not getting triggered example: https://github.com/apache/beam/pull/10870 Can someone take a look. Thanks, Ankur

Re: [VOTE + INPUT] Beam Mascot Designs, 2nd iteration - Deadline Friday, March 27

1. Do you prefer red or black colored line art? Black On Thu, Mar 26, 2020 at 11:07 AM Pablo Estrada wrote: > 1. I am slightly inclined for black lines > 2. I have two pieces of feedback: > - It feels like the head and the eyes on this design are less elongated. I > think I liked the more

Re: [VOTE + INPUT] Beam Mascot Designs, 2nd iteration - Deadline Friday, March 27

1. I am slightly inclined for black lines 2. I have two pieces of feedback: - It feels like the head and the eyes on this design are less elongated. I think I liked the more oval-like eyes from previous designs. (And the head also became less oval-like?, maybe?) - I like the white hand tips and

Re: Are docker image tags shared within a jenkins worker?

The Python HDFS IT uses the jenkins BUILD_TAG to create unique names: PROJECT_NAME=$(echo hdfs_IT-${BUILD_TAG:-non-jenkins}) The BUILD_TAG is unique and easily traced back to the Jenkins job that made it. It might need some sanitizing though if it contains any invalid characters. On Tue, Mar 24,

Re: [BEAM-9322] Python SDK discussion on correct output tag names

The issue seems to be that a PCollection can have a "tag" associated with it and PTransform expansion can return an arbitrary nested dictionary/tuple yet we need to figure out what the user wanted as the local name for the PCollection from all this information. Will this break people who rely on

Re: [PROPOSAL] Snowflake Java Connector for Apache Beam

Hi, Thank you for your enthusiasm and for so many questions/comments :) I hope to address them all. Alexey, as far as I know, copy methods have better performance than inserts/selects. I think currently in Beam's JDBC loading and unloading is provided by selects and inserts as well. But I saw

Re: Question on SerializableCoder

For the system property approach, the underlying execution engine (Flink/Spark/...) need to use a separate JVM for each job instance and not run multiple jobs within one JVM otherwise the users JVM initializer may step on someone else's JVM initializer logic. The execution engine also needs to

Re: Question on SerializableCoder

Luke, can a similar approach be used in classic runners? What would we need to change to achieve it? On Thu, Mar 26, 2020 at 4:06 PM Luke Cwik wrote: > > From the private@ thread, I suggested: > "With the JvmInitializer[1] being supported by Dataflow and the portable Java > container, users

Re: Question on SerializableCoder

>From the private@ thread, I suggested: "With the JvmInitializer[1] being supported by Dataflow and the portable Java container, users would be able to write code which sets the system property jdk.serialFilter or by configuring ObjectInputFilter.Config.setSerialFilter(filter)[2]" This could

Re: /zetasql/local_service/liblocal_service_jni.jnilib was not found inside JAR

Great work. I've tried these jars, and it works on my end. /Gleb On Wed, Mar 25, 2020 at 7:58 PM Andrew Pilloud wrote: > This should be fixed in ZetaSQL 2020.03.2, which will be coming out soon. > We've verified it on multiple machines with OS X 10.14.6. If you have > another version, I could

Re: Question on SerializableCoder

Thanks for all the feedback. Two possible ideas that occur to me: - Create TypeSafeSerializableCoder or somesuch which extends SerializableCoder and enforces the check as in the PR. Update the documentation to suggest using the new coder if you don't need to support collections as the raw type