Re: Build blocking on

Thanks Udi for trying that! In fact, the go dependency resolution is flaky. Did not look into that, but just rerunning usually works. Of course, less than optimal, but, well... Running build target is of course just an aggregation of task to run. And unfortunately just running that ./gradlew

Re: Build blocking on

It shouldn't stall. That's a bug. OTOH, I never use the `build` target. I'll try running that myself. On Mon, Mar 25, 2019, 07:24 Michael Luckey wrote: > Hi, > > trying to run './gradlew build' on vanilla setup, my build consistently > stalls during execution of python gcp tests, e.g. on both

Re: Build blocking on

It's concerning to me that 1) the Go dependency resolution via gogradle is flaky, and 2) that it can block other languages. I suppose 2) makes sense since it's part of the container bootstrapping code, but that makes 1) a serious problem, of which I wasn't aware. I should have time to investigate

Re: Beam Dependency Check Report (2019-03-23)

Thanks for fixing this Yifan. Also, a kind reminder to everybody to pay attention to dependency reports and claim any JIRAs for dependencies that you would like to see upgraded. Thanks, Cham On Sun, Mar 24, 2019 at 10:46 PM Yifan Zou wrote: > The job failed because of PyYAML load(file) is

Re: Frequent failures on beam8

the beam8 is disabled by now. On Mon, Mar 25, 2019 at 2:06 PM Mikhail Gryzykhin wrote: > Yifan is looking into this. > > On Mon, Mar 25, 2019 at 1:55 PM Boyuan Zhang wrote: > >> Hey all, >> >> Could anyone help take a look at beam8 >> ? Seems

Re: Writing bytes to BigQuery with beam

On Mon, Mar 25, 2019 at 2:16 PM Pablo Estrada wrote: > +Chamikara Jayalath with the new BigQuery sink, > schema autodetection is supported (it's a very simple thing to have). Do > you think we should not have it? > Best > -P. > Ah good to know. But IMO users should be able to write to existing

Re: Writing bytes to BigQuery with beam

Thanks everyone for input on this thread. I think there is a confusion between not specifying the schema, and asking BigQuery to do schema autodetection. This is not the same thing, however in recent changes to BQ IO that happened after 2.11 release, we are forcing schema autodetection, when

Re: Build blocking on

Okay, `./gradlew build` failed pretty quickly for me: > Task :beam-sdks-go:resolveBuildDependencies FAILED cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[https://code.googlesource.com/gocloud] does not exist in /usr/local/google/home/ehudm/.gradle/go/repo/

Re: Build blocking on

Just for the record, using a vm here, because did not yet get all task running on my mac, and did not want to mess with my setup. So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores and further sudo apt update sudo apt install gcc sudo apt install make sudo apt install

Re: [ANNOUNCE] New committer announcement: Mark Liu

Yeaah Mark! : ) Congrats : D On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou wrote: > Congratulations Mark! > > On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan > wrote: > >> Well done congratulations Mark!!! >> >> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke wrote: >> >>> Congratulations

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark! On Sun, Mar 24, 2019 at 23:18 Pablo Estrada wrote: > Yeaah Mark! : ) Congrats : D > > On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou wrote: > >> Congratulations Mark! >> >> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan >> wrote: >> >>> Well done congratulations

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations! On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy wrote: > Congrats! :) > > > > pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy > napisał(a): > >> Congratulations, Mark! >> >> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada wrote: >> >>> Yeaah Mark! : ) Congrats : D >>> >>> On

Beam Dependency Check Report (2019-03-25)

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congrats! :) pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy napisał(a): > Congratulations, Mark! > > On Sun, Mar 24, 2019 at 23:18 Pablo Estrada wrote: > >> Yeaah Mark! : ) Congrats : D >> >> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou wrote: >> >>> Congratulations Mark! >>> >>> On Sun,

Re: Writing bytes to BigQuery with beam

On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev wrote: > We received feedback on https://issuetracker.google.com/issues/129006689 - > BQ developers say that schema identification is done and they discourage to > use schema autodetection in tables using BYTES. In light of this, I think > may be

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark!  On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov wrote: > Congratulations! > > On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy wrote: > >> Congrats! :) >> >> >> >> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy >> napisał(a): >> >>> Congratulations, Mark! >>> >>> On

Re: Apache Beam, Spark, Hadoop and AWS cross accounts

I don't think I'm fully understanding the input part of your pipeline since it looks like it's using some custom IO, but I am fairly confident that FileIO.write() will never produce empty files, so the behavior you describe sounds expected if you're reading in an empty file. FileIO doesn't copy

Re: [ANNOUNCE] New committer announcement: Mark Liu

Nice! Congratulations, Mark. On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk < ka.kucharc...@gmail.com> wrote: > Congratulations, Mark!  > > On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov wrote: > >> Congratulations! >> >> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy >> wrote: >> >>>

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey wrote: > Nice! Congratulations, Mark. > > On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk < > ka.kucharc...@gmail.com> wrote: > >> Congratulations, Mark!  >> >> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov wrote: >>

Build blocking on

Hi, trying to run './gradlew build' on vanilla setup, my build consistently stalls during execution of python gcp tests, e.g. on both of - > :beam-sdks-python:testPy2Gcp - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp Console output: snip test_big_query_standard_sql

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark!  On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson wrote: > Congratulations Mark! > > > On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey > wrote: > >> Nice! Congratulations, Mark. >> >> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk < >> ka.kucharc...@gmail.com> wrote:

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congrats Mark! On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: > Congratulations, Mark!  > > On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson > wrote: > >> Congratulations Mark! >> >> >> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey >> wrote: >> >>> Nice! Congratulations, Mark. >>> >>> On

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark! > On 25 Mar 2019, at 18:36, Mark Liu wrote: > > Thank you all! It's a great pleasure to work on Beam! > > Mark > > On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu > wrote: > Congratulations, Mark! > > On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri

Re: Writing bytes to BigQuery with beam

On Mon, Mar 25, 2019 at 2:03 AM Juta Staes wrote: > > On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev > wrote: > >> We received feedback on https://issuetracker.google.com/issues/129006689 - >> BQ developers say that schema identification is done and they discourage to >> use schema

Re: [ANNOUNCE] New committer announcement: Mark Liu

congratulations, Mark!!! On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang wrote: > Congratulations Mark! > > On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri wrote: > >> Congrats Mark! >> >> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: >> >>> Congratulations, Mark!  >>> >>> On Mon, Mar 25, 2019

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri wrote: > Congrats Mark! > > On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: > >> Congratulations, Mark!  >> >> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson >> wrote: >> >>> Congratulations Mark! >>> >>> >>> On Mon, Mar 25,

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congrats Mark! On Mon, Mar 25, 2019 at 10:50 AM Alexey Romanenko wrote: > Congratulations, Mark! > > On 25 Mar 2019, at 18:36, Mark Liu wrote: > > Thank you all! It's a great pleasure to work on Beam! > > Mark > > On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu wrote: > >> Congratulations, Mark!

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark! On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri wrote: > Congrats Mark! > > On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: > >> Congratulations, Mark!  >> >> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson >> wrote: >> >>> Congratulations Mark! >>> >>> >>> On Mon, Mar 25,

Re: docs: java-dependencies

I was looking at this recently to try and figure out how to generate this in a reliable way for Java, but hadn't come up with a clean solution yet. For the related Python dependencies page [1], I recently wrote a script that scrapes the dependencies and generates an HTML table that can be copied

Re: [ANNOUNCE] New committer announcement: Mark Liu

Thank you all! It's a great pleasure to work on Beam! Mark On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu wrote: > Congratulations, Mark! > > On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri wrote: > >> Congrats Mark! >> >> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: >> >>> Congratulations,

Re: [ANNOUNCE] New committer announcement: Mark Liu

Wonderful, congrats Mark! On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold wrote: > congratulations, Mark!!! > > On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang wrote: > >> Congratulations Mark! >> >> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri wrote: >> >>> Congrats Mark! >>> >>> On Mon, Mar 25,

Frequent failures on beam8

Hey all, Could anyone help take a look at beam8 ? Seems like there are many tests failed on beam8 owing to infra problems. Thanks!

Re: Frequent failures on beam8

Yifan is looking into this. On Mon, Mar 25, 2019 at 1:55 PM Boyuan Zhang wrote: > Hey all, > > Could anyone help take a look at beam8 > ? Seems like there are > many tests failed on beam8 owing to infra problems. > > Thanks! >

Re: Writing bytes to BigQuery with beam

+Chamikara Jayalath with the new BigQuery sink, schema autodetection is supported (it's a very simple thing to have). Do you think we should not have it? Best -P. On Mon, Mar 25, 2019 at 11:01 AM Chamikara Jayalath wrote: > > > On Mon, Mar 25, 2019 at 2:03 AM Juta Staes wrote: > >> >> On Mon,

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster wrote: > Wonderful, congrats Mark! > > On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold wrote: > >> congratulations, Mark!!! >> >> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang wrote: >> >>> Congratulations Mark! >>> >>> On Mon,

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congrats Mark! On Mon, Mar 25, 2019 at 12:14 PM Ankur Goenka wrote: > Congratulations Mark! > > On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster > wrote: > >> Wonderful, congrats Mark! >> >> On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold >> wrote: >> >>> congratulations, Mark!!! >>> >>> On Mon,