FileBasedSource does not match files on HDFS

Hello, I'm testing some pipelines on a dataproc cluster with hadoop version 2.8.2, beam 2.3.0-SNAPSHOT. I have observed on our pipeline as well as the wordcount that ships with beam, that FileBasedSource does not "match" any files when using hdfs prefix - verified this with apex runner and direct

Re: jackson to parse options?

Removing large dependencies is great but I can only assume it will be a lot of work so if you can get it to work in a backwards compatible way great. On Tue, Jan 9, 2018 at 1:52 PM, Romain Manni-Bucau wrote: > It conflicts easily between libs and containers. Shade is not

Re: jackson to parse options?

It conflicts easily between libs and containers. Shade is not a good option too - see the thread on this topic :(. At the end i see using the cli solution closer to user - vs framework dev for json - and hurting less in terms of classpath so pby sthg to test no? Le 9 janv. 2018 22:47, "Lukasz

Re: jackson to parse options?

Romain, how has Jackson been a classpath breaker? On Tue, Jan 9, 2018 at 1:20 PM, Romain Manni-Bucau wrote: > Hmm, beam already owns the cli parsing - that is what I meant - it only > misses the arg delimiter (ie quoting) and adding it is easy no? > > Le 9 janv. 2018

Re: jackson to parse options?

Hmm, beam already owns the cli parsing - that is what I meant - it only misses the arg delimiter (ie quoting) and adding it is easy no? Le 9 janv. 2018 21:19, "Robert Bradshaw" a écrit : > On Tue, Jan 9, 2018 at 11:48 AM, Romain Manni-Bucau > wrote:

Jenkins build is still unstable: beam_Release_NightlySnapshot #647

See

Re: jackson to parse options?

On Tue, Jan 9, 2018 at 11:48 AM, Romain Manni-Bucau wrote: > > Le 9 janv. 2018 19:50, "Robert Bradshaw" a écrit : > > From what I understand: > > 1) The command line argument values (both simple and more complex) are > all JSON-representable. > > And

Re: jackson to parse options?

Le 9 janv. 2018 19:50, "Robert Bradshaw" a écrit : >From what I understand: 1) The command line argument values (both simple and more complex) are all JSON-representable. And must be all CLI representable 2) The command line is a mapping of keys to these values. Which

Re: jackson to parse options?

>From what I understand: 1) The command line argument values (both simple and more complex) are all JSON-representable. 2) The command line is a mapping of keys to these values. As such, it seems quite natural to represent the whole set as a single JSON map, rather than using a different, custom

Re: Dataflow runner examples build fail

Thanks Alan. Luke just updated my account on the platform: you were faster than I ;) So, no need to increase the quota then. I just started a new nightly snapshot on Jenkins to check if it's OK. Thanks ! Regards JB On 09/01/2018 19:10, Alan Myrvold wrote: The google cloud project 

Re: Dataflow runner examples build fail

The google cloud project apache-beam-testing was near quota. There were ~28 old dataflow jobs running, mostly python sdk. robertwb and I cancelled the old jobs and will look why these got stuck. After cancelling the old jobs, the apache-beam-testing project is no longer near quota. On Tue, Jan 9,

Re: jackson to parse options?

Ah, you don't want JSON within JSON. I see, if thats the case just migrate all of them to string tokenization and drop the Jackson usage for string -> string[] conversion. On Mon, Jan 8, 2018 at 10:06 PM, Romain Manni-Bucau wrote: > Lukasz, the use case is to znsure the

Re: [DISCUSS] Towards Beam 2.3.0

Hi, yes, I proposed some weeks ago a best effort to have a release every two months. I'm also volunteer to do those releases and iteratively improve the process. Regards JB On 01/09/2018 06:28 PM, Konstantinos Katsiapis wrote: +1 Our team works on tensorflow.transform

Re: [DISCUSS] Towards Beam 2.3.0

+1 Our team works on tensorflow.transform (which depends on Apache Beam) and a regular Beam release cadence would make our releases a lot easier (eg our 0.5 release will likely require and depend on Beam 2.3). Is the plan to make "regular" something like

Re: Switching to Java 8

Hi, +1 as well, excellent news ! I would add also: remove (AFAIK in some IOs) the enforcer configuration (like [1]) that were put when java 8 was needed in a java 7 build. [1]    [1.8,) Etienne Le 08/01/2018 à 14:02, Jean-Baptiste Onofré a écrit : I created

Re: Dataflow runner examples build fail

+1 2018-01-08 22:38 GMT+01:00 Daniel Oliveira : > +1 > > On Mon, Jan 8, 2018 at 10:07 AM, Kenneth Knowles wrote: > >> +1 >> >> On Mon, Jan 8, 2018 at 9:33 AM, Henning Rohde wrote: >> >>> +1 >>> >>> On Mon, Jan 8, 2018 at 1:32 AM, Ted

Jenkins build is still unstable: beam_Release_NightlySnapshot #646

See