Re: Simplifying beam pipeline construction

Thanks for commenting. Yes, I think it's worth adding this API regardless (pending no strong objection) but removing the old one (which I'd like to do) would he a higher bar for sure (though still probably achievable). On Mon, Sep 18, 2017 at 9:12 PM, Kenneth Knowles wrote: > I commented on the

Re: Simplifying beam pipeline construction

I commented on the doc. Overall, I'm positively disposed, but wary of the magnitude of the change. Lots of strong points and good ideas for clarifying things for users. Can we do some of it ASAP regardless? On the front end, we could revamp our examples to use the proposed design pattern. On the b

Simplifying beam pipeline construction

In the effort to simplify and clean up the Beam API, especially with an eye towards making Beam more friendly towards interactive use, I propose getting rid of the Pipline object. See the full proposal at https://s.apache.org/no-beam-pipeline . I'd like to hear people's thoughts on the idea. - Rob

Re: How to write NaN using BigQuerySink in Python?

Thanks for the quick response, Cham. In my use case (supporting the VCF format), each value in the repeated sequence has an associated context. In other words, the index of the values is important for determining its context and some values may be

Re: How to write NaN using BigQuerySink in Python?

NaN and Inf values are not JSON compliant and hence not supported. We use JSON BigQuery load when writing to BigQuery using DataflowRunner. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L155 Other values including 'None' are supported. Why do you need to re

How to write NaN using BigQuerySink in Python?

Is there a way to write 'NaN' to BigQuery using the Python beam.io.BigQuerySink? It complains that NaN is not supported in JSON if I try using float('NaN'). Context: given that null values are not supported in repeated fields for BigQuery (e.g. having [0, None, 1]), I like to find a way to repres

Re: flaky testSplit in beam-sdks-java-io-elasticsearch-tests-5

Thanks ! I'm on it. Regards JB On 09/18/2017 11:30 AM, Etienne Chauchot wrote: Hi, FYI: here is the fix for ElasticsearchIOTest.testSplit flaky test in artifact beam-sdks-java-io-elasticsearch-tests-5. https://github.com/apache/beam/pull/3860 Best Etienne Le 15/09/2017 à 15:56, Etienne

Re: flaky testSplit in beam-sdks-java-io-elasticsearch-tests-5

Hi, FYI: here is the fix for ElasticsearchIOTest.testSplit flaky test in artifact beam-sdks-java-io-elasticsearch-tests-5. https://github.com/apache/beam/pull/3860 Best Etienne Le 15/09/2017 à 15:56, Etienne Chauchot a écrit : Yes, sure, please ignore this test, I'll fix it soon. Best,

Build failed in Jenkins: beam_Release_NightlySnapshot #536

See -- [...truncated 3.90 MB...] File " line 40, in from apit