Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

reply on the vote thread to cancel the vote. There’s no need to wait 72 hours." On Wed, Feb 1, 2017 at 11:45 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ken, agree. I'm waiting for other -1 to formally cancel the vote. Thanks ! Regards JB On 02/02/2017 04:11 AM, Kenne

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

hours." On Wed, Feb 1, 2017 at 11:45 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ken, agree. I'm waiting for other -1 to formally cancel the vote. Thanks ! Regards JB On 02/02/2017 04:11 AM, Kenneth Knowles wrote: Unless there is a strenuous objection, I think we shou

Re: [DISCUSS] Python SDK status and next steps

Just to be clear, the prerequisite to be able to build the Python SDK are: apt-get install python-setuptools apt-get install python-pip It's also required by the default "regular" build. Regards JB On 01/31/2017 11:02 AM, Jean-Baptiste Onofré wrote: Just one thing I notice

Re: [DISCUSS] Python SDK status and next steps

it'd be better to have it under a separated profile or so? On Tue, Jan 31, 2017 at 11:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Just to be clear, the prerequisite to be able to build the Python SDK are: apt-get install python-setuptools apt-get install python-pip It'

Re: [DISCUSS] Python SDK status and next steps

Awesome, thanks Sergio ! Much appreciated ;) Regards JB On 01/31/2017 01:42 PM, Sergio Fernández wrote: PR #1879 provides the basics: https://github.com/apache/beam/pull/1879 On Tue, Jan 31, 2017 at 1:33 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: No, that's fine as soon

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

also not have proper late data dropping and it's fine to have a release with some missing features. (There's more besides dropping.) I think we should go ahead and fix for 0.6. On Tue, Jan 31, 2017, 18:23 Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Aljoscha, so you propose to

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

On 01/31/2017 07:13 PM, Dan Halperin wrote: Should we revert the CLs that lost the functionality? I'd really not like to ship a release with such a functional regression On Tue, Jan 31, 2017 at 10:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Fair enough. Let's do that.

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

s.apache.org/job/beam_PostCommit_Java_RunnableOnService_Dataflow/2130/ On Mon, Jan 30, 2017 at 4:40 PM, Dan Halperin <dhalp...@apache.org> wrote: I am worried about https://issues.apache.org/jira/browse/BEAM-1346 for RC1 and would at least wait for resolution there before proceeding. On Mon, Jan 30, 2017 at 3:48 AM

Re: Report to the Board, February 2017 edition

Hi It looks good to me. Thanks Davor Regards JB On Feb 6, 2017, 19:32, at 19:32, Davor Bonaci wrote: >We are expected to submit a project report to the ASF Board of >Directors >ahead of its next meeting. The report is due on Wednesday, 2/8. > >This is the second is the series

Re: [CANCEL][VOTE] Apache Beam, version 0.5.0, release candidate #1

It sounds good. Let's do that. Regards JB On 02/02/2017 12:12 PM, Aljoscha Krettek wrote: We should go with the option Kenn proposed instead of a "hard revert". On Thu, 2 Feb 2017 at 10:55 Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi everyone, Due to the regress

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #2

+1 (binding) Regards JB⁣​ On Feb 4, 2017, 05:32, at 05:32, Aljoscha Krettek wrote: >+1 > >- Checked signatures >- Verified that source release builds >- Verified that code contains fix for the regression in the Flink >Runner >- Tried Quickstart against Staging Repository >-

Re: Better developer instructions for using Maven?

the checkstyle. On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov <kirpic...@google.com.invalid> wrote: Try just -Prelease. On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson < je...@smokinghand.com wrote: Fails because I don't have a secret key. On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptis

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

PTransform that implements the batching in its own DoFn and that takes user defined functions for customization. Etienne Le 17/01/2017 à 17:30, Jean-Baptiste Onofré a écrit : Hi I guess you mean discussion on the mailing list about that, right ? AFAIR the ide⁣​a is to provide a utility cl

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

ss of the bundles. For example, possible use case is to batch a call to an external service (for performance). I was thinking about providing a PTransform that implements the batching in its own DoFn and that takes user defined functions for customization. Etienne Le 17/01/2017 à 17:

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

a formality, "PPMC" is not committee that plays a role anymore, you're a PMC now ;-) On Fri, Jan 27, 2017 at 9:55 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve th

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

+1 (binding) Regards JB On 01/27/2017 09:55 PM, Jean-Baptiste Onofré wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete

Re: IO Authoring Guide - first draft

I/edit?usp=sharing -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: IO Integration tests - concrete proposal

>> - write tests: use another storage place than read tests (using >> namespace for example) and clean it after each test. >> ? >> >> Etienne >>> S >>> >>> On Wed, Jan 18, 2017 at 7:57 AM Etienne Chauchot ><echauc...@gmail.com> >>

Re: IO Integration tests - concrete proposal

to create user guide have sat in draft form >for a >while, I've wanted to finish up the integration testing environment for >IOs >first. > >S > >On Wed, Jan 25, 2017 at 8:52 AM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >Hi > >It's what I mentioned in

Re: Conceptually, what are bundles?

. were problems present in early versions of the runners solved by adding the concept of bundles? Thanks so much, Matt [1] http://elixir-lang.org/ -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Welcome aboard !⁣ Regards JB On Jan 27, 2017, 01:27, at 01:27, Davor Bonaci wrote: >Please join me and the rest of Beam PMC in welcoming the following >contributors as our newest committers. They have significantly >contributed >to the project in different ways, and we look

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

g it to DoFn as this code snippet implies). >I'm >reasonably sure there are less invasive ways of implementing it. >On Thu, Jan 26, 2017 at 12:22 PM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Agree, I'm curious as well. >> >> I guess it w

Re: Release 0.5.0

JB On 01/23/2017 07:32 PM, Jean-Baptiste Onofré wrote: Hi Davor, Yes, it makes sense to prepare 0.5.0. If no other wants to take this release, I'm volunteer. Regards JB On 01/23/2017 07:29 PM, Davor Bonaci wrote: It's been about a month since the last release, so we should start preparing

Re: Better developer instructions for using Maven?

;>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson < >>>>>>>>> >>>>>>>> je...@smokinghand.com >>>>> >>>>>> >>>>>>> wrote: >>>>>>>&g

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

perElementFn belongs as a ParDo that proceeds this PTransform, rather than as part of it. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

[VOTE] Apache Beam, version 0.5.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: Next major milestone: first stable release

If we have a consensus, I’d like to start marking progress on this effort rather quickly. Perhaps we can jointly coordinate a project-wide effort to polish the last few things and reach the first stable release. Thanks! Davor [1] http://semver.org/ -- Jean-Baptiste Onofré jbono...@apach

Re: Hbase IO preview

ep you informed once the branch is ready for testing/contribution so you can try it (or help us improve it). For the moment you can track the progress following this JIRA https://issues.apache.org/jira/browse/BEAM-1157 On Wed, Dec 14, 2016 at 3:25 PM, Jean-Baptiste Onofré <j...@nanthrax.net>

Re: Beam on Kubernetes

a company email but neither works): If you have an *@apache.org <http://apache.org> or @google.com <http://google.com> *email address, you can create an account <https://apachebeam.slack.com/signup/>. On Tue, Feb 21, 2017 at 3:01 PM, Jean-Baptiste Onofré <j...@nanthrax.net

Re: Better developer instructions for using Maven?

;da...@apache.org> wrote: (I'm also in favor of not overloading existing flags; they have some meaning/semantics that developers have come to expect.) On Sun, Feb 19, 2017 at 12:34 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Thanks Aviem, Not sure if we should use skipTests as it

Re: Build failed in Jenkins: beam_PostCommit_Python_Verify #1308

--- >> begin captured logging << ---- root: DEBUG: PValue computed by side (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by start (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by ViewAsIterable(side.None)/CreatePCollectionView (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by compute (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/WindowInto(WindowIntoFn) (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/ToVoidKey (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/Group (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/UnKey (tag 1): refcount: 1 => 0 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/py-validatesrunner-1487491859.1487491878.825933/requirements.txt... - >> end captured logging << - -- Ran 14 tests in 18.228s FAILED (errors=13) Build step 'Execute shell' marked build as failure -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Build failed in Jenkins: beam_PostCommit_Python_Verify #1308

at 12:13 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: It seems Python build is failing. Gonna take a look but if a guy from the Python can help me, it would be great. Thanks ! Regards JB On 02/19/2017 09:11 AM, Apache Jenkins Server wrote: See <https://builds.apach

Re: Metrics for Beam IOs.

am/runners to handle backlog as well. Of course there will be some source specific metrics too (errors, i/o ops etc). -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Jenkins build is unstable: beam_Release_NightlySnapshot #331

it is a one-time issue or something actionable. Thoughts? On Thu, Feb 16, 2017 at 11:44 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: See <https://builds.apache.org/job/beam_Release_ NightlySnapshot/331/changes> -- Jean-Baptiste Onofré jbono...@apa

Re: Metrics for Beam IOs.

lue is atomically incremented (or checkpointed) when the bundle it was reported in is committed. Aviem has already implemented Metrics support for the Spark runner. I am working on support for the Dataflow runner. On Fri, Feb 17, 2017 at 7:50 AM Jean-Baptiste Onofré <j...@nanthrax.net> w

Re: Metrics for Beam IOs.

nner involvement to ensure that a given metric value is atomically incremented (or checkpointed) when the bundle it was reported in is committed. Aviem has already implemented Metrics support for the Spark runner. I am working on support for the Dataflow runner. On Fri, Feb 17, 2017 at 7:50 AM Jean

Re: Metrics for Beam IOs.

the runner to push metrics more directly to other metrics stores. This layer needs some thought since it has to handle the complexity of attempted/committed metrics to be consistent with the model. On Sat, Feb 18, 2017, 5:44 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Amit, befor

Re: Metrics for Beam IOs.

g mechanism is "pull" ? Codahale ScheduledReporter will sample the values every X and send to ... So any runner using a "pull-like" would use an adapter ? On Sat, Feb 18, 2017 at 6:27 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ben, ok it's what I thought. Thanks

Re: Metrics for Beam IOs.

ers, right ? perhaps providing them a resilient underlying FS ? and if he wants, setup monitoring (which even with the API proposed he'd have to do). I just don't see why it should be a part of the runner and/or Metrics API. On Sat, Feb 18, 2017 at 6:35 PM Jean-Baptiste Onofré <j...@nanthrax.net&g

Re: Better developer instructions for using Maven?

Thanks Kenn, it's perfectly clear now ;) Regards JB On 02/22/2017 04:40 PM, Kenneth Knowles wrote: On Wed, Feb 22, 2017 at 6:50 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Can we finalize a decision about this ? So basically, the question is: 1. Do we enable checkstyle, fi

Re: Metrics for Beam IOs.

this and if this should be part of Beam. What do you guys think about the pending issues? Did I miss something else ? Ismaël On Sat, Feb 18, 2017 at 9:02 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Yes, agree Ben. More than the collected metrics, my question is more how to "expose&qu

Re: Interest in a (virtual) contributor meeting?

decisions are reserved for the mailing list discussions. If you'd be interested in attending, please reply back. If there's sufficient interest, I'd be happy to try to organize something in the near future. Thanks! Davor -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Metrics for Beam IOs.

Hi Ismael Good point to discuss the metric here. Imho, the IOs should use the metric API to provide specific IO metrics (mile number of split, throughput, reading/writing rate, number of splits, etc). In Camel, each processor (aka IO) provides such metric indicators. The purpose is to provide

Re: Metrics for Beam IOs.

Hi Aviem Agree with your comments, it's pretty close to my previous ones. Regards JB On Feb 14, 2017, 12:04, at 12:04, Aviem Zur wrote: >Hi Ismaël, > >You've raised some great points. >Please see my comments inline. > >On Tue, Feb 14, 2017 at 3:37 PM Ismaël Mejía

Re: Better developer instructions for using Maven?

s:check' => Total time: >11:13 >min (Added 01:36 min) > >Are these times reasonable enough to add rat and findbugs to the >default >build? > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Hi >> >>

Re: Better developer instructions for using Maven?

On Jenkins it's possible to run several jobs in the same time but on different executor. That's the easiest way. Regards JB On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" wrote: >This question got lost in the discussion, but there is a small >improvement >that we can do: >

Re: Release 0.5.0

! Davor [1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http

Re: Better developer instructions for using Maven?

LID> wrote: >Before we added checkstyle it was under a minute. Now it's over five? >That's awful IMO >On Fri, Feb 10, 2017 at 07:14 Aviem Zur <aviem...@gmail.com> wrote: > >> Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457 >> >> On Fri, Feb 10

Re: IO Integration tests - concrete proposal

3e761ab6ff4e8fa7b469@%3Cdev.beam.apache.org%3E [4] JDBC IO IT using postgres https://github.com/ssisk/beam/tree/io-testing/sdks/java/io/jdbc - have not been reviewed yet, so may contain code errors, but it does run & pass :) -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: On my activity at the project

. Please feel free to reach out to me in case anything pops up. It's great to see Beam as an established top level project. Everyone at the Beam community can be really proud! Best, Max -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Composite Types and the Runner API

Thanks, Thomas -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSS] Python SDK status and next steps

, etc. Great work everyone! On Tue, Jan 17, 2017 at 8:26 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi I didn't try the Python SDK recently but you provided a clear "state of the art". Anyway I'm in favor of merging things as quick as possible (assuming it's in a go

Re: IO testing: failure scenarios

Agree. It's predictable or unpredictable errors. It also includes the checkpoint case we discussed yesterday. Thanks for bringing this point. Regards JB⁣​ On Jan 19, 2017, 12:07, at 12:07, Stephen Sisk wrote: >This is a discussion that I don't think affects any

Re: Introduction

with all of you! Kind Regards, -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Next major milestone: first stable release

c to check against from such annotations, though tooling is secondary to documentation. Kenn -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: First stable release: version designation?

nces -- names are important and well correlated with success. Thanks! Davor [1] https://lists.apache.org/thread.html/c35067071aec9029d9100ae973c629 9aa919c31d0de623ac367128e2@%3Cdev.beam.apache.org%3E -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Release 0.6.0

Thanks Ahmet ! Regards JB On 03/02/2017 07:42 AM, Ahmet Altay wrote: Sure, I can wait. To be clear, Thursday night in which time zone? Thank you, Ahmet On Wed, Mar 1, 2017 at 10:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ahmet, Can you wait up to Thursday night ?

Re: Pipeline termination in the unified Beam model

tion until it receives a "poison pill" message. This seems handy. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

By the way Stephen, when BEAM-59 will be done, hadoop IO will only contains the hadoop format support (no HdfsFileSource or HdfsSink required as it will use the "regular" FileIO). Agree ? Regards JB On 03/02/2017 03:27 PM, Jean-Baptiste Onofré wrote: Hi Stephen, I ag

Re: Adding to the Contributor List

Done Regards JB On 02/27/2017 06:17 PM, Neelesh Salian wrote: Thanks JB. On Mon, Feb 27, 2017 at 9:14 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Neelesh, sure, I will update. Regards JB On 02/27/2017 06:08 PM, Neelesh Salian wrote: Hi, Had to change my userna

[VOTE] Release 0.4.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 0.4.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: Running a Specific Test

Hi Jesse Mvn test -Dtest=RegexTest Should work Don't forget the test goal. And no need to provide the fqcn. Regards JB⁣​ On Dec 29, 2016, 18:55, at 18:55, Jesse Anderson wrote: >Does anyone know the Maven way to run a specific unit test with Beam? >I've >tried: >mvn

[RESULT][VOTE] Release 0.4.0, release candidate #1

Hi, I'm happy to announce that the vote passed with the following result: +1 (binding): Dan Halperin, Davor Bonaci, Aljoscha Krettek, Jean-Baptiste Onofré There are no disapproving votes. Thanks all for your vote. Regards JB On 12/29/2016 08:33 AM, Jean-Baptiste Onofré wrote: Hi everyone

Re: Checkstyle Errors

found. I thought this was due to the extra parameters. I reran with the plain "mvn verify" and it still didn't find them. From the output, it doesn't look like they're being run at all. Thanks, Jesse -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: PCollection to PCollection Conversion

ing", I think JB has a good point leveraging JAXB and Jackson - though I think this should be in extensions as it is not as lean as toString. Thanks, Amit On Wed, Nov 30, 2016 at 5:13 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Jesse, yes, I started something there (

Re: [DISCUSSION] using NexMark for Beam

ntributing, or have more ideas, you are welcome :) Etienne -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSSION] rename StateSpecs.combiningValue?

ate (but there is none). Maybe StateSpecs.accumulatorCombiningValue would be a better name? The same goes for StateSpecs.*CombiningValue methods that all return AccumulatorCombiningState but maybe the name would be too long. WDYT? Etienne -- Jean-Baptiste Onofré jbono...@apache.org http://bl

Re: Beam spark 2.x runner status

't think of them off the top of my head right now). @JB simply add a profile for the Spark dependencies and run the tests - you'll have a very definitive answer ;-) . If this passes, try on a cluster running Spark 2 as well. Let me know of I can assist. On Thu, Mar 23, 2017 at 6:55 AM Jean-Ba

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Mar 20, 2017 at 7:34 AM Eugene Kirpichov <kirpic...@google.com> wrote: Thomas - yes, 9:30 works, shall we do that? JB - excellent! You can start experimenting already, using direct runner! On Mon, Mar 20, 2017, 2:26 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Eugene, Thanks fo

Re: Beam spark 2.x runner status

HBASE-16179 where compatibility modules are created to isolate changes in Spark 2.x API so that code in hbase-spark module can be reused. FYI -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: splitIntoBundles vs. generateInitialSplits

11, 2017 at 4:30 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Eugene and Stas, Just back from couple of days off and jump on this discussion. I agree with Stas: it's worth to create a Jira about that. The only "semantic" difference is unbounded vs bounded source,

Re: why Source#validate() is not declared to throw any exception

} catch (IOException | InterruptedException e) { throw new RuntimeException(e); } Why is validate() not declared to throw any exception ? If validation doesn't pass, there is nothing to clean up ? Thanks -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Tal

Re: Beam spark 2.x runner status

t 9:07 AM Cody Innowhere <e.neve...@gmail.com> wrote: Hi guys, Is there anybody who's currently working on Spark 2.x runner? A old PR for spark 2.x runner was closed a few days ago, so I wonder what's the status now, and is there a roadmap for this? Thanks~ -- Jean-Baptiste Onofré jbono...@apac

Re: [RESULT] [VOTE] Release 0.6.0, release candidate #2

4 of which are binding: * Aljoscha Krettek * Davor Bonaci * Ismaël Mejía * Jean-Baptiste Onofré * Robert Bradshaw * Ted Yu * Tibor Kiss There are no disapproving votes. Thanks everyone! Ahmet -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [RESULT] [VOTE] Release 0.6.0, release candidate #2

, Jesse Anderson <je...@bigdatainstitute.io> wrote: Excellent! On Wed, Mar 15, 2017, 6:13 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ahmet, it seems Jira is not up to date: 0.6.0 version is not flagged as "Released". Can you fix that please ? Thanks ! Regards JB

Re: [RESULT] [VOTE] Release 0.6.0, release candidate #2

ank you for catching this. As a side note, I did not have enough permissions do this and asked Davor to do. I will add this to the release notes. Ahmet On Wed, Mar 15, 2017 at 7:16 AM, Jesse Anderson <je...@bigdatainstitute.io> wrote: Excellent! On Wed, Mar 15, 2017, 6:13 AM Jean-Baptiste Onofré

Re: Beam spark 2.x runner status

ng time, this should be merged into master really fast for the benefit of everybody. Ismaël On Wed, Mar 15, 2017 at 1:57 PM, Jean-Baptiste Onofré < j...@nanthrax.net> wrote: Hi Amit, What do you think of the following: - in the mean time that you reintroduce the Spark 2 branch, what about

Re: Beam spark 2.x runner status

or which translates into DataSets. On Thu, Mar 16, 2017 at 9:33 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi guys, sorry, due to the time zone shift, I answer a bit late ;) I think we can have the same runner dealing with the two major Spark version, introducing some adapters

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

should make implementation of SDF easier. On Wed, Feb 8, 2017 at 7:06 PM, Eugene Kirpichov < kirpic...@google.com wrote: Thanks! Looking forward to this work. On Wed, Feb 8, 2017 at 3:50 AM Jean-Baptiste Onofré < j...@nanthrax.net wrote: Thanks for the update Eugene. I will work on the

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

/python/apache_beam/io/iobase.py#L561 [4] https://s.apache.org/splittable-do-fn -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: why Source#validate() is not declared to throw any exception

-1773 FYI On Mon, Mar 20, 2017 at 11:26 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: I think it would make sense to allow the validate method to throw Exception. On Mon, Mar 20, 2017, 11:21 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ted, validate() is suppo

Re: why Source#validate() is not declared to throw any exception

allow the validate method to throw Exception. On Mon, Mar 20, 2017, 11:21 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ted, validate() is supposed to throw runtime exception (IllegalStateException, RuntimeException, ...) to "traverse" the executor. Regards JB On 03/21/2017

Re: First IO IT Running!

-testing.appspot.com/explore?dashboard=5714163003293696 [3] https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73 -- --- Jason Kuster Apache Beam / Google Cloud Dataflow -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthr

Re: Docker image dependencies

ps://issues.apache.org/jira/browse/BEAM-1644 S -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSSION] Consistent use of loggers

be documented well as if we provide no binding there will be no logging out of the box unless the user adds a binding. On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré < j...@nanthrax.net> wrote: Hi Aviem, Good point. I think, in our dependencies set, we should just depend to slf4j-

Re: IO ITs: Hosting Docker images

people's ability to dev work (or they >need >> to be okay if it goes out.) Can you give some reasons why you think >it's >> better than the other options? I put it on the list, but I'm strongly >not a >> fan. >> >> S >> >> On Sat, Apr 8, 2017 at

Re: Build failing on the Apex runner

Weise <t...@apache.org> wrote: I'm not sure what the issue is, but that commit shouldn't affect the test listed and the Apex tests pass when I run them. Thomas On Sat, Apr 8, 2017 at 11:15 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hii huys, FYI, the build is failing

Re: Build failing on the Apex runner

That explains ;) Thanks, Regards JB On 04/09/2017 07:34 PM, Pramod Immaneni wrote: There was a respin done of that particular version of netlet. It is not a usual practice and it was an exception. On Sun, Apr 9, 2017 at 10:32 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: I

Re: Build failing on the Apex runner

uldn't affect the test listed and the Apex tests pass when I run them. Thomas On Sat, Apr 8, 2017 at 11:15 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hii huys, FYI, the build is failing on the Apex runner tests (due to NullPointerException): ParDoTranslatorTest.testSeria

Re: HDFS and Google Cloud Storage

nelFactory (or the new name FileSystem). All others, e.g., HTTP, TCP, KV-store, DB, message-queue, should be source/sink transforms? Thanks, Shen On Tue, Apr 11, 2017 at 10:29 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Shen, We are doing a refactoring of the file IO (IOChannel

Re: HDFS and Google Cloud Storage

ector, how can I determine whether it should be implemented as a source transform or as a scheme for the TextIO? Thanks, Shen -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [PROPOSAL]: a new feature branch for SQL DSL

ing an operational question about where the state for a given SQL pipeline will live? -Tyler On Sun, Apr 9, 2017 at 12:39 PM Mingmin Xu <mingm...@gmail.com> wrote: Thanks @JB, will come out the initial PR soon. On Sun, Apr 9, 2017 at 12:28 PM, Jean-Baptiste Onofré < j...@nanthrax.n

Re: Want to contribute to Beam project

in this exciting journey. Please guide me to where and how to start so that I can quickly pace with the active development and it would be great if you can assign something to me to start. Regards, Tarush -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [PROPOSAL] ORC support

-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Low availability on my end in the coming 3 weeks

is more mature now so my absence won't be noticed :-) but I still asked JB to formally take my place with anything concerning the Spark runner and he kindly agreed. Thanks, Amit -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [PROPOSAL] Standard IO Metrics

IOs. I've written this up as a proposal found here: https://s.apache.org/standard-io-metrics As usual, this document is commentable, please go over it and make comments where appropriate. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Build failing on the Apex runner

ApexStateInternalsTest.initStateInternals:83 » NullPointer ApexStateInternalsTest.initStateInternals:83 » NullPointer ApexStateInternalsTest.initStateInternals:83 » NullPointer I gonna submit a PR to fix that (it's due to commit 42f63342be667649eff5ea65dd9714c8b2f75cff). Regards JB -- Jean-Baptiste

Re: Let's make Beam transforms comply with PTransform Style Guide

Cool, I gonna take a look on PubSub later today (I would like to finish CassandraIO, HDFS refactoring and Spark 2 support first ;)). Regards JB On 04/21/2017 06:03 AM, tarush grover wrote: Hi, I can take kinesis one. Regards, Tarush On Thu, 20 Apr 2017 at 11:18 AM, Jean-Baptiste Onofré &l

Re: Let's make Beam transforms comply with PTransform Style Guide

Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Cool, I gonna take a look on PubSub later today (I would like to finish CassandraIO, HDFS refactoring and Spark 2 support first ;)). Regards JB On 04/21/2017 06:03 AM, tarush grover wrote: Hi, I can take kinesis one. Regards, Tarush On T

Re: splitIntoBundles vs. generateInitialSplits

way these two methods carry out a very similar task, would it be reasonable to think of *splitIntoBundles *as *generate*Initial*Splits? * (strikethrough due to "initial" not being applicable in the case of bounded sources) Regards, Stas -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

  1   2   3   4   5   6   7   8   9   10   >