Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-02-02 Thread Jean-Baptiste Onofré
reply on the vote thread to cancel the vote. There’s no need to wait 72 hours." On Wed, Feb 1, 2017 at 11:45 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ken, agree. I'm waiting for other -1 to formally cancel the vote. Thanks ! Regards JB On 02/02/2017 04:11 AM, Kenne

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-02-01 Thread Jean-Baptiste Onofré
to use new DoFn, maybe more. On Tue, Jan 31, 2017 at 10:21 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Basically, my question is: is it a regression ? If yes, definitely a -1 and we should cancel the release. Correct me if I'm wrong, but the commits in the LateDataDroppingDoFnR

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-02-02 Thread Jean-Baptiste Onofré
hours." On Wed, Feb 1, 2017 at 11:45 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ken, agree. I'm waiting for other -1 to formally cancel the vote. Thanks ! Regards JB On 02/02/2017 04:11 AM, Kenneth Knowles wrote: Unless there is a strenuous objection, I think we shou

Re: Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2517

2017-02-02 Thread Jean-Baptiste Onofré
For the record: testAuthenticationWithBadPassword 59ms Fixed As said, I will watch for potential new failure in the future. Regards JB On 02/03/2017 05:48 AM, Jean-Baptiste Onofré wrote: New build (#2519) worked on JmsIO. Let's keep the Jira open and I will watch for failure. Thanks

Re: [DISCUSS] Python SDK status and next steps

2017-01-31 Thread Jean-Baptiste Onofré
e/BEAM-1251 Great summary, Ahmet. Thanks. Cheers, -- Sergio Fernández Partner Technology Manager Redlink GmbH m: +43 6602747925 e: sergio.fernan...@redlink.co w: http://redlink.co -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSS] Python SDK status and next steps

2017-01-31 Thread Jean-Baptiste Onofré
Just to be clear, the prerequisite to be able to build the Python SDK are: apt-get install python-setuptools apt-get install python-pip It's also required by the default "regular" build. Regards JB On 01/31/2017 11:02 AM, Jean-Baptiste Onofré wrote: Just one thing I notice

Re: [DISCUSS] Python SDK status and next steps

2017-01-31 Thread Jean-Baptiste Onofré
it'd be better to have it under a separated profile or so? On Tue, Jan 31, 2017 at 11:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Just to be clear, the prerequisite to be able to build the Python SDK are: apt-get install python-setuptools apt-get install python-pip It'

Re: [DISCUSS] Python SDK status and next steps

2017-01-31 Thread Jean-Baptiste Onofré
Awesome, thanks Sergio ! Much appreciated ;) Regards JB On 01/31/2017 01:42 PM, Sergio Fernández wrote: PR #1879 provides the basics: https://github.com/apache/beam/pull/1879 On Tue, Jan 31, 2017 at 1:33 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: No, that's fine as soon

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-31 Thread Jean-Baptiste Onofré
also not have proper late data dropping and it's fine to have a release with some missing features. (There's more besides dropping.) I think we should go ahead and fix for 0.6. On Tue, Jan 31, 2017, 18:23 Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Aljoscha, so you propose to

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-31 Thread Jean-Baptiste Onofré
On 01/31/2017 07:13 PM, Dan Halperin wrote: Should we revert the CLs that lost the functionality? I'd really not like to ship a release with such a functional regression On Tue, Jan 31, 2017 at 10:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Fair enough. Let's do that.

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-31 Thread Jean-Baptiste Onofré
s.apache.org/job/beam_PostCommit_Java_RunnableOnService_Dataflow/2130/ On Mon, Jan 30, 2017 at 4:40 PM, Dan Halperin <dhalp...@apache.org> wrote: I am worried about https://issues.apache.org/jira/browse/BEAM-1346 for RC1 and would at least wait for resolution there before proceeding. On Mon, Jan 30, 2017 at 3:48 AM

Re: Report to the Board, February 2017 edition

2017-02-07 Thread Jean-Baptiste Onofré
Hi It looks good to me. Thanks Davor Regards JB On Feb 6, 2017, 19:32, at 19:32, Davor Bonaci wrote: >We are expected to submit a project report to the ASF Board of >Directors >ahead of its next meeting. The report is due on Wednesday, 2/8. > >This is the second is the series

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-02-08 Thread Jean-Baptiste Onofré
;>>> and you just want to start an async RPC for each batch. Then in >>>>> addition >>>>> to >>>>> start the final RPC in finishBundle, you also need to wait for all >the >>>>> RPCs >>>>> to complete. >>>>

Re: [CANCEL][VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-02-02 Thread Jean-Baptiste Onofré
It sounds good. Let's do that. Regards JB On 02/02/2017 12:12 PM, Aljoscha Krettek wrote: We should go with the option Kenn proposed instead of a "hard revert". On Thu, 2 Feb 2017 at 10:55 Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi everyone, Due to the regress

Re: Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2517

2017-02-02 Thread Jean-Baptiste Onofré
New build (#2519) worked on JmsIO. Let's keep the Jira open and I will watch for failure. Thanks, Regards JB On 02/03/2017 05:26 AM, Jean-Baptiste Onofré wrote: Hmmm. Test on the PR worked. Let me take a look. Regards JB⁣​ On Feb 3, 2017, 03:27, at 03:27, Kenneth Knowles &l

Re: Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2517

2017-02-02 Thread Jean-Baptiste Onofré
Hmmm. Test on the PR worked. Let me take a look. Regards JB⁣​ On Feb 3, 2017, 03:27, at 03:27, Kenneth Knowles wrote: >Failure is JmsIO. This doesn't look related to the changes, but doesn't >look like a flake, so I'm not sure what to make of it. > >The test is looking

JB off for 2 weeks

2017-02-02 Thread Jean-Baptiste Onofré
Hi guys, I'm in vacation tonight for 2 weeks. I will have access to my e-mail and I will try to follow to different discussions and threads. However, I can have some delay in my replies depending the tequila I will have in my blood ;) See you soon ! Regards JB -- Jean-Baptiste Onofré

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

2017-02-08 Thread Jean-Baptiste Onofré
Thanks for the update Eugene. I will work on the spark runner with Amit. Regards JB On Feb 7, 2017, 19:12, at 19:12, Eugene Kirpichov wrote: >Hello, > >I'm almost done adding support for Splittable DoFn >http://s.apache.org/splittable-do-fn to Dataflow streaming

Re: Let's make Beam transforms comply with PTransform Style Guide

2017-02-08 Thread Jean-Baptiste Onofré
Thanks Eugene. I will tackle some Jira when back next week. Regards JB On Feb 7, 2017, 18:16, at 18:16, Eugene Kirpichov wrote: >Hey all, > >I bit the bullet and audited all PTransform classes in Beam Java SDK >and >filed JIRA issues for all violations I could

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #2

2017-02-04 Thread Jean-Baptiste Onofré
+1 (binding) Regards JB⁣​ On Feb 4, 2017, 05:32, at 05:32, Aljoscha Krettek wrote: >+1 > >- Checked signatures >- Verified that source release builds >- Verified that code contains fix for the regression in the Flink >Runner >- Tried Quickstart against Staging Repository >-

Re: Request for becoming a contributor

2017-01-24 Thread Jean-Baptiste Onofré
assign JIRAs at >this point? > >On Tue, Jan 24, 2017 at 9:28 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Hi >> >> I would leave without assignee when nobody's working on the jira. >Then the >> contributor can open a PR and the reviewer can assign

Re: Better developer instructions for using Maven?

2017-01-24 Thread Jean-Baptiste Onofré
the checkstyle. On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov <kirpic...@google.com.invalid> wrote: Try just -Prelease. On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson < je...@smokinghand.com wrote: Fails because I don't have a secret key. On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptis

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-01-26 Thread Jean-Baptiste Onofré
PTransform that implements the batching in its own DoFn and that takes user defined functions for customization. Etienne Le 17/01/2017 à 17:30, Jean-Baptiste Onofré a écrit : Hi I guess you mean discussion on the mailing list about that, right ? AFAIR the ide⁣​a is to provide a utility cl

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-01-26 Thread Jean-Baptiste Onofré
ss of the bundles. For example, possible use case is to batch a call to an external service (for performance). I was thinking about providing a PTransform that implements the batching in its own DoFn and that takes user defined functions for customization. Etienne Le 17/01/2017 à 17:

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-30 Thread Jean-Baptiste Onofré
a formality, "PPMC" is not committee that plays a role anymore, you're a PMC now ;-) On Fri, Jan 27, 2017 at 9:55 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve th

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-30 Thread Jean-Baptiste Onofré
+1 (binding) Regards JB On 01/27/2017 09:55 PM, Jean-Baptiste Onofré wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete

Re: TextIO binary file

2017-01-30 Thread Jean-Baptiste Onofré
other than Strings. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: IO Authoring Guide - first draft

2017-01-27 Thread Jean-Baptiste Onofré
I/edit?usp=sharing -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: IO Integration tests - concrete proposal

2017-01-25 Thread Jean-Baptiste Onofré
>> - write tests: use another storage place than read tests (using >> namespace for example) and clean it after each test. >> ? >> >> Etienne >>> S >>> >>> On Wed, Jan 18, 2017 at 7:57 AM Etienne Chauchot ><echauc...@gmail.com> >>

Re: IO Integration tests - concrete proposal

2017-01-25 Thread Jean-Baptiste Onofré
to create user guide have sat in draft form >for a >while, I've wanted to finish up the integration testing environment for >IOs >first. > >S > >On Wed, Jan 25, 2017 at 8:52 AM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >Hi > >It's what I mentioned in

Re: Conceptually, what are bundles?

2017-01-26 Thread Jean-Baptiste Onofré
. were problems present in early versions of the runners solved by adding the concept of bundles? Thanks so much, Matt [1] http://elixir-lang.org/ -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

2017-01-26 Thread Jean-Baptiste Onofré
Welcome aboard !⁣ Regards JB On Jan 27, 2017, 01:27, at 01:27, Davor Bonaci wrote: >Please join me and the rest of Beam PMC in welcoming the following >contributors as our newest committers. They have significantly >contributed >to the project in different ways, and we look

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-01-26 Thread Jean-Baptiste Onofré
g it to DoFn as this code snippet implies). >I'm >reasonably sure there are less invasive ways of implementing it. >On Thu, Jan 26, 2017 at 12:22 PM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Agree, I'm curious as well. >> >> I guess it w

Re: Release 0.5.0

2017-01-27 Thread Jean-Baptiste Onofré
JB On 01/23/2017 07:32 PM, Jean-Baptiste Onofré wrote: Hi Davor, Yes, it makes sense to prepare 0.5.0. If no other wants to take this release, I'm volunteer. Regards JB On 01/23/2017 07:29 PM, Davor Bonaci wrote: It's been about a month since the last release, so we should start preparing

Re: Better developer instructions for using Maven?

2017-01-25 Thread Jean-Baptiste Onofré
;>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson < >>>>>>>>> >>>>>>>> je...@smokinghand.com >>>>> >>>>>> >>>>>>> wrote: >>>>>>>&g

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-01-27 Thread Jean-Baptiste Onofré
perElementFn belongs as a ParDo that proceeds this PTransform, rather than as part of it. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

[VOTE] Apache Beam, version 0.5.0, release candidate #1

2017-01-27 Thread Jean-Baptiste Onofré
Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: Beam on Kubernetes

2017-02-21 Thread Jean-Baptiste Onofré
0, 2017 at 2:05 PM, Davor Bonaci <da...@apache.org> wrote: I think these are great ideas for simplifying the getting started experience across runners -- we'd love a contribution in this space! On Mon, Feb 20, 2017 at 12:46 AM, Jean-Baptiste Onofré < j...@nanthrax.net> wrote: Hi

Re: Next major milestone: first stable release

2017-02-21 Thread Jean-Baptiste Onofré
If we have a consensus, I’d like to start marking progress on this effort rather quickly. Perhaps we can jointly coordinate a project-wide effort to polish the last few things and reach the first stable release. Thanks! Davor [1] http://semver.org/ -- Jean-Baptiste Onofré jbono...@apach

Re: Hbase IO preview

2017-02-22 Thread Jean-Baptiste Onofré
ep you informed once the branch is ready for testing/contribution so you can try it (or help us improve it). For the moment you can track the progress following this JIRA https://issues.apache.org/jira/browse/BEAM-1157 On Wed, Dec 14, 2016 at 3:25 PM, Jean-Baptiste Onofré <j...@nanthrax.net>

Re: Beam on Kubernetes

2017-02-22 Thread Jean-Baptiste Onofré
a company email but neither works): If you have an *@apache.org <http://apache.org> or @google.com <http://google.com> *email address, you can create an account <https://apachebeam.slack.com/signup/>. On Tue, Feb 21, 2017 at 3:01 PM, Jean-Baptiste Onofré <j...@nanthrax.net

Re: Better developer instructions for using Maven?

2017-02-22 Thread Jean-Baptiste Onofré
;da...@apache.org> wrote: (I'm also in favor of not overloading existing flags; they have some meaning/semantics that developers have come to expect.) On Sun, Feb 19, 2017 at 12:34 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Thanks Aviem, Not sure if we should use skipTests as it

Re: Build failed in Jenkins: beam_PostCommit_Python_Verify #1308

2017-02-19 Thread Jean-Baptiste Onofré
--- >> begin captured logging << ---- root: DEBUG: PValue computed by side (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by start (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by ViewAsIterable(side.None)/CreatePCollectionView (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by compute (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/WindowInto(WindowIntoFn) (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/ToVoidKey (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/Group (tag 1): refcount: 1 => 0 root: DEBUG: PValue computed by assert_that/UnKey (tag 1): refcount: 1 => 0 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/py-validatesrunner-1487491859.1487491878.825933/requirements.txt... - >> end captured logging << - -- Ran 14 tests in 18.228s FAILED (errors=13) Build step 'Execute shell' marked build as failure -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Build failed in Jenkins: beam_PostCommit_Python_Verify #1308

2017-02-19 Thread Jean-Baptiste Onofré
at 12:13 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: It seems Python build is failing. Gonna take a look but if a guy from the Python can help me, it would be great. Thanks ! Regards JB On 02/19/2017 09:11 AM, Apache Jenkins Server wrote: See <https://builds.apach

Re: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-19 Thread Jean-Baptiste Onofré
OK, I just checked and it's part of the PR. I'm checking out locally to check. I will let you know. Regards JB On 02/19/2017 08:33 PM, Jean-Baptiste Onofré wrote: Hi Dipti, regarding Elasticsearch, it seems that the classloader doesn't contain the EsInputSplit. Using spark-submit, you have

Re: IT issues faced for HadoopInputFormat IO on Spark and Dataflow runners

2017-02-19 Thread Jean-Baptiste Onofré
tems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and del

Re: Metrics for Beam IOs.

2017-02-17 Thread Jean-Baptiste Onofré
am/runners to handle backlog as well. Of course there will be some source specific metrics too (errors, i/o ops etc). -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
it was reported in is committed. Aviem has already implemented Metrics support for the Spark runner. I am working on support for the Dataflow runner. On Fri, Feb 17, 2017 at 7:50 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi guys, As I'm back from vacation, I'm back on this topic ;) It's a

Re: Jenkins build is unstable: beam_Release_NightlySnapshot #331

2017-02-18 Thread Jean-Baptiste Onofré
it is a one-time issue or something actionable. Thoughts? On Thu, Feb 16, 2017 at 11:44 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: See <https://builds.apache.org/job/beam_Release_ NightlySnapshot/331/changes> -- Jean-Baptiste Onofré jbono...@apa

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
lue is atomically incremented (or checkpointed) when the bundle it was reported in is committed. Aviem has already implemented Metrics support for the Spark runner. I am working on support for the Dataflow runner. On Fri, Feb 17, 2017 at 7:50 AM Jean-Baptiste Onofré <j...@nanthrax.net> w

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
nner involvement to ensure that a given metric value is atomically incremented (or checkpointed) when the bundle it was reported in is committed. Aviem has already implemented Metrics support for the Spark runner. I am working on support for the Dataflow runner. On Fri, Feb 17, 2017 at 7:50 AM Jean

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
leaves consistency up to disincline (meh...), but may make it easier on implementors to get things done as they have more freedom. -Stas On Sat, Feb 18, 2017 at 10:16 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Amit, my point is: how do we provide metric today to end user and how

Re: tf.Transform library for using TensorFlow with Beam

2017-02-23 Thread Jean-Baptiste Onofré
/research.googleblog.com/2017/02/preprocessing-for-machine- learning-with.html -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
ect metrics requires the execution engine to be involved, since it depends on how and which bundles are retried. What I'm not sure about is how much can be shared and/or made available for these runners vs. how much is tied to the execution engine. On Sat, Feb 18, 2017, 8:52 AM Jean-Baptiste Onofré <j

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
the runner to push metrics more directly to other metrics stores. This layer needs some thought since it has to handle the complexity of attempted/committed metrics to be consistent with the model. On Sat, Feb 18, 2017, 5:44 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Amit, befor

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
g mechanism is "pull" ? Codahale ScheduledReporter will sample the values every X and send to ... So any runner using a "pull-like" would use an adapter ? On Sat, Feb 18, 2017 at 6:27 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ben, ok it's what I thought. Thanks

Re: Metrics for Beam IOs.

2017-02-18 Thread Jean-Baptiste Onofré
ers, right ? perhaps providing them a resilient underlying FS ? and if he wants, setup monitoring (which even with the API proposed he'd have to do). I just don't see why it should be a part of the runner and/or Metrics API. On Sat, Feb 18, 2017 at 6:35 PM Jean-Baptiste Onofré <j...@nanthrax.net&g

Re: Better developer instructions for using Maven?

2017-02-22 Thread Jean-Baptiste Onofré
Thanks Kenn, it's perfectly clear now ;) Regards JB On 02/22/2017 04:40 PM, Kenneth Knowles wrote: On Wed, Feb 22, 2017 at 6:50 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Can we finalize a decision about this ? So basically, the question is: 1. Do we enable checkstyle, fi

Re: Metrics for Beam IOs.

2017-02-22 Thread Jean-Baptiste Onofré
this and if this should be part of Beam. What do you guys think about the pending issues? Did I miss something else ? Ismaël On Sat, Feb 18, 2017 at 9:02 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Yes, agree Ben. More than the collected metrics, my question is more how to "expose&qu

Re: Interest in a (virtual) contributor meeting?

2017-02-23 Thread Jean-Baptiste Onofré
decisions are reserved for the mailing list discussions. If you'd be interested in attending, please reply back. If there's sufficient interest, I'd be happy to try to organize something in the near future. Thanks! Davor -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Issue with Coder documentation regarding context

2017-02-09 Thread Jean-Baptiste Onofré
+1 We should create a Jira and submit a PR about documentation. Regards JB On Feb 9, 2017, 14:12, at 14:12, Aviem Zur wrote: >Hi, > >I think improvements can be made to the documentation of `encode` and >`decode` methods in `Coder`. > >A coder may be used to encode/decode

Re: Metrics for Beam IOs.

2017-02-14 Thread Jean-Baptiste Onofré
Hi Ismael Good point to discuss the metric here. Imho, the IOs should use the metric API to provide specific IO metrics (mile number of split, throughput, reading/writing rate, number of splits, etc). In Camel, each processor (aka IO) provides such metric indicators. The purpose is to provide

Re: Metrics for Beam IOs.

2017-02-14 Thread Jean-Baptiste Onofré
Hi Aviem Agree with your comments, it's pretty close to my previous ones. Regards JB On Feb 14, 2017, 12:04, at 12:04, Aviem Zur wrote: >Hi Ismaël, > >You've raised some great points. >Please see my comments inline. > >On Tue, Feb 14, 2017 at 3:37 PM Ismaël Mejía

Re: Better developer instructions for using Maven?

2017-02-10 Thread Jean-Baptiste Onofré
<aljos...@apache.org> >wrote: > >> +1 to what Dan said >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles <k...@google.com.invalid> >> wrote: >> >> > +1 >> > >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré&quo

Re: Better developer instructions for using Maven?

2017-02-10 Thread Jean-Baptiste Onofré
s:check' => Total time: >11:13 >min (Added 01:36 min) > >Are these times reasonable enough to add rat and findbugs to the >default >build? > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Hi >> >>

Re: We've hit 2000 PRs!

2017-02-16 Thread Jean-Baptiste Onofré
) Dan -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

2017-02-15 Thread Jean-Baptiste Onofré
Hi I guess your saw my comment in the PR. Basically I was waiting the refactoring of IOChannelFactory to refactore hdfs IO as hadoop file format on top of IOChannelFactory. I would have wait a bit and I would be more than happy to help you on the PR. Regards JB On Feb 15, 2017, 14:55, at

Re: Better developer instructions for using Maven?

2017-02-15 Thread Jean-Baptiste Onofré
On Jenkins it's possible to run several jobs in the same time but on different executor. That's the easiest way. Regards JB On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" wrote: >This question got lost in the discussion, but there is a small >improvement >that we can do: >

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

2017-02-15 Thread Jean-Baptiste Onofré
Hi It's what I said in the hadoop file format PR. When I discussed with Davor and Pei about the refactoring of the IOChannelFactory, I proposed to refactore hdfs IO to deal with hadoop file format on top of the file IO. Regards JB On Feb 15, 2017, 15:13, at 15:13, Raghu Angadi

Re: Release 0.5.0

2017-01-23 Thread Jean-Baptiste Onofré
! Davor [1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http

Re: Better developer instructions for using Maven?

2017-02-11 Thread Jean-Baptiste Onofré
LID> wrote: >Before we added checkstyle it was under a minute. Now it's over five? >That's awful IMO >On Fri, Feb 10, 2017 at 07:14 Aviem Zur <aviem...@gmail.com> wrote: > >> Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457 >> >> On Fri, Feb 10

Re: IO Integration tests - concrete proposal

2017-01-18 Thread Jean-Baptiste Onofré
3e761ab6ff4e8fa7b469@%3Cdev.beam.apache.org%3E [4] JDBC IO IT using postgres https://github.com/ssisk/beam/tree/io-testing/sdks/java/io/jdbc - have not been reviewed yet, so may contain code errors, but it does run & pass :) -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: On my activity at the project

2017-01-15 Thread Jean-Baptiste Onofré
. Please feel free to reach out to me in case anything pops up. It's great to see Beam as an established top level project. Everyone at the Beam community can be really proud! Best, Max -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Composite Types and the Runner API

2017-01-17 Thread Jean-Baptiste Onofré
Thanks, Thomas -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSS] Python SDK status and next steps

2017-01-17 Thread Jean-Baptiste Onofré
, etc. Great work everyone! On Tue, Jan 17, 2017 at 8:26 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi I didn't try the Python SDK recently but you provided a clear "state of the art". Anyway I'm in favor of merging things as quick as possible (assuming it's in a go

Re: IO testing: failure scenarios

2017-01-19 Thread Jean-Baptiste Onofré
Agree. It's predictable or unpredictable errors. It also includes the checkpoint case we discussed yesterday. Thanks for bringing this point. Regards JB⁣​ On Jan 19, 2017, 12:07, at 12:07, Stephen Sisk wrote: >This is a discussion that I don't think affects any

Re: Beam Fn API

2017-01-20 Thread Jean-Baptiste Onofré
mentation. 1: https://s.apache.org/beam-fn-api 2: https://github.com/apache/beam/pull/1801 Please comment on the overview within this thread, and any specific code comments on the PR directly. Luke -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Introduction

2017-02-27 Thread Jean-Baptiste Onofré
with all of you! Kind Regards, -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Next major milestone: first stable release

2017-03-01 Thread Jean-Baptiste Onofré
c to check against from such annotations, though tooling is secondary to documentation. Kenn -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: First stable release: version designation?

2017-03-01 Thread Jean-Baptiste Onofré
nces -- names are important and well correlated with success. Thanks! Davor [1] https://lists.apache.org/thread.html/c35067071aec9029d9100ae973c629 9aa919c31d0de623ac367128e2@%3Cdev.beam.apache.org%3E -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Release 0.6.0

2017-03-01 Thread Jean-Baptiste Onofré
Pacific time is fine. Regards JB On 03/02/2017 07:42 AM, Ahmet Altay wrote: Sure, I can wait. To be clear, Thursday night in which time zone? Thank you, Ahmet On Wed, Mar 1, 2017 at 10:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ahmet, Can you wait up to Thursday

Re: Release 0.6.0

2017-03-01 Thread Jean-Baptiste Onofré
Thanks Ahmet ! Regards JB On 03/02/2017 07:42 AM, Ahmet Altay wrote: Sure, I can wait. To be clear, Thursday night in which time zone? Thank you, Ahmet On Wed, Mar 1, 2017 at 10:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Ahmet, Can you wait up to Thursday night ?

Re: Pipeline termination in the unified Beam model

2017-03-02 Thread Jean-Baptiste Onofré
tion until it receives a "poison pill" message. This seems handy. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

2017-03-02 Thread Jean-Baptiste Onofré
By the way Stephen, when BEAM-59 will be done, hadoop IO will only contains the hadoop format support (no HdfsFileSource or HdfsSink required as it will use the "regular" FileIO). Agree ? Regards JB On 03/02/2017 03:27 PM, Jean-Baptiste Onofré wrote: Hi Stephen, I ag

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

2017-03-02 Thread Jean-Baptiste Onofré
you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Adding to the Contributor List

2017-02-27 Thread Jean-Baptiste Onofré
Done Regards JB On 02/27/2017 06:17 PM, Neelesh Salian wrote: Thanks JB. On Mon, Feb 27, 2017 at 9:14 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Neelesh, sure, I will update. Regards JB On 02/27/2017 06:08 PM, Neelesh Salian wrote: Hi, Had to change my userna

Re: Enforcer Rule- JDK1.7 for Beam

2017-02-27 Thread Jean-Baptiste Onofré
all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: splitIntoBundles vs. generateInitialSplits

2017-01-11 Thread Jean-Baptiste Onofré
ing if the names were intentionally made different, i.e. "into bundles" vs "into splits"? In a way these two methods carry out a very similar task, would it be reasonable to think of *splitIntoBundles *as *generate*Initial*Splits? * (strikethrough due to "initial"

Re: Some Thoughts on IO Integration Tests

2017-01-13 Thread Jean-Baptiste Onofré
igured for running in CI environment, they will likely fail when running on the user's machine, resulting in annoyance for the user. If anyone has thoughts on these, please let me know. Best, Jason -- --- Jason Kuster Apache Beam (Incubating) / Google Cloud Dataflow -- --- Jas

[CANCEL][VOTE] Release 0.4.0-incubating, release candidate #3

2016-12-28 Thread Jean-Baptiste Onofré
Hi all, as we have now graduated, and the IPMC don't have to vote for our releases, we cancel this release in order to start a fresh 0.4.0 release (without -incubating). Thanks anyway for your vote and sorry for the inconvenience. Regards JB On 12/16/2016 02:06 PM, Jean-Baptiste Onofré

[VOTE] Release 0.4.0, release candidate #1

2016-12-28 Thread Jean-Baptiste Onofré
Hi everyone, Please review and vote on the release candidate #1 for the version 0.4.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: [VOTE] Release 0.4.0, release candidate #1

2016-12-28 Thread Jean-Baptiste Onofré
2/29/2016 08:33 AM, Jean-Baptiste Onofré wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.4.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your re

Re: Running a Specific Test

2016-12-29 Thread Jean-Baptiste Onofré
Hi Jesse Mvn test -Dtest=RegexTest Should work Don't forget the test goal. And no need to provide the fqcn. Regards JB⁣​ On Dec 29, 2016, 18:55, at 18:55, Jesse Anderson wrote: >Does anyone know the Maven way to run a specific unit test with Beam? >I've >tried: >mvn

Re: Apache News Round-up

2016-12-30 Thread Jean-Baptiste Onofré
Thanks Davor and others ;) It's nothing compared with what you are all doing. I'm just trying to help some projects ;) Regards JB On 12/31/2016 06:23 AM, Davor Bonaci wrote: I stumbled across Apache News Round-up for this week [1], and our own Jean-Baptiste Onofré is noted as one of the top

[RESULT][VOTE] Release 0.4.0, release candidate #1

2017-01-02 Thread Jean-Baptiste Onofré
Hi, I'm happy to announce that the vote passed with the following result: +1 (binding): Dan Halperin, Davor Bonaci, Aljoscha Krettek, Jean-Baptiste Onofré There are no disapproving votes. Thanks all for your vote. Regards JB On 12/29/2016 08:33 AM, Jean-Baptiste Onofré wrote: Hi everyone

Re: Checkstyle Errors

2017-01-04 Thread Jean-Baptiste Onofré
found. I thought this was due to the extra parameters. I reran with the plain "mvn verify" and it still didn't find them. From the output, it doesn't look like they're being run at all. Thanks, Jesse -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Beam ML

2017-01-06 Thread Jean-Baptiste Onofré
/edit#heading=h.n51rhya8bv4f Are there any other docs/advances related to this? Best regards, Vladisav -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: PCollection to PCollection Conversion

2016-12-28 Thread Jean-Baptiste Onofré
ing", I think JB has a good point leveraging JAXB and Jackson - though I think this should be in extensions as it is not as lean as toString. Thanks, Amit On Wed, Nov 30, 2016 at 5:13 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi Jesse, yes, I started something there (

Re: [DISCUSSION] using NexMark for Beam

2017-03-21 Thread Jean-Baptiste Onofré
ntributing, or have more ideas, you are welcome :) Etienne -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: [DISCUSSION] rename StateSpecs.combiningValue?

2017-03-30 Thread Jean-Baptiste Onofré
ate (but there is none). Maybe StateSpecs.accumulatorCombiningValue would be a better name? The same goes for StateSpecs.*CombiningValue methods that all return AccumulatorCombiningState but maybe the name would be too long. WDYT? Etienne -- Jean-Baptiste Onofré jbono...@apache.org http://bl

  1   2   3   4   5   6   7   8   9   >