Re: Make runner implementation smoother?

2017-05-24 Thread Romain Manni-Bucau
) can provide #apache-beam on freenode Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.

graph generator?

2017-05-25 Thread Romain Manni-Bucau
/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.

Re: spark-submit forces jackson 2.4.4

2017-10-02 Thread Romain Manni-Bucau
> > > Jacob > > On Mon, Oct 2, 2017 at 11:17 AM, Jacob Marble <jmar...@kochava.com> wrote: > > > There is a lot of chatter about AWS and Jackson on their forums, etc. I > > have been using the AWS SDK and Jackson 2.8.9 for a couple of weeks > without &

Re: spark-submit forces jackson 2.4.4

2017-10-02 Thread Romain Manni-Bucau
Hi Jacob, isn't aws API only supporting jackson 2.6 and not 2.8? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
d but can we get something more reliable for the user? Maybe we need a @BeforeBatch or something like that. > > Regards > JB > > > On 11/15/2017 09:38 AM, Romain Manni-Bucau wrote: >> >> Hi guys, >> >> The subject is a bit provocative but the topic is real and coming

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
on the transform (or dofn) it would be used to control when the checkpoint is done. Thinking out loud it sounds close to jbatch checkpoint algorithm (https://docs.oracle.com/javaee/7/api/javax/batch/api/chunk/CheckpointAlgorithm.html) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
espected and you can process multiple times the same records. Any plan to make this API reliable and controllable from a beam point of view (at least in a max manner)? Thanks, Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
Hi Reuven, how does it help since you will still "send to the transform" the data before the commit point of beam and therefore not be able to reprocess the same data in case of a failure between your eager flush and next commit point? Romain Manni-Bucau @rmannibucau | Blog | Old Blo

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
Hmm, I didn't find the doc - if you have the link not far it would be appreciated - but "before" sounds not enough, it should be "after" in case there was a "flush" no? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
and aligned with the config machine. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 9:53 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > Just to answer a previous question: > > An ASF github search gives me these stats: > > - m

[discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
he doesn't modify at all - other languages. Wdyt? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
the frontend-maven-plugin if needed for other languages. That said it can be an interesting other thread since people consuming these languages will probably want their mainstream build tool and a "standard" repository layout rather than a java one. But this is harder to measure. > > On

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
hacks). Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-27 19:46 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > I have collected data by running several builds against master using Gradle > and Maven without using Gradle's support for increm

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
Hmm, issue is the same with gradle (locally python build takes 15mn alone which is as much as the java build and it is not parallelized I think) pl is not as smooth since it means doing it on each command whereas the proposal is automatically activated through settings.xml Romain Manni-Bucau

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
that was the goal: validate there was no side effect of the changes on the whole project. Now the "not java" part of the build will not be impacted by java changed so this is the part i want to skip since it takes a lot of time and I have guarantees it is safe to skip them. Romain M

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
have missed them) like the signing of artifacts etc. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 18:14 GMT+01:00 Kenneth Knowles <k...@google.com>: > Yea, I think voting is the next step. Luke - I think you are obviously the > right person to set

Re: [discuss] java profile

2017-11-28 Thread Romain Manni-Bucau
, a process, and some outputs is enough to know > when the process needs to be rerun. > > On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: > >> Hmm, no. >> >> Incremental build is never correctly implemented cause there is just

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
se > >>> >> of incorrectly specified inputs/outputs for certain tasks. > >>> >> > >>> >> The data is available here > >>> >> > >>> >> > >>> >> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_ > NJqEQakUgnNIQ7Qbjzu8

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-28 Thread Romain Manni-Bucau
-1 (non binding) gradle discourages contributions which is a big pitfall for an asf project and maven/gradle comparison is unfair due to the threading setup of maven (hardcoded thread count and no parallelize builder tusage). Le 28 nov. 2017 19:38, "Jason Kuster" a écrit

Re: gradle dirty files blocking maven build

2017-11-25 Thread Romain Manni-Bucau
re you running ? I don't find any ".gogradle" >>> files. >>> >>> Thanks, >>> Manu >>> >>> On Fri, Nov 24, 2017 at 5:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>> Let me try on my local

gradle dirty files blocking maven build

2017-11-24 Thread Romain Manni-Bucau
the build very slow in the best case and just locked in the worse one. Just in case you observe it, "find . -name '.gogradle' | xargs rm -Rf" solves it. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-28 Thread Romain Manni-Bucau
ap the sql usage in multiple UDF would make it powerful and ready to use. Le 29 nov. 2017 07:01, "Romain Manni-Bucau" <rmannibu...@gmail.com> a écrit : > My user wishes - whatever version, it is just a number after all ;): > > - make coder usage simpler and consistent (PCo

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-28 Thread Romain Manni-Bucau
My user wishes - whatever version, it is just a number after all ;): - make coder usage simpler and consistent (PCollection TypeDescriptor and Coder are duplicated in term of API) - have a beam api (split from the sdk and internals and impl) - have SDF supported by runners - have a SDFRunner

Re: [discuss] java profile

2017-11-29 Thread Romain Manni-Bucau
for incremental setup and is boring as well as ut slows down the dev workflow for cli driven dev. On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 29 nov. 2017 01:29, "Robert Bradshaw" <rober...@google.com> a écrit : >

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
@Ben: would all IO be rewritten to use that and the bundle concept dropped from the API to avoid any ambiguity and misleading usage like in current IOs? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 18:43 GMT+01:00 Ben Chambers <bchamb...@google.com>:

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
gt; On Thu, Nov 30, 2017 at 6:17 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: >> >> It sounds reasonable to me. >> >> And agree for Spark, I would like to merge Spark 2 update first. >> >> Regards >> JB >> >> On 11/30/2017 03:

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
rom the whole user API and make it all @Internal, no? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 18:58 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Agree, but maybe we can inform the runner if wanted no ? > > Honestly, from my side, I'm fin

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
2017-11-30 20:36 GMT+01:00 Kenneth Knowles <k...@google.com>: > On Thu, Nov 30, 2017 at 11:28 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: >> >> This is my short term concern yes. Note that the opposite is not sane >> neither (too big) cause it forces

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
handling/idempotence but this is more complicated so can be worth another thread once this one is fixed which can end up on being specific each time :(. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 20:21 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
Le 30 nov. 2017 19:23, "Kenneth Knowles" <k...@google.com> a écrit : On Thu, Nov 30, 2017 at 10:03 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hmm, > > ESIO: https://github.com/apache/beam/blob/master/sdks/java/io/elas > ticsearch/src/main/java/org/a

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
gradle build --no-daemon (with gradle 4.2) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-27 20:21 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > What is the gradle command you are using to build just the Python SDK? > > On Mon, Nov 27, 2

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
: > Manually whitelisting/blacklisting sub-modules is error prone since it > hides issues due to incorrectly maintaining that list is the same argument > as if the build process doesn't correctly invoke an incremental build > process. > > On Mon, Nov 27, 2017 at 1:45 PM, Romain Ma

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Le 27 nov. 2017 23:07, "Eugene Kirpichov" <kirpic...@google.com.invalid> a écrit : On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > > Romain, as menti

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Le 27 nov. 2017 21:34, "Lukasz Cwik" <lc...@google.com.invalid> a écrit : On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > > Romain, as mentioned earlie

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
: > Incremental builds aren't correctly setup right now so your likely to see > Python/Go rebuild even if there were no changes. See > https://issues.apache.org/jira/browse/BEAM-3253 > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> &g

Re: gradle dirty files blocking maven build

2017-11-28 Thread Romain Manni-Bucau
/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd [etc...] There is really something fishy in the build which is "breaking" the filesystem and making any indexing tool (like an IDE) broken. Romain M

Re: gradle dirty files blocking maven build

2017-11-25 Thread Romain Manni-Bucau
i, Nov 24, 2017 at 5:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Let me try on my local copy. > > > > Thanks for the report. > > > > Regards > > JB > > > > On 11/24/2017 10:04 AM, Romain Manni-Bucau wrote: > > >

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
Guys, what about moving getMaxBundleSize from flink options to pipeline options. I think all runners can support it right? Spark code needs the merge of the v2 before being able to be implemented probably but I don't see any blocker. wdyt? Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: On voting and process

2017-11-29 Thread Romain Manni-Bucau
Hi guys I think everyone has a good will and the issue is more coming from the fact that now 2 build systems have to be maintained - and are not symmetrically maintained :(. Will be fixee soon so no need to discuss it much IMHO. Next time a major change will be done the impacts once merged

Re: How to cope with Maven transient network issues?

2017-12-02 Thread Romain Manni-Bucau
dle to use that all the time? > > On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> If you use wagon http - and not lightweigh - it should have retries by >> default. >> >> Le 1 déc. 2017 22:31, "Valentyn Tymofi

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google.com>: > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: >> >> Need to check but if plugin dependencies were not tuned it should be the >> default with a retry

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
s on asf repo we must work with infra to fix it rather than working around the clients. If on a repo we dont control we can try to get rid of it maybe? On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpic...@google.com> a écrit : On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpic...@google.com> a écrit : > &g

Re: How to cope with Maven transient network issues?

2017-12-05 Thread Romain Manni-Bucau
FYI https://github.com/apache/maven-wagon/pull/37 Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > > > Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpic...@google.com>

Re: Euphoria Java 8 DSL - proposal

2017-12-17 Thread Romain Manni-Bucau
Hi guys A DSL would be very welcomed, in particular if fluent. Open question: did you study to implement Stream API (surely extending it to have a BeamStream and a few more features like sides etc)? Would be very natural and integrable easily anywhere and avoid a new API discovery. Hazelcast

Re: Euphoria Java 8 DSL - proposal

2017-12-18 Thread Romain Manni-Bucau
/cookit-project/blob/master/cookit-core/src/test/java/com/github/rmannibucau/cookit/api/RecipesTest.java#L177 Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github &

Re: [INFO] Spark runner updated to Spark 2.2.1

2017-12-18 Thread Romain Manni-Bucau
Congrats, was waited for a long time! Very impatient to see the announcement of the 2.3! Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://g

Re: [DISCUSS] Migrating Apache Beam PreCommits/PostCommits to Gradle Criteria

2017-12-15 Thread Romain Manni-Bucau
If it is the ~same Im not sure i see the point to move but please dont keep gradle and mvn in master - more than jenkins ;). It makes it hard to follow the project and do prs. Le 15 déc. 2017 20:42, "Lukasz Cwik" a écrit : > I'm proposing that the criteria to remove a

remove avro from core?

2017-12-19 Thread Romain Manni-Bucau
ion and remove it as a hard requirement of the core to bring more consistency and modularity to beam. Wdyt? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https:

Re: Apache Ignite as a distributed processing back-ends

2017-12-12 Thread Romain Manni-Bucau
; >> As for Ignite community, we would prefer to hold the integration in your >> repo. >> >> — >> Denis >> >> On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <rmannibu...@gmail.com> >> wrote: >> >> Hi >> >> This sounds awesome to ha

Re: makes bundle concept usable?

2017-11-17 Thread Romain Manni-Bucau
ot;retry" it seems it covers the needs. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 9:23 GMT+01:00 Eugene Kirpichov <kirpic...@google.com.invalid>: > I must admit I'm still failing to understand the problem, so let's step > back even further. >

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
2017-11-16 12:18 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > On Wed, Nov 15, 2017 at 9:16 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> @Reuven: it looks like a good workaround >> @Ken: thks a lot for the link! >> >> @all: >>

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
to keep the existing API? Le 17 nov. 2017 19:20, "Raghu Angadi" <rang...@google.com.invalid> a écrit : On Fri, Nov 17, 2017 at 1:02 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Yep, just take ES IO, if a part of a bundle fails you are in an > unmanaged state

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
custom state persistence. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 7:44 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Thanks for the explanation. Agree, we might talk about different things > using the same wording. > > I'm also

Re: [VOTE] Release 2.2.0, release candidate #3

2017-11-10 Thread Romain Manni-Bucau
Both issues are particular cases. Can the 2.2.0 be out and a 2.2.1 done quickly after? Would be very appreciated to have the 2.2.0 fixes to not depend on snapshots anymore due to some blockers found in the core of previous releases. Le 10 nov. 2017 21:23, "Chamikara Jayalath"

Re: [VOTE] Release 2.2.0, release candidate #4

2017-11-20 Thread Romain Manni-Bucau
Tested on some custom transforms and integrations, +1 (non-binding), thanks for the perseverance Reuven! Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-20 9:14 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > FYI these generated files have been rem

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
cal PCollection being written to this IO, give the code of a hypothetical DoFn implementing the write using your API, and explain what you'd expect to happen at runtime. I'm going to re-engage in this thread after such an example is given. On Sat, Nov 18, 2017, 5:00 AM Romain Manni-Bucau <rmannibu...@g

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
early a subset of the whole bundle and avoid to reprocess it if it fails later. So to summarize I see 2 outcomes: 1. impl SDF in all runners 2. make the bundle size upper bounded - through a pipeline option - in all runners, not sure this one is doable everywhere since I mainly checked spark

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
rder replays. The implementation > strategy works in Dataflow but not in Flink which has a horizontal > checkpoint. KafkaIO checks for compatibility. > > On Wed, Nov 15, 2017 at 12:38 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> > wrote: > > > Hi guys, > > >

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
shouldCheckpoint(); IV. @Checkpointer Serializable getCheckpoint(); in the dofn per element Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 6:06 GMT+01:00 Raghu Angadi <rang...@google.com.invalid>: > How would you define it (rough API is fine)?. Wit

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
entry cost and usage of beam way further. In my mind it is exactly what jbatch/spring-batch uses but adapted to beam (stream in particular) case. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 6:49 GMT+01:00 Reuven Lax <re...@google.com.invalid>: >

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
hich can work I have to admit if SDF are implemented by runners. Is there a roadmap/status on that? Last time I checked SDF was a great API without support :(. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 7:25 GMT+01:00 Eugene Kirpichov <kirpic...@google.com.i

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
boolean shouldCheckpoint(); Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 11:04 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > And the control is given to the DoFn developer via annotations, right ? > > So, bundle would be "hid

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
batches? Beam doesnt everywhere so I guess it is not important - at least for my cases this statement is true. > > Reuven > > On Wed, Nov 15, 2017 at 5:58 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Overall goal is to ensure each 100 elements max, a &quo

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
ossible to use today. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 10:27 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > It's in the dev list archives, not sure if there's a doc yet. > > I'm not quite sure I understand what you mean by a "flus

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
point algorithm. Tend to think the first is easier. > 3. Agree to have a core PTransform for that. > > Regards > JB > > > On 11/15/2017 02:16 PM, Romain Manni-Bucau wrote: >> >> @Reuven: it looks like a good workaround >> @Ken: thks a lot for the link! >> &

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
get this kind of "composite" pattern in the beam core? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 14:12 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > In case the connection is not clear to folks on this thread, I

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Romain Manni-Bucau
ch modules, jenkins/travis have this ability since they support scripting - document how to setup a "fastBuild" profile in its settings.xml which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast development iterations Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: [VOTE] Release 2.2.0, release candidate #2

2017-11-03 Thread Romain Manni-Bucau
https://repository.apache.org/content/repositories/orgapachebeam-1022/org/apache/beam/beam-parent/2.2.0/beam-parent-2.2.0-source-release.zip doesn't look "corrupted" so it looks ok to only update the dist area and reuse the same staging repo Romain Manni-Bucau @rmannibucau | Blog

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Romain Manni-Bucau
gradle branch doesnt build for me (some rat issues) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Great ! > > What explain these difference ? I'm curious especially for the clean b

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
a "?" can probably be exclude as well if created by the build at some point. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Thanks for the update. I was swamped on some meeting

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
it (except time ;)) but it is always a bit annoying to git clone then not be able to build. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-09 18:07 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > Hmm, I have had good luck when following the Python quick st

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
2017-11-09 18:38 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> (this is another topic so we can maybe open another thread) issue is >> not much about python

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
The 1.3.5 file is when i installed the python dependencies manually to make the build passing (the pip command never passed on my computer and therefore the build always has been broken until i installed it manually - independently from the build tool). Romain Manni-Bucau @rmannibucau | Blog

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
vement > > over Maven or that you did not and you experienced build times that were > > equivalent to Maven? > > > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> > > wrote: > > > >> 2017-11-09 18:38 GMT+01:

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
ime criteria. Do you share it as well or prefer time over other criteria - which leads to other conclusions and options indeed and can make us not understanding each other? On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > I will try next week yes but

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
2017 00:10, "Lukasz Cwik" <lc...@google.com.invalid> a écrit : > I wouldn't mind merging this change in so I could setup those Gradle > Jenkins precommits. > > As per our contribution guidelines, any committer willing to sign off on > the PR? > > On Thu, Nov 9

Re: Apache Ignite as a distributed processing back-ends

2017-12-07 Thread Romain Manni-Bucau
Hi This sounds awesome to have an Ignite runner which could compete with hazelcast-jet. The entry point would be https://beam.apache.org/contribute/runner-guide/ IMHO. Being on Ignite cluster also opens a lot of doors - reusing the filesystem or distributed structures. Very exiting. Le 8 déc.

Re: How to cope with Maven transient network issues?

2017-12-01 Thread Romain Manni-Bucau
If you use wagon http - and not lightweigh - it should have retries by default. Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" a écrit : > Has this ever been brought up in Maven dev community? Perhaps they have > some suggestions. It sounds like a reasonable feature request. >

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Romain Manni-Bucau
Hi Even if not a commiter or even PMC, I'd like to mention a few points from an external eye: - Maven stays the most common build tool and easier one for any user. It means it is the best one to hope contributions IMHO. - Maven has incremental support but if there is any blocker the community is

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
work stay clean and actually portable and doesnt impact runners as poc done until today did. Works for me. On Thu, May 3, 2018 at 10:05 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi guys > > Since some time there are efforts to have a language portable support in >

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
nt run sometimes - a colleague hit that yesterday :(. What is a "beam cluster" - opposed to a spark or foink cluster? How would it work on windows servers? On Fri, May 4, 2018, 11:19 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > 2018-05-05 2:33 GMT+02:00 Andrew

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
ain, >> >> Docker, unlike selinux, solves a great number of tangible problems for us >> with IMO a relatively small tax. It does not have to be the only way. Some >> of the concerns you bring up along with possibilities were also discussed >> here: https://s.apache.org/be

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
t's entirely fine for some runners to use Jython, Graal, etc to provide a specialized offering similar to the direct runners, but it would be disjoint from portability IMO. On Fri, May 4, 2018 at 10:14 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 4 mai 2018 17:55, "L

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
beam cluster with the spark runner would include a spark cluster, plus > what's needed for portability, plus the beam sdk. > > > On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > > >> Le 5 mai 2018 08:43, "Reuven Lax&quo

Re: Graal instead of docker?

2018-05-09 Thread Romain Manni-Bucau
Le mer. 9 mai 2018 00:57, Henning Rohde a écrit : > There are indeed lots of possibilities for interesting docker alternatives > with different tradeoffs and capabilities, but in generally both the runner > as well as the SDK must support them for it to work. As mentioned,

Re: Beam SQL Improvements

2018-04-27 Thread Romain Manni-Bucau
few reasons which will make it not the best choice my opinion, > but I may be wrong. Can you put together a design doc or a prototype? > > Thank you, > Anton > > > On Thu, Apr 26, 2018 at 10:17 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> &g

Re: Graal instead of docker?

2018-05-12 Thread Romain Manni-Bucau
s outside >> the expected code of conduct. >> >> On Fri, May 11, 2018 at 11:48 AM Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> >>> Also beam community is java - dont answer it is python or go without >>> checking ;). Not sure

Re: Graal instead of docker?

2018-05-11 Thread Romain Manni-Bucau
Le mer. 9 mai 2018 17:41, Eugene Kirpichov <kirpic...@google.com> a écrit : > > > On Wed, May 9, 2018 at 1:08 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> >> >> Le mer. 9 mai 2018 00:57, Henning Rohde <hero...@google.com> a éc

Re: Graal instead of docker?

2018-05-11 Thread Romain Manni-Bucau
nst stopping current impl track, revert it and move to a higher level runner? > Andrew > > On Fri, May 11, 2018 at 8:53 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> >> >> Le mer. 9 mai 2018 17:41, Eugene Kirpichov <kirpic...@google.com> a &g

Re: Graal instead of docker?

2018-05-08 Thread Romain Manni-Bucau
Le mar. 8 mai 2018 10:16, Robert Bradshaw <rober...@google.com> a écrit : > On Sun, May 6, 2018 at 1:30 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > Wow, this mail should be on the website Robert, thanks for it > > > I still have a point

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
>>> >>> Please also keep in mind that execution of user code is only a small >>> part of the overall portability picture, and dependency on Docker is an >>> even smaller part of that (there is only 1 mention of the word "Docker" in >>> all of Beam

Re: Graal instead of docker?

2018-05-06 Thread Romain Manni-Bucau
erFnObjectInJava, > executing everything as if it were pure Java. In particular the overheads > of unnecessarily crossing the language boundaries many times in a single > fused graph are often prohibitive. > > Sorry for the long email, but hopefully this helps shed some light on (at > lea

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
buted runners that they must implement the full >>> Portability APIs to be considered Beam multi language compliant but they >>> can prefer for performance reasons to translate without the portability >>> APIs the Java to Java case. >>> >> >> >> This is

Beam high level directions (was "Graal instead of docker?")

2018-05-16 Thread Romain Manni-Bucau
ssume the constraints it brings. And finally it should be less intrusive in all its layers and try to add features more transversally when possible (and it is possible in a lot of cases). If you bring features for free with new releases, everybody wins, if you announce features and no runner support it,

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
aks. This is why it can require some way to not recreate it. A quick fix, if you are in bytebuddy already, can be to add it to setup/teardown pby, being more global would be nicer but is more challenging. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannib

Re: Beam SQL Improvements

2018-05-23 Thread Romain Manni-Bucau
once distributed you serialized the instances so kind of broke the lifecycle of the original instance and have no real release/close hook on them anymore right? Not sure we can do better than dofn/source embedded instances today. Le mer. 23 mai 2018 08:02, Romain Manni-Bucau <rmann

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
If so, how an IO can use as() with the type it expects? Doesnt it lead to have a tons of these modules at the end? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com>

Re: Beam SQL Improvements

2018-05-23 Thread Romain Manni-Bucau
saner this way, no? Also it needs to ensure all converters are present before running the pipeline probably, no implicit environment converter support is probably good to start to avoid late surprises. > My $0.01 > > Regards > JB > > On 23/05/2018 07:51, Romain Manni-Bucau wrote:

Re: Beam high level directions (was "Graal instead of docker?")

2018-05-17 Thread Romain Manni-Bucau
eases pipeline >> portability, especially for pipelines that are not in the same language as >> the runner-core, such as for Python and Go. >> >> It's not clear to me what runners would be doing in that scenario either. >> Do you have a proposal about where the interface bound

  1   2   3   4   5   6   >