Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

2018-10-14 Thread Romain Manni-Bucau
ktpub.com/application-development/java-ee-8-high-performance> Le dim. 14 oct. 2018 à 15:19, Reuven Lax a écrit : > What in Beam codebase is not ready, and how do we know that user code > doesn't have the same issue? > > On Sun, Oct 14, 2018 at 6:04 AM Romain Manni-Buc

Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

2018-10-14 Thread Romain Manni-Bucau
that so it is fine to update it normally. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin

Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

2018-10-14 Thread Romain Manni-Bucau
not with java>=8 AFAIK Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com

Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

2018-10-14 Thread Romain Manni-Bucau
No, only parameter types and return type is used to lookup methods. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: [DISCUSS] Beam public roadmap

2018-10-10 Thread Romain Manni-Bucau
What about a link in the menu. It should contain a list of features and estimate date with probable error (like "in 5 months +- 1 months) otherwise it does not bring much IMHO. Le mer. 10 oct. 2018 23:32, Kenneth Knowles a écrit : > Hi all, > > We made an attempt at putting together a sort of ro

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
Le mer. 10 oct. 2018 21:31, Robert Bradshaw a écrit : > > > On Wed, Oct 10, 2018, 4:56 PM Romain Manni-Bucau > wrote: > >> >> >> >> Le mer. 10 oct. 2018 à 14:59, Maximilian Michels a >> écrit : >> >>> Hi, >>> >>>

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

2018-10-10 Thread Romain Manni-Bucau
eam enhancement plans and assign them some fix versions before defining what support model of Beam can be. Just the 2 cts of an outsider. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
p their machines as well. Today the main blocker is that default "profile" (script) is not matching dev persona and therefore there is no real hope to have external contributions outside google related guys as mentionned by previous ficgures which is sad for a project promishing unification and wo

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
topic. Le mer. 10 oct. 2018 11:58, Robert Bradshaw a écrit : > On Wed, Oct 10, 2018 at 10:25 AM Romain Manni-Bucau > wrote: > >> On the split point: a mono-repo works for me as well. The main point is >> "N separate builds". >> >> On the portable thing

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
model/ > > > > website/ > > > > java/ > > > > go/ > > > > ... > > > > > > > > possibly even with their own build system (unified only > > through a > >

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
> possibly even with their own build system (unified only through a > > > top-level "build everything" script that descends into each subdir > and > > > runs the appropriate command). I'm not saying we should do this > (there > > > is value

Re: Splitting the repo

2018-10-10 Thread Romain Manni-Bucau
d probably even make separate releases out of this > > single repo (if we wanted, though given that our releases are time-based > > rather than feature-based, I don't see much advantage here). > > > > Also, there was the comment. > > > > On Wed, Oct 10, 201

Re: [DISCUSS] Gradle for the build ?

2018-10-09 Thread Romain Manni-Bucau
@Reuven: any idea what is missing? I don't expect it to be ready very quickly but having 2 repos does not hurt that much if both are working better than a single one so can be worth a try maybe? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> |

Re: [DISCUSS] Gradle for the build ?

2018-10-09 Thread Romain Manni-Bucau
se incremental build cause you want to have a fully reliable build. So at the end the incremental build support is not that significative for end users and contibutors. The cost of not having the IDE support, however, is just a blocker. So my 2cts would be to stop trying to be good theorically

Re: [DISCUSS] Gradle for the build ?

2018-10-09 Thread Romain Manni-Bucau
For me the vendoring issue is ok cause it should belong to another shade loduke released with beam when needed. It is not an uncommon practise. Now the lack of IDE integration for tests/debug (using gradle runner is a workaround and still hurts by its slowness compared to native run) is a clear sh

Re: [DISCUSS] Gradle for the build ?

2018-10-09 Thread Romain Manni-Bucau
state for a community driven project. Let's fix it! Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https:/

Re: Java > 8 support

2018-10-06 Thread Romain Manni-Bucau
@Reuven: bytebuddy by itself no but the way beam tries to inject the proxy class is. There are other strategies you can use in bytebuddy which work. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: Java > 8 support

2018-10-06 Thread Romain Manni-Bucau
> > ** • *Google, Inc.* > > * • arifka...@google.com <mailto:arifka...@google.com> > > > > > > > > > > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau > <mailto:rmannibu...@gmail.com>> wrote: > > > > Hi Arif, > >

Re: Java > 8 support

2018-10-05 Thread Romain Manni-Bucau
want to use modules (but there is no real reason to do that yet in java). Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: Does anyone have a strong intelliJ setup?

2018-10-04 Thread Romain Manni-Bucau
wledge about the setup (like for validates runners which are not "just tests") and there is no trivial way to make the IDE aware of it until you generate the IDE files (.idea) > On 01.10.18 23:32, Romain Manni-Bucau wrote: > > Personally i drop all caches - idea + ivy + mav

Re: Does anyone have a strong intelliJ setup?

2018-10-01 Thread Romain Manni-Bucau
Personally i drop all caches - idea + ivy + maven beam folder, build in console skipping test execution - important cause idea is not able to import the project without a correctly ran gradle setup and a failure can corrupt later imports, then I kill gradle daemon and finally import beam in idea us

Re: [VOTE] Release 2.7.0, release candidate #2

2018-09-25 Thread Romain Manni-Bucau
+1 tested spark local and a 1 node cluster, direct runner and java test tools Le mar. 25 sept. 2018 13:08, Łukasz Gajowy a écrit : > +1 > > I ran the Java - Quickstart on core runners (local) and Dataflow > Quickstarts (on Dataflow instance) either way. Looks good. I wanted to > learn how easy i

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-21 Thread Romain Manni-Bucau
ou have any progress on this? One thing I >>>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven >>>>>>> Central, so >>>>>>> referring to it by version is not expected to work (and it looks like >>>&g

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-19 Thread Romain Manni-Bucau
x27;t test userClassPathFirst option of spark but it can be an interesting thing to enable in beam runner. However it is still very confusing to have it not running just upgrading beam version and the spark error is very hard to understand. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau>

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-18 Thread Romain Manni-Bucau
gt; > mailto:aromanenko@gmail.com>> wrote: > > > > Perhaps it could help, but I run simple WordCount (built with > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine > > for me. > > > >> On 14 Sep

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-17 Thread Romain Manni-Bucau
@Charles: just "mvn verify" (it is an integration-test) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | Lin

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-17 Thread Romain Manni-Bucau
v2.7.0. I dont have much time this week to check out this particular issue but hopefully next one should be more doable if the issue is still pending. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: Donating the Dataflow Worker code to Apache Beam

2018-09-14 Thread Romain Manni-Bucau
Le ven. 14 sept. 2018 à 09:48, Robert Bradshaw a écrit : > On Fri, Sep 14, 2018 at 8:00 AM Romain Manni-Bucau > wrote: > >> Well IBM runner is outside Beam for instance so this is not really a >> point IMHO. >> >> My view is simple: >> 1. does this module

Re: Donating the Dataflow Worker code to Apache Beam

2018-09-13 Thread Romain Manni-Bucau
ke it doable and not add a dead module to beam tree. Am I missing anything? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau

Re: Donating the Dataflow Worker code to Apache Beam

2018-09-13 Thread Romain Manni-Bucau
means Google should find another way to manage this dependency if this is the case IMHO. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-13 Thread Romain Manni-Bucau
ms to be a regression since 2.6 works OOTB. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.lin

Re: Donating the Dataflow Worker code to Apache Beam

2018-09-13 Thread Romain Manni-Bucau
If usable by itself without google karma (can you use a worker without dataflow itself?) it sounds awesome otherwise it sounds weird IMHO. Le jeu. 13 sept. 2018 21:36, Kai Jiang a écrit : > +1 (non googler) > > big help for transparency and for future runners. > > Best, > Kai > > On Thu, Sep 13,

Re: [Discuss] Upgrade story for Beam's execution engines

2018-09-12 Thread Romain Manni-Bucau
Hi guys, Isnt the issue "only" that beam has this code instead of engines? Assuming beam runner facing api is stable - which must be the case anyway - and that each engine has its integration (flink-beam instead of beam-runners-flink), then this issue disappears by construction. It also has the

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-11 Thread Romain Manni-Bucau
ng non beam issues with such a delivery which is a blocker to upgrade. So beam alone is ok but if you add anything - and since you will likely for any pipeline - then your app is no more in a workable state while shades are a recommended solution. > On Tue, Sep 11, 2018 at 2:48 PM Romain

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-11 Thread Romain Manni-Bucau
just by a quick visual check. It requires you to add tooling on top of it which is not really good overall. Wonder if it wouldn't be better to revert that if it can't be completed short term and reapplied when possible (probably using a working branch). Romain Manni-Bucau @rmannibucau

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-11 Thread Romain Manni-Bucau
like the new shading policy impl was merged a bit too fast ;) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-11 Thread Romain Manni-Bucau
samples. Let me try to reproduce and I >> will create the Jira. >> >> Regards >> JB >> >> On 11/09/2018 11:44, Romain Manni-Bucau wrote: >> > -1, seems spark integration is broken (tested with spark 2.3.1 and >> 2.2.1): >> >

Re: [VOTE] Release 2.7.0, release candidate #1

2018-09-11 Thread Romain Manni-Bucau
workarounds can be put in place so +1 to fix it as well if possible. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: Cleanup resources on pipeline cancelation

2018-08-02 Thread Romain Manni-Bucau
fires when the pipeline shuts down). I >> believe that SDF support is being added to the portability layer now, so >> eventually all portable runners will support it, and maybe we can live with >> the status quo until then. >> >> On Wed, Aug 1, 2018 at 9:59 PM Romain

Re: Cleanup resources on pipeline cancelation

2018-08-01 Thread Romain Manni-Bucau
g time). > > The answer might be that Beam does not have a good lifecycle story here, > and something needs to be built. > > Reuven > > On Tue, Jul 31, 2018 at 10:04 PM Romain Manni-Bucau > wrote: > >> Hi Andrew, >> >> IIRC sources should clean up th

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

2018-08-01 Thread Romain Manni-Bucau
Hi Pablo, +1, tested on my apps and libs and words after some fixed due to some breaking changes in ArgProvider - but guess it is not "public" to need to be reported. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net

Re: Cleanup resources on pipeline cancelation

2018-07-31 Thread Romain Manni-Bucau
Hi Andrew, IIRC sources should clean up their resources per method since they dont have a better lifecycle. Readers can create anything longer and release it at close time. Le mer. 1 août 2018 00:31, Andrew Pilloud a écrit : > Some of our IOs create external resources that need to be cleaned u

Re: [ANNOUNCEMENT] Nexmark included to the CI

2018-07-12 Thread Romain Manni-Bucau
Awesome Etienne, this is really important for the (user) community to have that visibility since it is one of the most important aspect of the Beam's quality, kudo! Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/>

bad logger import?

2018-06-26 Thread Romain Manni-Bucau
Hi guys, answering a question on slack i realized flink ExecutableStageDoFnOperator.java uses JUL instead of SLF4J, not sure it is intended so thought I would mention it here. Side note: archetype and some test code does as well but it is less an issue. Romain Manni-Bucau @rmannibucau <ht

Re: FYI on Slack Channels

2018-06-26 Thread Romain Manni-Bucau
+1 sounds very good side note: any channel must invite @asfarchivebot, I did it for the ones before "etc" but if you add others please ensure it is done Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-17 Thread Romain Manni-Bucau
+1 (same tests than for take #1) Le dim. 17 juin 2018 07:18, Jean-Baptiste Onofré a écrit : > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 2.5.0, as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific com

Re: [DISCUSS] Releasing Beam in the presence of emergencies

2018-06-17 Thread Romain Manni-Bucau
Hi guys If you take back this thread [1], then beam will only support two branches and no others. On the shortcut release cycle it is not really desired at apache and not the normal process so the 3 days vote is required but not a blocker IMHO. If really an issue the merge can be done quickly on

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

2018-06-13 Thread Romain Manni-Bucau
eeds to find back or create the right classloader and when the last instance using this classloader disappear it destroys it. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> |

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

2018-06-13 Thread Romain Manni-Bucau
/org/talend/sdk/component/runtime/manager/asm/ProxyGenerator.java#L118 . > > On Wed, Jun 6, 2018 at 9:56 PM Romain Manni-Bucau > wrote: > >> Note sure the example is atomic enough but in >> https://github.com/Talend/component-runtime/blob/master/component-runtime-

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-07 Thread Romain Manni-Bucau
+1 (non-binding), mainstream usage is not broken by the pom changes and runtime has no known regression compared to the 2.4.0 (side note: kudo to JB for this build tool change release, I know how it can hurt ;)) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

2018-06-06 Thread Romain Manni-Bucau
33, Lukasz Cwik a écrit : > Romain, can you point to an example of a global singleton registry that > does this right for class loading (it may allow people to work towards such > an effort)? > > On Tue, Jun 5, 2018 at 10:06 PM Romain Manni-Bucau > wrote: > >> It is actuall

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-06-05 Thread Romain Manni-Bucau
Also maybe deactivate the daemon (--no-daemon) since its cache can get corrupted ~easily. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://g

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

2018-06-05 Thread Romain Manni-Bucau
It is actually very localised in runner code where beam should reset the classloader when the deserialization happens and then the runner owns the classloader all the way in evaluators. If IO change the classloader they must indeed handle it too and patch the deserialization too. Here again (we m

Re: Beam SQL Improvements

2018-06-04 Thread Romain Manni-Bucau
ck. I'd >>> like to understand your concerns. Also keep in mind that I'm tagging all >>> these classes as Experimental for now, so we can definitely change these >>> interfaces around if we decide they are not the best ones. >>> >>> Reuven >>

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
once distributed you serialized the instances so kind of broke the lifecycle of the original instance and have no real release/close hook on them anymore right? Not sure we can do better than dofn/source embedded instances today. Le mer. 23 mai 2018 08:02, Romain Manni-Bucau a écrit : > >

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
Also it needs to ensure all converters are present before running the pipeline probably, no implicit environment converter support is probably good to start to avoid late surprises. > My $0.01 > > Regards > JB > > On 23/05/2018 07:51, Romain Manni-Bucau wrote: > > How does

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
. This is why it can require some way to not recreate it. A quick fix, if you are in bytebuddy already, can be to add it to setup/teardown pby, being more global would be nicer but is more challenging. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannib

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
nd here)? If so, how an IO can use as() with the type it expects? Doesnt it lead to have a tons of these modules at the end? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpres

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
écrit : > We can do even better btw. Building a SchemaRegistry where automatic > conversions can be registered between schema and Java data types. With this > the user won't even need a DoFn to do the conversion. > > On Tue, May 22, 2018, 10:13 AM Romain Manni-Bucau &

Re: Beam SQL Improvements

2018-05-22 Thread Romain Manni-Bucau
for now, at least for IO and some user transforms. Wdyt? Le ven. 27 avr. 2018 18:36, Romain Manni-Bucau a écrit : > Can give it a try end of may, sure. (holidays and work constraints will > make it hard before). > > Le 27 avr. 2018 18:26, "Anton Kedin" a écrit : >

Re: Beam high level directions (was "Graal instead of docker?")

2018-05-17 Thread Romain Manni-Bucau
ecially for pipelines that are not in the same language as >> the runner-core, such as for Python and Go. >> >> It's not clear to me what runners would be doing in that scenario either. >> Do you have a proposal about where the interface boundaries would be? >> &g

Re: Beam high level directions (was "Graal instead of docker?")

2018-05-16 Thread Romain Manni-Bucau
forts. >> >> > Technically, there is plenty to agree with. I think as you learn about >> Beam you will find that many of your suggestions are already handled in >> some way. You may also continue to learn sometimes about the specific >> reasons things are done in a differen

Beam high level directions (was "Graal instead of docker?")

2018-05-16 Thread Romain Manni-Bucau
ly N api). it must embrace the environments it runs in and assume the constraints it brings. And finally it should be less intrusive in all its layers and try to add features more transversally when possible (and it is possible in a lot of cases). If you bring features for free with new releases, everybo

Re: Graal instead of docker?

2018-05-12 Thread Romain Manni-Bucau
that time to process things and potentially advise/steer the > conversation. > > On Fri, May 11, 2018 at 12:42 PM, Kenneth Knowles wrote: > >> Romain, >> >> You probably did not mean to, but I think this message crosses outside >> the expected code of con

Re: Graal instead of docker?

2018-05-11 Thread Romain Manni-Bucau
urrent impl track, revert it and move to a higher level runner? > Andrew > > On Fri, May 11, 2018 at 8:53 AM Romain Manni-Bucau > wrote: > >> >> >> Le mer. 9 mai 2018 17:41, Eugene Kirpichov a >> écrit : >> >>> >>> >>> On

Re: Graal instead of docker?

2018-05-11 Thread Romain Manni-Bucau
Le mer. 9 mai 2018 17:41, Eugene Kirpichov a écrit : > > > On Wed, May 9, 2018 at 1:08 AM Romain Manni-Bucau > wrote: > >> >> >> Le mer. 9 mai 2018 00:57, Henning Rohde a écrit : >> >>> There are indeed lots of possibilities for interesting dock

Re: Graal instead of docker?

2018-05-09 Thread Romain Manni-Bucau
Le mer. 9 mai 2018 00:57, Henning Rohde a écrit : > There are indeed lots of possibilities for interesting docker alternatives > with different tradeoffs and capabilities, but in generally both the runner > as well as the SDK must support them for it to work. As mentioned, docker > (as used in th

Re: Graal instead of docker?

2018-05-08 Thread Romain Manni-Bucau
Le mar. 8 mai 2018 10:16, Robert Bradshaw a écrit : > On Sun, May 6, 2018 at 1:30 AM Romain Manni-Bucau > wrote: > > > Wow, this mail should be on the website Robert, thanks for it > > > I still have a point to try to understand better: my view is that once > subm

Re: Graal instead of docker?

2018-05-06 Thread Romain Manni-Bucau
rFnObjectInJava, > executing everything as if it were pure Java. In particular the overheads > of unnecessarily crossing the language boundaries many times in a single > fused graph are often prohibitive. > > Sorry for the long email, but hopefully this helps shed some light on (at >

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
it would be reasonable to keep an >>> eye on it as a potential future lightweight alternative to containers for >>> pipelines where Graal's level of support is sufficient for this particular >>> pipeline. >>> >>> Please also keep in mind that ex

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
y to frame this: if Beam was originally written in Go, we > would be having a different discussion. A pipeline written entirely in java > wouldn't be possible, so instead to enable Hazelcast, we would have to be > able to run the java from portability without running the container. &g

Re: Graal instead of docker?

2018-05-05 Thread Romain Manni-Bucau
include a spark cluster, plus > what's needed for portability, plus the beam sdk. > > > On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau > wrote: > > > > >> Le 5 mai 2018 08:43, "Reuven Lax" a écrit : > > >> I don't believe we

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
a colleague hit that yesterday :(. What is a "beam cluster" - opposed to a spark or foink cluster? How would it work on windows servers? On Fri, May 4, 2018, 11:19 PM Romain Manni-Bucau wrote: > > > 2018-05-05 2:33 GMT+02:00 Andrew Pilloud : > >> What docker

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
es a great number of tangible problems for us >> with IMO a relatively small tax. It does not have to be the only way. Some >> of the concerns you bring up along with possibilities were also discussed >> here: https://s.apache.org/beam-fn-api-container-contract. I encourage >&g

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
rely fine for some runners to use Jython, Graal, etc to provide a specialized offering similar to the direct runners, but it would be disjoint from portability IMO. On Fri, May 4, 2018 at 10:14 AM Romain Manni-Bucau wrote: > > > Le 4 mai 2018 17:55, "Lukasz Cwik" a écrit : >

Re: Graal instead of docker?

2018-05-04 Thread Romain Manni-Bucau
an and actually portable and doesnt impact runners as poc done until today did. Works for me. On Thu, May 3, 2018 at 10:05 PM Romain Manni-Bucau wrote: > Hi guys > > Since some time there are efforts to have a language portable support in > beam but I cant really find a case it "wor

Graal instead of docker?

2018-05-03 Thread Romain Manni-Bucau
Hi guys Since some time there are efforts to have a language portable support in beam but I cant really find a case it "works" being based on docker except for some vendor specific infra. Current solution: 1. Is runner intrusive (which is bad for beam and prevents adoption of big data vendors) 2

Re: Gradle Status: Migrated!

2018-05-01 Thread Romain Manni-Bucau
Hi Scott While https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057 is open, gradle is a concurrent of maven but maven must stay the default build tool cause gradle breaks users. Le 1 mai 2018 01:59, "Scott Wegner" a écrit : > Many many of you have been hacking diligently on t

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-30 Thread Romain Manni-Bucau
Le 30 avr. 2018 19:39, "Jean-Baptiste Onofré" a écrit : Hi guys, now that I'm back from vacations, I bring back 2.5.0 release on the table ;) This is also related to the current status of build (Maven/Gradle). FYI, I gonna start the Jira triage tomorrow and I launched couple of build on my mac

Re: Beam SQL Improvements

2018-04-27 Thread Romain Manni-Bucau
e it not the best choice my opinion, > but I may be wrong. Can you put together a design doc or a prototype? > > Thank you, > Anton > > > On Thu, Apr 26, 2018 at 10:17 PM Romain Manni-Bucau > wrote: > >> >> >> Le 26 avr. 2018 23:13, "Anton Kedin&qu

Re: Beam SQL Improvements

2018-04-26 Thread Romain Manni-Bucau
the point of Row/Schema is to avoid it and define everything in Beam model, which can be extended, mapped to JSON, Avro, BigQuery Schemas, custom binary format etc., with same semantics across beam SDKs. This is what jsonp would allow with the benefit of a natural pojo support through jsonb.

Re: Beam SQL Improvements

2018-04-26 Thread Romain Manni-Bucau
t;> could be another one. >> >> Regards >> JB >> Le 26 avr. 2018, à 12:08, Romain Manni-Bucau a >> écrit: >>> >>> Hmm, >>> >>> avro has still the pitfalls to have an uncontrolled stack which brings >>> way too much

Re: [BEAM-4088] Test isolation differences in Gradle

2018-04-26 Thread Romain Manni-Bucau
+1, thks Etienne for the hard work and to have spent time on it even if gradle was making it more complicated, really appreciated. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpres

Re: DoFn thread constraint.

2018-04-26 Thread Romain Manni-Bucau
Do you want to give it a try doing a PR on https://github.com/apache/beam-site? Better to fix an issue by the person getting the issue in general in that area ;) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: DoFn thread constraint.

2018-04-26 Thread Romain Manni-Bucau
Hi John, The fact a runner caches a fn per thread is an internal implementation detail but a fn will only be activated by one thread max at a time (like stateless or any object pool). This means your fn can fail. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: Beam SQL Improvements

2018-04-26 Thread Romain Manni-Bucau
Hmm, avro has still the pitfalls to have an uncontrolled stack which brings way too much dependencies to be part of any API, this is why I proposed a JSON-P based API (JsonObject) with a custom beam entry for some metadata (headers "à la Camel"). Romain Manni-Bucau @rmannibu

Re: Beam parent SNAPSHOTs are not being built

2018-04-25 Thread Romain Manni-Bucau
s between modules and determine the impact of a change ("what is the common part" ~ parent+bom, "what does it impact if we change that dep"). It can be key for beam to keep that to ensure upstream projects can determine some of their impacts statically through mavenrepo tools. >

Re: Which features of Dropwizard do you wish to use most with Beam metrics?

2018-04-25 Thread Romain Manni-Bucau
Hi Alex When i built sirona I started this way: counters, gauges, validations (healthcheck). It works not bad and enables distributed aggregations but when you think about it, it is a very limited set of what you want as a user. Quickly you will want exceptions and audit event etc.. This all lea

Re: Beam parent SNAPSHOTs are not being built

2018-04-25 Thread Romain Manni-Bucau
Hmm, gradle move not supposed to break any user and there are other parents usages, as mentionned previously, so not sure beam can really bypass it. Maven discussed consumer vs build pom since years for that reason ;). Le 26 avr. 2018 01:59, "Eric Beach" a écrit : > Thanks Scott and Ben. > > Thi

Re: Beam to move to JSON-B

2018-04-25 Thread Romain Manni-Bucau
beam.apache.org%3E or work on > https://issues.apache.org/jira/browse/BEAM-3608 and similar to improve > things. Should be very easy now. I didn't work on it because work on poms > is throwaway. Now is a good time to pick it up. > > Kenn > > On Wed, Apr 25, 2018 at 12:10 AM

Beam to move to JSON-B

2018-04-25 Thread Romain Manni-Bucau
s up the libs in both. This is critical to let ops handle vulnerabilities properly and without impacting the whole company ecosystem in a lot of cases. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http

Re: slack @the-asf?

2018-04-20 Thread Romain Manni-Bucau
s. >> >> Sorry for the delay, I was busy. >> >> Regards >> JB >> Le 18 mars 2018, à 19:57, Innocent Djiofack a >> écrit: >>> >>> Thanks Romain! >>> >>> 2018-03-18 14:17 GMT-04:00 Romain Manni-Bucau : >>> >>&

Re: [BEAM-4088] Test isolation differences in Gradle

2018-04-18 Thread Romain Manni-Bucau
No, otherwise it wouldnt pass in idea and maven. Le 19 avr. 2018 00:57, "Kenneth Knowles" a écrit : Does the test think a thread leaked when no thread leaked? On Wed, Apr 18, 2018 at 1:51 PM Romain Manni-Bucau wrote: > The leaking threadq are already dumped but we dont track w

Re: [BEAM-4088] Test isolation differences in Gradle

2018-04-18 Thread Romain Manni-Bucau
gt; dynamic analysis since it is not very practical to unit test for leaks > (thread, memory, file handle, whatever) and other global correctness > criteria. > > Kenn > > On Wed, Apr 18, 2018 at 10:13 AM Romain Manni-Bucau > wrote: > >> >> >> Le 18 avr. 2

Re: [PROPOSAL] JIRA notification recipient changes

2018-04-18 Thread Romain Manni-Bucau
commits@ or dev@ are common choices. Others are generally redundant or means it is ignored by everybody which is fine but means you just drop notif from jira instead of redirecting them ;) Just my 2 cts Le 18 avr. 2018 20:55, "Kenneth Knowles" a écrit : > Annoyingly, it appears that the reporte

Re: [BEAM-4088] Test isolation differences in Gradle

2018-04-18 Thread Romain Manni-Bucau
://docs.gradle.org/current/dsl/org.gradle.api. tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:maxParallelForks [5] https://docs.gradle.org/current/dsl/org.gradle.api. tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery On Thu, Apr 12, 2018 at 10:03 AM Romain Manni-Bucau

Re: Gradle Status [April 6]

2018-04-15 Thread Romain Manni-Bucau
eful behavior. That > said, I'm not sure how it interacts with Idea. > > Kenn > > On Sun, Apr 15, 2018 at 12:23 AM Romain Manni-Bucau > wrote: > >> If you dont the diff is empty and gradle runs nothing, no? Saw it with >> gradle 4.6 >> >> Le 15

Re: Gradle Status [April 6]

2018-04-15 Thread Romain Manni-Bucau
If you dont the diff is empty and gradle runs nothing, no? Saw it with gradle 4.6 Le 15 avr. 2018 00:49, "Kenneth Knowles" a écrit : > You shouldn't do :module:cleanTest. If that is necessary that's a major > bug in the build. > > Kenn > > On Fri, Apr 1

Re: Gradle Status [April 6]

2018-04-13 Thread Romain Manni-Bucau
8 at 11:14 AM Reuven Lax wrote: > >> Is there a Jira for this 3 second delay? Also you're initial complaint >> was not about the 3 second delay, so it wasn't clear that's what you were >> complaining about. >> >> Reuven >> >> On Fri, Apr

Re: Gradle Status [April 6]

2018-04-12 Thread Romain Manni-Bucau
Reuven On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira wrote: > Sorry Romain, I'm not quite sure what you're asking. Can you clarify? > > On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau > wrote: > >> Well you are the only one to not have the drawbacks to us

  1   2   3   4   5   6   >