Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

ktpub.com/application-development/java-ee-8-high-performance> Le dim. 14 oct. 2018 à 15:19, Reuven Lax a écrit : > What in Beam codebase is not ready, and how do we know that user code > doesn't have the same issue? > > On Sun, Oct 14, 2018 at 6:04 AM Romain Manni-Bucau &

Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

on that so it is fine to update it normally. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin

Re: Can we allow SimpleFunction and SerializableFunction to throw Exception?

not with java>=8 AFAIK Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com

Re: [DISCUSS] Beam public roadmap

What about a link in the menu. It should contain a list of features and estimate date with probable error (like "in 5 months +- 1 months) otherwise it does not bring much IMHO. Le mer. 10 oct. 2018 23:32, Kenneth Knowles a écrit : > Hi all, > > We made an attempt at putting together a sort of

Re: Splitting the repo

Le mer. 10 oct. 2018 21:31, Robert Bradshaw a écrit : > > > On Wed, Oct 10, 2018, 4:56 PM Romain Manni-Bucau > wrote: > >> >> >> >> Le mer. 10 oct. 2018 à 14:59, Maximilian Michels a >> écrit : >> >>> Hi, >>> >>>

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

nhancement plans and assign them some fix versions before defining what support model of Beam can be. Just the 2 cts of an outsider. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordp

Re: Splitting the repo

their machines as well. Today the main blocker is that default "profile" (script) is not matching dev persona and therefore there is no real hope to have external contributions outside google related guys as mentionned by previous ficgures which is sad for a project promishing unifica

Re: Splitting the repo

as it for that topic. Le mer. 10 oct. 2018 11:58, Robert Bradshaw a écrit : > On Wed, Oct 10, 2018 at 10:25 AM Romain Manni-Bucau > wrote: > >> On the split point: a mono-repo works for me as well. The main point is >> "N separate builds". >> >> On the portable thing

Re: Splitting the repo

website/ > > > > java/ > > > > go/ > > > > ... > > > > > > > > possibly even with their own build system (unified only > > through a > > > > top-level "bu

Re: Splitting the repo

own build system (unified only through a > > > top-level "build everything" script that descends into each subdir > and > > > runs the appropriate command). I'm not saying we should do this > (there > > > is value in having a single consistent build sys

Re: Splitting the repo

ses out of this > > single repo (if we wanted, though given that our releases are time-based > > rather than feature-based, I don't see much advantage here). > > > > Also, there was the comment. > > > > On Wed, Oct 10, 2018 at 7:35 AM Romain Manni-Bucau > >

Re: [DISCUSS] Gradle for the build ?

@Reuven: any idea what is missing? I don't expect it to be ready very quickly but having 2 repos does not hurt that much if both are working better than a single one so can be worth a try maybe? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: [DISCUSS] Gradle for the build ?

nt to have a fully reliable build. So at the end the incremental build support is not that significative for end users and contibutors. The cost of not having the IDE support, however, is just a blocker. So my 2cts would be to stop trying to be good theorically at the cost of loosing the users

Re: [DISCUSS] Gradle for the build ?

For me the vendoring issue is ok cause it should belong to another shade loduke released with beam when needed. It is not an uncommon practise. Now the lack of IDE integration for tests/debug (using gradle runner is a workaround and still hurts by its slowness compared to native run) is a clear

Re: Java > 8 support

@Reuven: bytebuddy by itself no but the way beam tries to inject the proxy class is. There are other strategies you can use in bytebuddy which work. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: Java > 8 support

> > ** • *Google, Inc.* > > * • arifka...@google.com <mailto:arifka...@google.com> > > > > > > > > > > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau > <mailto:rmannibu...@gmail.com>> wrote: > > > > Hi Arif, >

Re: Java > 8 support

want to use modules (but there is no real reason to do that yet in java). Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: Does anyone have a strong intelliJ setup?

or validates runners which are not "just tests") and there is no trivial way to make the IDE aware of it until you generate the IDE files (.idea) > On 01.10.18 23:32, Romain Manni-Bucau wrote: > > Personally i drop all caches - idea + ivy + maven beam folder, build in >

Re: Does anyone have a strong intelliJ setup?

Personally i drop all caches - idea + ivy + maven beam folder, build in console skipping test execution - important cause idea is not able to import the project without a correctly ran gradle setup and a failure can corrupt later imports, then I kill gradle daemon and finally import beam in idea

Re: [VOTE] Release 2.7.0, release candidate #2

+1 tested spark local and a 1 node cluster, direct runner and java test tools Le mar. 25 sept. 2018 13:08, Łukasz Gajowy a écrit : > +1 > > I ran the Java - Quickstart on core runners (local) and Dataflow > Quickstarts (on Dataflow instance) either way. Looks good. I wanted to > learn how easy

Re: [VOTE] Release 2.7.0, release candidate #1

on this? One thing I >>>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven >>>>>>> Central, so >>>>>>> referring to it by version is not expected to work (and it looks like >>>>>>> this >>&g

Re: [VOTE] Release 2.7.0, release candidate #1

userClassPathFirst option of spark but it can be an interesting thing to enable in beam runner. However it is still very confusing to have it not running just upgrading beam version and the spark error is very hard to understand. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: [VOTE] Release 2.7.0, release candidate #1

ilto:aromanenko@gmail.com>> wrote: > > > > Perhaps it could help, but I run simple WordCount (built with > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine > > for me. > > > >> On 14 Sep 2018,

Re: [VOTE] Release 2.7.0, release candidate #1

@Charles: just "mvn verify" (it is an integration-test) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |

Re: [VOTE] Release 2.7.0, release candidate #1

to be in v2.7.0. I dont have much time this week to check out this particular issue but hopefully next one should be more doable if the issue is still pending. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: Donating the Dataflow Worker code to Apache Beam

Le ven. 14 sept. 2018 à 09:48, Robert Bradshaw a écrit : > On Fri, Sep 14, 2018 at 8:00 AM Romain Manni-Bucau > wrote: > >> Well IBM runner is outside Beam for instance so this is not really a >> point IMHO. >> >> My view is simple: >> 1. does this module

Re: Donating the Dataflow Worker code to Apache Beam

it doable and not add a dead module to beam tree. Am I missing anything? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |

Re: Donating the Dataflow Worker code to Apache Beam

means Google should find another way to manage this dependency if this is the case IMHO. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://

Re: [VOTE] Release 2.7.0, release candidate #1

to be a regression since 2.6 works OOTB. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin

Re: Donating the Dataflow Worker code to Apache Beam

If usable by itself without google karma (can you use a worker without dataflow itself?) it sounds awesome otherwise it sounds weird IMHO. Le jeu. 13 sept. 2018 21:36, Kai Jiang a écrit : > +1 (non googler) > > big help for transparency and for future runners. > > Best, > Kai > > On Thu, Sep

Re: [Discuss] Upgrade story for Beam's execution engines

Hi guys, Isnt the issue "only" that beam has this code instead of engines? Assuming beam runner facing api is stable - which must be the case anyway - and that each engine has its integration (flink-beam instead of beam-runners-flink), then this issue disappears by construction. It also has the

Re: [VOTE] Release 2.7.0, release candidate #1

ues with such a delivery which is a blocker to upgrade. So beam alone is ok but if you add anything - and since you will likely for any pipeline - then your app is no more in a workable state while shades are a recommended solution. > On Tue, Sep 11, 2018 at 2:48 PM Romain Manni-Bucau &g

Re: [VOTE] Release 2.7.0, release candidate #1

done just by a quick visual check. It requires you to add tooling on top of it which is not really good overall. Wonder if it wouldn't be better to revert that if it can't be completed short term and reapplied when possible (probably using a working branch). Romain Manni-Bucau @rmannibucau <https:

Re: [VOTE] Release 2.7.0, release candidate #1

like the new shading policy impl was merged a bit too fast ;) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn &

Re: [VOTE] Release 2.7.0, release candidate #1

mples. Let me try to reproduce and I >> will create the Jira. >> >> Regards >> JB >> >> On 11/09/2018 11:44, Romain Manni-Bucau wrote: >> > -1, seems spark integration is broken (tested with spark 2.3.1 and >> 2.2.1): >> > >>

Re: [VOTE] Release 2.7.0, release candidate #1

if workarounds can be put in place so +1 to fix it as well if possible. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: Cleanup resources on pipeline cancelation

he pipeline shuts down). I >> believe that SDF support is being added to the portability layer now, so >> eventually all portable runners will support it, and maybe we can live with >> the status quo until then. >> >> On Wed, Aug 1, 2018 at 9:59 PM Romain Manni-Buca

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

Hi Pablo, +1, tested on my apps and libs and words after some fixed due to some breaking changes in ArgProvider - but guess it is not "public" to need to be reported. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net

Re: Cleanup resources on pipeline cancelation

Hi Andrew, IIRC sources should clean up their resources per method since they dont have a better lifecycle. Readers can create anything longer and release it at close time. Le mer. 1 août 2018 00:31, Andrew Pilloud a écrit : > Some of our IOs create external resources that need to be cleaned

Re: [ANNOUNCEMENT] Nexmark included to the CI

Awesome Etienne, this is really important for the (user) community to have that visibility since it is one of the most important aspect of the Beam's quality, kudo! Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

bad logger import?

Hi guys, answering a question on slack i realized flink ExecutableStageDoFnOperator.java uses JUL instead of SLF4J, not sure it is intended so thought I would mention it here. Side note: archetype and some test code does as well but it is less an issue. Romain Manni-Bucau @rmannibucau <ht

Re: FYI on Slack Channels

+1 sounds very good side note: any channel must invite @asfarchivebot, I did it for the ones before "etc" but if you add others please ensure it is done Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

+1 (same tests than for take #1) Le dim. 17 juin 2018 07:18, Jean-Baptiste Onofré a écrit : > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 2.5.0, as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific

Re: [DISCUSS] Releasing Beam in the presence of emergencies

Hi guys If you take back this thread [1], then beam will only support two branches and no others. On the shortcut release cycle it is not really desired at apache and not the normal process so the 3 days vote is required but not a blocker IMHO. If really an issue the merge can be done quickly on

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

find back or create the right classloader and when the last instance using this classloader disappear it destroys it. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

/component/runtime/manager/asm/ProxyGenerator.java#L118 . > > On Wed, Jun 6, 2018 at 9:56 PM Romain Manni-Bucau > wrote: > >> Note sure the example is atomic enough but in >> https://github.com/Talend/component-runtime/blob/master/component-runtime-manager/src/mai

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

+1 (non-binding), mainstream usage is not broken by the pom changes and runtime has no known regression compared to the 2.4.0 (side note: kudo to JB for this build tool change release, I know how it can hurt ;)) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

33, Lukasz Cwik a écrit : > Romain, can you point to an example of a global singleton registry that > does this right for class loading (it may allow people to work towards such > an effort)? > > On Tue, Jun 5, 2018 at 10:06 PM Romain Manni-Bucau > wrote: > >> It is actuall

Re: [PROPOSAL] Preparing 2.5.0 release next week

Also maybe deactivate the daemon (--no-daemon) since its cache can get corrupted ~easily. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://g

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

It is actually very localised in runner code where beam should reset the classloader when the deserialization happens and then the runner owns the classloader all the way in evaluators. If IO change the classloader they must indeed handle it too and patch the deserialization too. Here again (we

Re: Beam SQL Improvements

>> like to understand your concerns. Also keep in mind that I'm tagging all >>> these classes as Experimental for now, so we can definitely change these >>> interfaces around if we decide they are not the best ones. >>> >>> Reuven >>> >>> On Tue,

Re: Beam SQL Improvements

once distributed you serialized the instances so kind of broke the lifecycle of the original instance and have no real release/close hook on them anymore right? Not sure we can do better than dofn/source embedded instances today. Le mer. 23 mai 2018 08:02, Romain Manni-Bucau <rmann

Re: Beam SQL Improvements

saner this way, no? Also it needs to ensure all converters are present before running the pipeline probably, no implicit environment converter support is probably good to start to avoid late surprises. > My $0.01 > > Regards > JB > > On 23/05/2018 07:51, Romain Manni-Bucau wrote:

Re: Beam SQL Improvements

aks. This is why it can require some way to not recreate it. A quick fix, if you are in bytebuddy already, can be to add it to setup/teardown pby, being more global would be nicer but is more challenging. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannib

Re: Beam SQL Improvements

If so, how an IO can use as() with the type it expects? Doesnt it lead to have a tons of these modules at the end? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com>

Re: Beam SQL Improvements

;re...@google.com> a écrit : > We can do even better btw. Building a SchemaRegistry where automatic > conversions can be registered between schema and Java data types. With this > the user won't even need a DoFn to do the conversion. > > On Tue, May 22, 2018, 10:13 AM Romain

Re: Beam SQL Improvements

for now, at least for IO and some user transforms. Wdyt? Le ven. 27 avr. 2018 18:36, Romain Manni-Bucau <rmannibu...@gmail.com> a écrit : > Can give it a try end of may, sure. (holidays and work constraints will > make it hard before). > > Le 27 avr. 2018 18:26, "Anton K

Re: Beam high level directions (was "Graal instead of docker?")

eases pipeline >> portability, especially for pipelines that are not in the same language as >> the runner-core, such as for Python and Go. >> >> It's not clear to me what runners would be doing in that scenario either. >> Do you have a proposal about where the interface bound

Re: Beam high level directions (was "Graal instead of docker?")

work on, and we are grateful for their efforts. >> >> > Technically, there is plenty to agree with. I think as you learn about >> Beam you will find that many of your suggestions are already handled in >> some way. You may also continue to learn sometimes about the spe

Beam high level directions (was "Graal instead of docker?")

ssume the constraints it brings. And finally it should be less intrusive in all its layers and try to add features more transversally when possible (and it is possible in a lot of cases). If you bring features for free with new releases, everybody wins, if you announce features and no runner support it,

Re: Graal instead of docker?

s outside >> the expected code of conduct. >> >> On Fri, May 11, 2018 at 11:48 AM Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> >>> Also beam community is java - dont answer it is python or go without >>> checking ;). Not sure

Re: Graal instead of docker?

nst stopping current impl track, revert it and move to a higher level runner? > Andrew > > On Fri, May 11, 2018 at 8:53 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> >> >> Le mer. 9 mai 2018 17:41, Eugene Kirpichov <kirpic...@google.com> a &g

Re: Graal instead of docker?

Le mer. 9 mai 2018 17:41, Eugene Kirpichov <kirpic...@google.com> a écrit : > > > On Wed, May 9, 2018 at 1:08 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> >> >> Le mer. 9 mai 2018 00:57, Henning Rohde <hero...@google.com> a éc

Re: Graal instead of docker?

Le mer. 9 mai 2018 00:57, Henning Rohde a écrit : > There are indeed lots of possibilities for interesting docker alternatives > with different tradeoffs and capabilities, but in generally both the runner > as well as the SDK must support them for it to work. As mentioned,

Re: Graal instead of docker?

Le mar. 8 mai 2018 10:16, Robert Bradshaw <rober...@google.com> a écrit : > On Sun, May 6, 2018 at 1:30 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > Wow, this mail should be on the website Robert, thanks for it > > > I still have a point

Re: Graal instead of docker?

erFnObjectInJava, > executing everything as if it were pure Java. In particular the overheads > of unnecessarily crossing the language boundaries many times in a single > fused graph are often prohibitive. > > Sorry for the long email, but hopefully this helps shed some light on (at > lea

Re: Graal instead of docker?

>>> >>> Please also keep in mind that execution of user code is only a small >>> part of the overall portability picture, and dependency on Docker is an >>> even smaller part of that (there is only 1 mention of the word "Docker" in >>> all of Beam

Re: Graal instead of docker?

buted runners that they must implement the full >>> Portability APIs to be considered Beam multi language compliant but they >>> can prefer for performance reasons to translate without the portability >>> APIs the Java to Java case. >>> >> >> >> This is

Re: Graal instead of docker?

beam cluster with the spark runner would include a spark cluster, plus > what's needed for portability, plus the beam sdk. > > > On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > > >> Le 5 mai 2018 08:43, "Reuven Lax&quo

Re: Graal instead of docker?

nt run sometimes - a colleague hit that yesterday :(. What is a "beam cluster" - opposed to a spark or foink cluster? How would it work on windows servers? On Fri, May 4, 2018, 11:19 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > 2018-05-05 2:33 GMT+02:00 Andrew

Re: Graal instead of docker?

ain, >> >> Docker, unlike selinux, solves a great number of tangible problems for us >> with IMO a relatively small tax. It does not have to be the only way. Some >> of the concerns you bring up along with possibilities were also discussed >> here: https://s.apache.org/be

Re: Graal instead of docker?

t's entirely fine for some runners to use Jython, Graal, etc to provide a specialized offering similar to the direct runners, but it would be disjoint from portability IMO. On Fri, May 4, 2018 at 10:14 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 4 mai 2018 17:55, "L

Re: Graal instead of docker?

work stay clean and actually portable and doesnt impact runners as poc done until today did. Works for me. On Thu, May 3, 2018 at 10:05 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi guys > > Since some time there are efforts to have a language portable support in >

Graal instead of docker?

Hi guys Since some time there are efforts to have a language portable support in beam but I cant really find a case it "works" being based on docker except for some vendor specific infra. Current solution: 1. Is runner intrusive (which is bad for beam and prevents adoption of big data vendors)

Re: Gradle Status: Migrated!

Hi Scott While https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057 is open, gradle is a concurrent of maven but maven must stay the default build tool cause gradle breaks users. Le 1 mai 2018 01:59, "Scott Wegner" a écrit : > Many many of you have been

Re: [PROPOSAL] Preparing 2.5.0 release next week

Le 30 avr. 2018 19:39, "Jean-Baptiste Onofré" a écrit : Hi guys, now that I'm back from vacations, I bring back 2.5.0 release on the table ;) This is also related to the current status of build (Maven/Gradle). FYI, I gonna start the Jira triage tomorrow and I launched

Re: Beam SQL Improvements

few reasons which will make it not the best choice my opinion, > but I may be wrong. Can you put together a design doc or a prototype? > > Thank you, > Anton > > > On Thu, Apr 26, 2018 at 10:17 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> &g

Re: Beam SQL Improvements

; >> On Thu, Apr 26, 2018 at 1:17 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> For now we have a generic schema interface. Json-b can be an impl, avro >>> could be another one. >>> >>> Regards >>> JB >>> Le 26 a

Re: Beam SQL Improvements

ric schema interface. Json-b can be an impl, avro >> could be another one. >> >> Regards >> JB >> Le 26 avr. 2018, à 12:08, Romain Manni-Bucau <rmannibu...@gmail.com> a >> écrit: >>> >>> Hmm, >>> >>> avro has still the pitfalls t

Re: [BEAM-4088] Test isolation differences in Gradle

+1, thks Etienne for the hard work and to have spent time on it even if gradle was making it more complicated, really appreciated. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpres

Re: DoFn thread constraint.

Do you want to give it a try doing a PR on https://github.com/apache/beam-site? Better to fix an issue by the person getting the issue in general in that area ;) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Ol

Re: DoFn thread constraint.

Hi John, The fact a runner caches a fn per thread is an internal implementation detail but a fn will only be activated by one thread max at a time (like stateless or any object pool). This means your fn can fail. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog

Re: Beam SQL Improvements

Hmm, avro has still the pitfalls to have an uncontrolled stack which brings way too much dependencies to be part of any API, this is why I proposed a JSON-P based API (JsonObject) with a custom beam entry for some metadata (headers "à la Camel"). Romain Manni-Bucau @rmannibu

Re: Beam parent SNAPSHOTs are not being built

enrepo tools. > > Reuven > > On Wed, Apr 25, 2018 at 9:34 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Hmm, gradle move not supposed to break any user and there are other >> parents usages, as mentionned previously, so not sure beam can really >

Re: Which features of Dropwizard do you wish to use most with Beam metrics?

Hi Alex When i built sirona I started this way: counters, gauges, validations (healthcheck). It works not bad and enables distributed aggregations but when you think about it, it is a very limited set of what you want as a user. Quickly you will want exceptions and audit event etc.. This all

Re: Beam parent SNAPSHOTs are not being built

Hmm, gradle move not supposed to break any user and there are other parents usages, as mentionned previously, so not sure beam can really bypass it. Maven discussed consumer vs build pom since years for that reason ;). Le 26 avr. 2018 01:59, "Eric Beach" a écrit : > Thanks

Re: Beam to move to JSON-B

.beam.apache.org%3E or work on > https://issues.apache.org/jira/browse/BEAM-3608 and similar to improve > things. Should be very easy now. I didn't work on it because work on poms > is throwaway. Now is a good time to pick it up. > > Kenn > > On Wed, Apr 25, 2018 at 12:10 AM R

Beam to move to JSON-B

bs in both. This is critical to let ops handle vulnerabilities properly and without impacting the whole company ecosystem in a lot of cases. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordp

Re: slack @the-asf?

t; > wrote: > >> Just back from flights. Thanks. >> >> Sorry for the delay, I was busy. >> >> Regards >> JB >> Le 18 mars 2018, à 19:57, Innocent Djiofack <djiofack...@gmail.com> a >> écrit: >>> >>> Thanks Romain! >>>

Re: [BEAM-4088] Test isolation differences in Gradle

No, otherwise it wouldnt pass in idea and maven. Le 19 avr. 2018 00:57, "Kenneth Knowles" <k...@google.com> a écrit : Does the test think a thread leaked when no thread leaked? On Wed, Apr 18, 2018 at 1:51 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > The le

Re: [BEAM-4088] Test isolation differences in Gradle

obably to use a > dynamic analysis since it is not very practical to unit test for leaks > (thread, memory, file handle, whatever) and other global correctness > criteria. > > Kenn > > On Wed, Apr 18, 2018 at 10:13 AM Romain Manni-Bucau <rmannibu...@gmail.com> &g

Re: [PROPOSAL] JIRA notification recipient changes

commits@ or dev@ are common choices. Others are generally redundant or means it is ignored by everybody which is fine but means you just drop notif from jira instead of redirecting them ;) Just my 2 cts Le 18 avr. 2018 20:55, "Kenneth Knowles" a écrit : > Annoyingly, it

Re: [BEAM-4088] Test isolation differences in Gradle

StackTraces=WzBd [4] https://docs.gradle.org/current/dsl/org.gradle.api. tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:maxParallelForks [5] https://docs.gradle.org/current/dsl/org.gradle.api. tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery On Thu, Apr 12, 2018 at 10:03 AM Ro

Re: Gradle Status [April 6]

; On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> There is a fake module xxx_test which should have the right classpath but >> since idea compilation is messed up you will still have to run >> :module:cleanTest :module:test --tests

Re: Gradle Status [April 6]

> On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax <re...@google.com> wrote: > >> Is there a Jira for this 3 second delay? Also you're initial complaint >> was not about the 3 second delay, so it wasn't clear that's what you were >> complaining about. >> >> Reuv

Re: Add a (temporary) Portable Flink branch to the ASF repo?

Maybe add a toggle in flinkoptions to activate it until it is tested and you are happy with it? Kind of --flinkExperimental=sdf,log,... (random values). This allows to hit master and continue to work on it. Le 13 avr. 2018 03:07, "Robert Bradshaw" a écrit : Thomas captures

Re: Gradle Status [April 6]

t : > Ah, I did not. Thanks Romain. > > I tried it again, restarting in between, and still had no differences. > Since it seems like there's no reason not to use "Gradle Test Runner", I'll > mention it in the contributor's guide. > > On Thu, Apr 12, 2018 at 10:

Re: Gradle Status [April 6]

em directly through "./gradlew test". >> >> However, I am having issues with a bunch of validatesRunner tests, mostly >> be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. >> Not sure if it's because of a code change or a gradle config, I'll

Re: grade test setup different than maven one?

n configuration. If so, that's a problem with the test. It should be > able to succeed in any build system, or as a standalone JUnit main built > from the suite, etc. > > Kenn > > On Thu, Apr 12, 2018 at 9:27 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >>

Re: grade test setup different than maven one?

on maven it is quite reliable, ran it > 10 times without any failure. I suspect (but didnt check by lack of time) gradle parallelism is different somehow and can lead to some flackyness here. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-12 18:20 GMT

  1   2   3   4   5   >