Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Jean-Baptiste Onofré
Fair enough. +1 especially for the documentation. Regards JB On 04/03/2017 08:48 PM, Aviem Zur wrote: Upon further inspection there seems to be an issue we may have overlooked: In cluster mode, some of the runners will have dependencies added directly to the classpath by the cluster, and since

IO ITs: Hosting Docker images

2017-04-03 Thread Stephen Sisk
Summary: For IO ITs that use data stores that need custom docker images in order to run, we can't currently use them in a kubernetes cluster (which is where we host our data stores.) I have a couple options for how to solve this and am looking for feedback from folks involved in creating IO

Adding logging for RunnableOnService/ValidatesRunner tests

2017-04-03 Thread Pablo Estrada
Hello there, I'm running RunnableOnService tests on the DirectRunner, with 'mvn clean verify' in runners/direct-java; and I'd like to add some logging to figure out what's going on in some failures. My question is: 1. Is there a way to run only a specific test with maven? 2. Is there extra

Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Ted Yu
+1 > On Apr 3, 2017, at 11:48 AM, Aviem Zur wrote: > > Upon further inspection there seems to be an issue we may have overlooked: > In cluster mode, some of the runners will have dependencies added directly > to the classpath by the cluster, and since SLF4J can only work

Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Aviem Zur
Upon further inspection there seems to be an issue we may have overlooked: In cluster mode, some of the runners will have dependencies added directly to the classpath by the cluster, and since SLF4J can only work with one binding, the first one in the classpath will be used. So while what we

Re: Update of Pei in Alibaba

2017-04-03 Thread Kenneth Knowles
Nice to hear from you again, Pei! This is awesome news. I'd love to help when you are ready to get it in the repo and hooked up to our testing infrastructure. Kenn On Fri, Mar 31, 2017 at 6:24 PM, Pei HE wrote: > Hi all, > On February, I moved from Seattle to Hangzhou,

Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Aljoscha Krettek
Yes, I think we can exclude log4j from the Flink dependencies. It’s somewhat annoying that they are there in the first place. The Flink doc has this to say about the topic: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html > On 3. Apr 2017, at 17:56, Aviem Zur

Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Aviem Zur
>* java.util.logging could be a good choice for the Direct Runner Yes, this will be great for users (Instead of having no logging when using direct runner). >* Logging backend could be runner-specific, particularly if it needs to >integrate into some other experience Good point, let's take a look

Re: Want to contribute to Beam project

2017-04-03 Thread tarush grover
Thanks Jean. I will take a look at Jira to take upon some tickets. Regards, Tarush On Sun, 2 Apr 2017 at 11:12 AM, Jean-Baptiste Onofré wrote: > Hi Tarush, > > welcome aboard ! > > You can take a look on https://beam.apache.org/contribute/. > > Any contribution is valuable

Re: Update of Pei in Alibaba

2017-04-03 Thread Ismaël Mejía
Thanks Jingsong for answering, and the Streamscope ref, I am going to check the paper, the concept of non-global-checkpointing sounds super interesting. It is nice that you guys are also trying to promote the move to a unified model. Regards, Ismaël On Sun, Apr 2, 2017 at 3:40 PM, JingsongLee

Re: [PROPOSAL] ORC support

2017-04-03 Thread Tibor Kiss
Thanks for your replies, I've created https://issues.apache.org/jira/browse/BEAM-1861 to track this effort. On Sun, Apr 2, 2017 at 7:40 AM, Jean-Baptiste Onofré wrote: > +1 > > By the way, around the same topic, I'm working on Apache CarbonData > support

Re: [DISCUSSION] rename StateSpecs.combiningValue?

2017-04-03 Thread Etienne Chauchot
+1 Etienne Le 30/03/2017 à 20:48, Kenneth Knowles a écrit : +1 for a different reason (also: now is the time to revisit names and other bits of the State API before it is too late :-) Folks may not have this catalog in their head. The classes / methods are: ValueState / value(...) BagState

Re: [PROPOSAL] @OnWindowExpiration

2017-04-03 Thread Etienne Chauchot
+1 Etienne Le 28/03/2017 à 22:27, Kenneth Knowles a écrit : Hi all, I have a little extension to the stateful DoFn annotations to circulate for feedback: Allow a method to be annotated with @OnWindowExpiration to automatically get a callback at some point after the window has expired, but