Re: Parsing JSON

2015-10-20 Thread Martin Junghanns
An example can be found here: Code to read JSON from line: https://github.com/dbs-leipzig/gradoop/blob/master/gradoop-flink/src/main/java/org/gradoop/io/json/JsonReader.java#L57 How it is called:

Re: Parsing JSON

2015-10-20 Thread Flavio Pompermaier
The easiest way is to have each json object in a single line so you could use the env.readTextFile method.. On 20 Oct 2015 09:03, "Papp, Stefan" wrote: > Hi, > > > I want to process data with JSON. Meaning, I have to receive JSON data and > prepare this data for

Re: A proposal about skew data handling in Flink

2015-10-20 Thread Fabian Hueske
Yes, that sounds good to me. Implement support for generic range partitioning first and go for the non-range-equally-splittable cases later. Best, Fabian 2015-10-20 5:21 GMT+02:00 Li, Chengxiang : > Thanks a lot for the comments, Fabian. I agree with you on the plan >

Re: [DISCUSS] Java code style

2015-10-20 Thread Matthias J. Sax
I am in favor of Google vanilla code style. As far as I followed the discussion there will be no style that everybody loves, but most people agree that there should be a unique style. Thus, adjusting Google style does not give any benefits. -Matthias On 10/20/2015 03:36 PM, Stephan Ewen wrote:

[jira] [Created] (FLINK-2876) Minutiae

2015-10-20 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-2876: - Summary: Minutiae Key: FLINK-2876 URL: https://issues.apache.org/jira/browse/FLINK-2876 Project: Flink Issue Type: Improvement Components: Documentation

Re: [DISCUSS] Java code style

2015-10-20 Thread Márton Balassi
+1 for both As we are planning to restructure the maven projects at the point that breaks the PRs anyway, so going on step further at this point in time is reasonable for me. On Tue, Oct 20, 2015 at 2:37 PM, Matthias J. Sax wrote: > big +1 for both! > > On 10/20/2015 02:31

[jira] [Created] (FLINK-2874) Certain Avro generated getters/setters not recognized

2015-10-20 Thread Ulf Karlsson (JIRA)
Ulf Karlsson created FLINK-2874: --- Summary: Certain Avro generated getters/setters not recognized Key: FLINK-2874 URL: https://issues.apache.org/jira/browse/FLINK-2874 Project: Flink Issue

[DISCUSS] Java code style

2015-10-20 Thread Ufuk Celebi
DISCLAIMER: This is not my personal idea, but a community discussion from some time ago. Don't kill the messenger. In March we were discussing issues with heterogeneity of the code [1]. The summary is that we had a consensus to enforce a stricter code style on our Java code base in order to make

Re: [DISCUSS] Java code style

2015-10-20 Thread Matthias J. Sax
big +1 for both! On 10/20/2015 02:31 PM, Ufuk Celebi wrote: > DISCLAIMER: This is not my personal idea, but a community discussion from > some time ago. Don't kill the messenger. > > In March we were discussing issues with heterogeneity of the code [1]. The > summary is that we had a consensus

Re: [DISCUSS] Java code style

2015-10-20 Thread Till Rohrmann
I like the idea to have a bit stricter code style which will increase code maintainability and makes it easier for people to go through the code. Furthermore, it will relieve us from code style comments while reviewing PRs which can be quite cumbersome. Personally, I like the Google code style.

Parsing JSON

2015-10-20 Thread Papp, Stefan
Hi, I want to process data with JSON. Meaning, I have to receive JSON data and prepare this data for analytics. In the beginning, we might receive this data via files, but I assume soon we will switch to a streaming variant. What is currently the best recommended practice with Flink? Thank

[jira] [Created] (FLINK-2873) WebRuntimeMonitor does not start when multiple log files are present

2015-10-20 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2873: Summary: WebRuntimeMonitor does not start when multiple log files are present Key: FLINK-2873 URL: https://issues.apache.org/jira/browse/FLINK-2873 Project: Flink

Re: [DISCUSS] Java code style

2015-10-20 Thread Henry Saputra
1) yes. Been dancing this issue for a while. Let's pull the trigger. Did the exercise with Tachyon while back and did help readability and homogeneity of code. 2) +1 for Google Java style with documented exceptions and explanation on why. On Tuesday, October 20, 2015, Ufuk Celebi

[jira] [Created] (FLINK-2880) Allow DeserializationSchema to throw IOException in deserialization method

2015-10-20 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2880: --- Summary: Allow DeserializationSchema to throw IOException in deserialization method Key: FLINK-2880 URL: https://issues.apache.org/jira/browse/FLINK-2880 Project:

Re: Travis Error Java 8

2015-10-20 Thread Maximilian Michels
Looks like either a Surefire bug or corrupt memory. Haven't seen this before. On Tue, Oct 20, 2015 at 6:18 PM, Matthias J. Sax wrote: > I never saw something like this before... Travis hick up? Can be > ignored? Or severe issues? > >

Re: Forwarding Strategies

2015-10-20 Thread Stephan Ewen
Hi! I am not sure what PARTITION_RANDOM is about, but FORWARD and PARTITION_FORCED_REBALANCE are the same to the output emitter, but look different to the Optimizer (PARTITION_FORCED_REBALANCE cannot be pushed down). Stephan On Tue, Oct 20, 2015 at 6:39 PM, Greg Hogan

[jira] [Created] (FLINK-2881) WebRuntimeMonitor overly restrictive file patterns

2015-10-20 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-2881: - Summary: WebRuntimeMonitor overly restrictive file patterns Key: FLINK-2881 URL: https://issues.apache.org/jira/browse/FLINK-2881 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-2879) Links in documentation are broken

2015-10-20 Thread Nikolaas Steenbergen (JIRA)
Nikolaas Steenbergen created FLINK-2879: --- Summary: Links in documentation are broken Key: FLINK-2879 URL: https://issues.apache.org/jira/browse/FLINK-2879 Project: Flink Issue Type:

Travis Error Java 8

2015-10-20 Thread Matthias J. Sax
I never saw something like this before... Travis hick up? Can be ignored? Or severe issues? https://travis-ci.org/mjsax/flink/jobs/86431071 -Matthias > # > # A fatal error has been detected by the Java Runtime Environment: > # > SUREFIRE-859: # SIGSEGV (0xb) at pc=0x7f8e02dd96b2,

Forwarding Strategies

2015-10-20 Thread Greg Hogan
Looking at org.apache.flink.runtime.operators.shipping.OutputEmitter, shipping strategies FORWARD, PARTITION_RANDOM, and PARTITION_FORCED_REBALANCE all call a local round-robin partitioning function. I'd like to patch this so that the round-robin count starts at the local task index, but shouldn't

Re: [DISCUSS] Java code style

2015-10-20 Thread Maximilian Michels
I'm a little less excited about this. You might not be aware but, for a large portion of the source code, we already follow the Google style guide. The main changes will be tabs->spaces and 80/100 characters line limit. Out of curiosity, I ran the official Google Style Checkstyle configuration to

Re: Travis Error Java 8

2015-10-20 Thread Stephan Ewen
Looks like a JVM bug. BTW: We may want to downgrade from G1 as the default GC in streaming mode to CMS. In the current Java 8 versions, G1 is still causing core dumps once in a while... Stephan On Tue, Oct 20, 2015 at 6:23 PM, Maximilian Michels wrote: > Looks like either a

[jira] [Created] (FLINK-2882) Improve performance of string conversions

2015-10-20 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-2882: - Summary: Improve performance of string conversions Key: FLINK-2882 URL: https://issues.apache.org/jira/browse/FLINK-2882 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-2877) Move Streaming API out of Staging

2015-10-20 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-2877: --- Summary: Move Streaming API out of Staging Key: FLINK-2877 URL: https://issues.apache.org/jira/browse/FLINK-2877 Project: Flink Issue Type:

[jira] [Created] (FLINK-2878) JobManager warns: Unexpected leader address pattern

2015-10-20 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-2878: - Summary: JobManager warns: Unexpected leader address pattern Key: FLINK-2878 URL: https://issues.apache.org/jira/browse/FLINK-2878 Project: Flink

[jira] [Created] (FLINK-2883) Combinable reduce produces wrong result

2015-10-20 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2883: Summary: Combinable reduce produces wrong result Key: FLINK-2883 URL: https://issues.apache.org/jira/browse/FLINK-2883 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-2884) Apply JMH on HashVsSortMiniBenchmark class.

2015-10-20 Thread GaoLun (JIRA)
GaoLun created FLINK-2884: - Summary: Apply JMH on HashVsSortMiniBenchmark class. Key: FLINK-2884 URL: https://issues.apache.org/jira/browse/FLINK-2884 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-2875) Add createRemoveEnvironment variant for HA

2015-10-20 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-2875: -- Summary: Add createRemoveEnvironment variant for HA Key: FLINK-2875 URL: https://issues.apache.org/jira/browse/FLINK-2875 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Java code style

2015-10-20 Thread Stephan Ewen
Just checking: Do we take Google's style guide as is, including spaces instead of tabs? I like the spaces, but that will make things hard... On Tue, Oct 20, 2015 at 3:01 PM, Gyula Fóra wrote: > +1 for both :) > > Till Rohrmann ezt írta (időpont:

Re: [DISCUSS] Java code style

2015-10-20 Thread Stephan Ewen
+1 for introducing a stricter style guide and starting with the Google style. Should we have a separate discussion whether we take the Google style guide vanilla, or whether we make slight adjustments? On Tue, Oct 20, 2015 at 3:26 PM, Till Rohrmann wrote: > That's how

Re: [DISCUSS] Java code style

2015-10-20 Thread Gyula Fóra
+1 for both :) Till Rohrmann ezt írta (időpont: 2015. okt. 20., K, 14:58): > I like the idea to have a bit stricter code style which will increase code > maintainability and makes it easier for people to go through the code. > Furthermore, it will relieve us from code

Re: [DISCUSS] Java code style

2015-10-20 Thread Till Rohrmann
That's how I've understood Ufuk's mail. Everyone should also be aware that the Google code style limits the number characters per line to either 80 or 100. But I guess that everyone will read it himself. On Tue, Oct 20, 2015 at 3:15 PM, Stephan Ewen wrote: > Just checking: Do