[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291061#comment-14291061 ] Robert Metzger commented on FLINK-629: -- I'll track the fix here: https://issues.apache.org/jira/browse/FLINK-1446 > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291060#comment-14291060 ] Robert Metzger commented on FLINK-629: -- Cool. I'm glad its working. I'll push the fix to master and release-0.8 branch. Thank you for verifying the fix. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289753#comment-14289753 ] mustafa elbehery commented on FLINK-629: [~rmetzger] I need this fix into release-0.8 maven repo, I have cherry picked the fix into 0.8 branch, would I push it and you merge it ? > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288646#comment-14288646 ] mustafa elbehery commented on FLINK-629: Its working now, Thanks a lot robert for your support :) :) > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287404#comment-14287404 ] Robert Metzger commented on FLINK-629: -- Maybe you can even cherry-pick the commit into the release-0.8 branch > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287403#comment-14287403 ] Robert Metzger commented on FLINK-629: -- I think you can very easily backport the fix to Flink 0.8.0 > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287392#comment-14287392 ] mustafa elbehery commented on FLINK-629: Its on Flink 9, I have to stick right now to Flink 8.0 .. I can not upgrade now > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286283#comment-14286283 ] Robert Metzger commented on FLINK-629: -- Thank you, I'll add that to the change. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286016#comment-14286016 ] Stephan Ewen commented on FLINK-629: Concerning the fix: Some classes are not instantiable by our utils, of by Kryo/Objenesis. I would suggest to catch exceptions and return null in that case. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285427#comment-14285427 ] Robert Metzger commented on FLINK-629: -- Can you try the "kryo_new_instance_fix" branch from my repository (https://github.com/rmetzger/flink.git) ? > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284534#comment-14284534 ] mustafa elbehery commented on FLINK-629: It is proper POJO with all the setters and getters. And I have set the type explicitly as mentioned above, in the InputFormat. I have attached the Pojo and the InputFormat .Java, Thanks in advance. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, SimpleTweetInputFormat.java, > Tweet.java, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284480#comment-14284480 ] Robert Metzger commented on FLINK-629: -- I think the issue is that Kryo is currently not properly creating instances of the types its serializing. I think we need to fix this issue for types which are instantiateable. As a quick workaround for you, can you try making your type a proper POJO type. You can do that by fulfilling all requirements for POJO types (getter setter for private fields). The log output will show you why a type is not a pojo. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283815#comment-14283815 ] mustafa elbehery commented on FLINK-629: I have tried to set the type explicitly By implementing ResultTypeQueryable, but still the problem that Kryo does not create an instance of the Tweet, it just create a reference .. Waiting for your reply > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: Selection_006.png, model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283640#comment-14283640 ] Robert Metzger commented on FLINK-629: -- This screenshot from github is related I guess (https://cloud.githubusercontent.com/assets/2375289/5814551/13a7bc88-a08d-11e4-9133-a0b0d0239533.png) ? So your problem is that your input format is getting a "null" instead a reference to a "Tweet" object? Is Flink treating your POJO (Tweet) as a POJO or as a generic type? What is {{TypeExtractor.createTypeInfo(Tweet.class)}} returning? > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-629) Add support for null values to the java api
[ https://issues.apache.org/jira/browse/FLINK-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283628#comment-14283628 ] mustafa elbehery commented on FLINK-629: @rmetzger I have a problem with the Kryo serializer, it creates a reference of my Pojo and not an object. I am using an event driven Json parser, in which I should fill the object manually, and for performance, I am trying to create only one object. The problem now that Kryo creates only a reference, not an object. > Add support for null values to the java api > --- > > Key: FLINK-629 > URL: https://issues.apache.org/jira/browse/FLINK-629 > Project: Flink > Issue Type: Improvement > Components: Java API >Reporter: Stephan Ewen >Assignee: Gyula Fora >Priority: Critical > Labels: github-import > Fix For: pre-apache > > Attachments: model.tar.gz > > > Currently, many runtime operations fail when encountering a null value. Tuple > serialization should allow null fields. > I suggest to add a method to the tuples called `getFieldNotNull()` which > throws a meaningful exception when the accessed field is null. That way, we > simplify the logic of operators that should not dead with null fields, like > key grouping or aggregations. > Even though SQL allows grouping and aggregating of null values, I suggest to > exclude this from the java api, because the SQL semantics of aggregating null > fields are messy. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/629 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: enhancement, java api, > Milestone: Release 0.5.1 > Created at: Wed Mar 26 00:27:49 CET 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)