[jira] [Commented] (FLINK-2243) Add finite spout functionality to Storm compatibility layer
[ https://issues.apache.org/jira/browse/FLINK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594523#comment-14594523 ] ASF GitHub Bot commented on FLINK-2243: --- GitHub user szape opened a pull request: https://github.com/apache/flink/pull/853 [FLINK-2243] [storm-compat] Added finite spout functionality to Storm compatibility layer Added finite spout functionality to Storm compatibility layer Storm spouts emit infinite streams. It is important to support the users of Flink's Storm compatibility layer by providing an interface that can add a finite nature to any storm spouts, given that Flink actually supports finite source streams. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink finite-storm-spout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/853.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #853 commit ce3c1cb6d80c51dcb2e27409448ce84eeec96f12 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:18:09Z [storm-compat] Removed failing simple join example commit d0a80b382c5de332f91e231cd9530efd357cd051 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:22:04Z [storm-compat] Added finite spout functionality to Storm compatibility layer commit d1a325e9f287300c5eca2bdfb90027aa5f27b6a6 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:43:10Z [storm-compat] Demonstrating finite Storm spout functionality on exclamation example -minor renaming -improving JavaDocs commit 4a59aa6704d3bd4b9a6e334ed93160a3e4eea68f Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:44:32Z [storm-compat] Fixing JavaDocs in word count example Add finite spout functionality to Storm compatibility layer --- Key: FLINK-2243 URL: https://issues.apache.org/jira/browse/FLINK-2243 Project: Flink Issue Type: Improvement Components: flink-contrib Reporter: Péter Szabó Assignee: Péter Szabó Priority: Minor Add finite spout functionality to Storm compatibility layer Storm spouts emit infinite streams. It is important to support the users of Flink's Storm compatibility layer by providing an interface that can add a finite nature to any storm spouts, given that Flink actually supports finite source streams. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2149) Simplify Gelly Jaccard similarity example
[ https://issues.apache.org/jira/browse/FLINK-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594558#comment-14594558 ] ASF GitHub Bot commented on FLINK-2149: --- Github user andralungu commented on the pull request: https://github.com/apache/flink/pull/770#issuecomment-113752297 Meging... Simplify Gelly Jaccard similarity example - Key: FLINK-2149 URL: https://issues.apache.org/jira/browse/FLINK-2149 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Priority: Trivial Labels: easyfix, starter The Gelly Jaccard similarity example can be simplified by replacing the groupReduceOnEdges method with the simpler reduceOnEdges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2149][gelly] Simplified Jaccard Example
Github user andralungu commented on the pull request: https://github.com/apache/flink/pull/770#issuecomment-113752297 Meging... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2243] [storm-compat] Added finite spout...
GitHub user szape opened a pull request: https://github.com/apache/flink/pull/853 [FLINK-2243] [storm-compat] Added finite spout functionality to Storm compatibility layer Added finite spout functionality to Storm compatibility layer Storm spouts emit infinite streams. It is important to support the users of Flink's Storm compatibility layer by providing an interface that can add a finite nature to any storm spouts, given that Flink actually supports finite source streams. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink finite-storm-spout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/853.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #853 commit ce3c1cb6d80c51dcb2e27409448ce84eeec96f12 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:18:09Z [storm-compat] Removed failing simple join example commit d0a80b382c5de332f91e231cd9530efd357cd051 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:22:04Z [storm-compat] Added finite spout functionality to Storm compatibility layer commit d1a325e9f287300c5eca2bdfb90027aa5f27b6a6 Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:43:10Z [storm-compat] Demonstrating finite Storm spout functionality on exclamation example -minor renaming -improving JavaDocs commit 4a59aa6704d3bd4b9a6e334ed93160a3e4eea68f Author: szape nemderogator...@gmail.com Date: 2015-06-19T08:44:32Z [storm-compat] Fixing JavaDocs in word count example --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2149][gelly] Simplified Jaccard Example
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/770 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2149) Simplify Gelly Jaccard similarity example
[ https://issues.apache.org/jira/browse/FLINK-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594627#comment-14594627 ] ASF GitHub Bot commented on FLINK-2149: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/770 Simplify Gelly Jaccard similarity example - Key: FLINK-2149 URL: https://issues.apache.org/jira/browse/FLINK-2149 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Priority: Trivial Labels: easyfix, starter The Gelly Jaccard similarity example can be simplified by replacing the groupReduceOnEdges method with the simpler reduceOnEdges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-2149) Simplify Gelly Jaccard similarity example
[ https://issues.apache.org/jira/browse/FLINK-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andra Lungu updated FLINK-2149: --- Fix Version/s: 0.10 Simplify Gelly Jaccard similarity example - Key: FLINK-2149 URL: https://issues.apache.org/jira/browse/FLINK-2149 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Priority: Trivial Labels: easyfix, starter Fix For: 0.10 The Gelly Jaccard similarity example can be simplified by replacing the groupReduceOnEdges method with the simpler reduceOnEdges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2149) Simplify Gelly Jaccard similarity example
[ https://issues.apache.org/jira/browse/FLINK-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andra Lungu resolved FLINK-2149. Resolution: Fixed Simplify Gelly Jaccard similarity example - Key: FLINK-2149 URL: https://issues.apache.org/jira/browse/FLINK-2149 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Priority: Trivial Labels: easyfix, starter The Gelly Jaccard similarity example can be simplified by replacing the groupReduceOnEdges method with the simpler reduceOnEdges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [contrib] Storm compatibility
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/764#issuecomment-113818896 Cool. Maybe we should write also a blog post about it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2243) Add finite spout functionality to Storm compatibility layer
[
https://issues.apache.org/jira/browse/FLINK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594811#comment-14594811
]
ASF GitHub Bot commented on FLINK-2243:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/853#discussion_r32888046
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/wordcount/SpoutSourceWordCount.java
---
@@ -145,7 +145,7 @@ private static boolean parseParameters(final String[]
args) {
}
return env.addSource(new StormFiniteSpoutWrapperString(new
StormInMemorySpout(WordCountData.WORDS), true),
- TypeExtractor.getForClass(String.class));
+
TypeExtractor.getForClass(String.class)).setParallelism(1);
--- End diff --
Why do you fix the parallelism here?
Add finite spout functionality to Storm compatibility layer
---
Key: FLINK-2243
URL: https://issues.apache.org/jira/browse/FLINK-2243
Project: Flink
Issue Type: Improvement
Components: flink-contrib
Reporter: Péter Szabó
Assignee: Péter Szabó
Priority: Minor
Add finite spout functionality to Storm compatibility layer
Storm spouts emit infinite streams. It is important to support the users of
Flink's Storm compatibility layer by providing an interface that can add a
finite nature to any storm spouts, given that Flink actually supports finite
source streams.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2243] [storm-compat] Added finite spout...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/853#discussion_r32888046
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/wordcount/SpoutSourceWordCount.java
---
@@ -145,7 +145,7 @@ private static boolean parseParameters(final String[]
args) {
}
return env.addSource(new StormFiniteSpoutWrapperString(new
StormInMemorySpout(WordCountData.WORDS), true),
- TypeExtractor.getForClass(String.class));
+
TypeExtractor.getForClass(String.class)).setParallelism(1);
--- End diff --
Why do you fix the parallelism here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] Fast calculation of medians of win...
Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/684#issuecomment-113820597 Merged the first commit as that was already clarified. As for the second one the placement to either `streaming-core` or `streaming-contrib` is still unclear. I moved it to contrib on the following branch as a subclass of `WindowedDataStream`. [1] I admit that the naming is weak, any suggestions are very welcome on that. @ggevay, @gyfora what do you think? [1] https://github.com/mbalassi/flink/commit/956145716dca11ddaffd4b9a0e95f7f9546a055d --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2243] [storm-compat] Added finite spout...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/853#discussion_r32888034
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/util/FiniteStormFileSpout.java
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.stormcompatibility.util;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.tuple.Values;
+import org.apache.flink.stormcompatibility.wrappers.FiniteStormSpout;
+
+import java.io.BufferedReader;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Implements a Storm Spout that reads data from a given local file. The
spout stops automatically
+ * when it reached the end of the file.
+ */
+public class FiniteStormFileSpout extends AbstractStormSpout implements
FiniteStormSpout {
+ private static final long serialVersionUID = -6996907090003590436L;
+
+ private final String path;
+ private BufferedReader reader;
+ private String line;
+ private boolean newLineRead;
+
+ public FiniteStormFileSpout(final String path) {
+ this.path = path;
+ }
+
+ @SuppressWarnings(rawtypes)
+ @Override
+ public void open(final Map conf, final TopologyContext context,
+ final SpoutOutputCollector collector) {
+ super.open(conf, context, collector);
+ try {
+ this.reader = new BufferedReader(new
FileReader(this.path));
+ } catch (final FileNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ newLineRead = false;
+ }
+
+ @Override
+ public void close() {
+ if (this.reader != null) {
+ try {
+ this.reader.close();
+ } catch (final IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @Override
+ public void nextTuple() {
+ if (!reachedEnd()) {
+ this.collector.emit(new Values(line));
+ }
+ newLineRead = false;
+ }
+
+ /**
+* Can be called before nextTuple() any times including 0.
+*/
+ public boolean reachedEnd() {
+ try {
+ readLine();
+ } catch (IOException e) {
+ e.printStackTrace();
--- End diff --
Is this exception used to signal the end of the stream? or is it a error
condition?
In any case, doing a `e.printStackStrace()` is not good because it will log
it to the stdout file.
I would rethrow a runtime exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2243) Add finite spout functionality to Storm compatibility layer
[
https://issues.apache.org/jira/browse/FLINK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594804#comment-14594804
]
ASF GitHub Bot commented on FLINK-2243:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/853#discussion_r32888034
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/util/FiniteStormFileSpout.java
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.stormcompatibility.util;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.tuple.Values;
+import org.apache.flink.stormcompatibility.wrappers.FiniteStormSpout;
+
+import java.io.BufferedReader;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Implements a Storm Spout that reads data from a given local file. The
spout stops automatically
+ * when it reached the end of the file.
+ */
+public class FiniteStormFileSpout extends AbstractStormSpout implements
FiniteStormSpout {
+ private static final long serialVersionUID = -6996907090003590436L;
+
+ private final String path;
+ private BufferedReader reader;
+ private String line;
+ private boolean newLineRead;
+
+ public FiniteStormFileSpout(final String path) {
+ this.path = path;
+ }
+
+ @SuppressWarnings(rawtypes)
+ @Override
+ public void open(final Map conf, final TopologyContext context,
+ final SpoutOutputCollector collector) {
+ super.open(conf, context, collector);
+ try {
+ this.reader = new BufferedReader(new
FileReader(this.path));
+ } catch (final FileNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ newLineRead = false;
+ }
+
+ @Override
+ public void close() {
+ if (this.reader != null) {
+ try {
+ this.reader.close();
+ } catch (final IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @Override
+ public void nextTuple() {
+ if (!reachedEnd()) {
+ this.collector.emit(new Values(line));
+ }
+ newLineRead = false;
+ }
+
+ /**
+* Can be called before nextTuple() any times including 0.
+*/
+ public boolean reachedEnd() {
+ try {
+ readLine();
+ } catch (IOException e) {
+ e.printStackTrace();
--- End diff --
Is this exception used to signal the end of the stream? or is it a error
condition?
In any case, doing a `e.printStackStrace()` is not good because it will log
it to the stdout file.
I would rethrow a runtime exception
Add finite spout functionality to Storm compatibility layer
---
Key: FLINK-2243
URL: https://issues.apache.org/jira/browse/FLINK-2243
Project: Flink
Issue Type: Improvement
Components: flink-contrib
Reporter: Péter Szabó
Assignee: Péter Szabó
Priority: Minor
Add finite spout functionality to Storm compatibility layer
Storm spouts emit infinite streams. It is important to support the users of
Flink's Storm compatibility layer by providing an interface that can add a
finite nature to any storm spouts, given that Flink actually supports finite
source streams.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2175) Allow multiple jobs in single jar file
[ https://issues.apache.org/jira/browse/FLINK-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594902#comment-14594902 ] ASF GitHub Bot commented on FLINK-2175: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/707#issuecomment-113846141 I would like to merge the change. It doesn't break anything existing and could be helpful for our users. Since you probably have to update the PR for the examples anyways, could you also quickly mention the feature in the documentation somewhere? Allow multiple jobs in single jar file -- Key: FLINK-2175 URL: https://issues.apache.org/jira/browse/FLINK-2175 Project: Flink Issue Type: Improvement Components: Examples, other, Webfrontend Reporter: Matthias J. Sax Assignee: Matthias J. Sax Priority: Minor Allow to package multiple jobs into a single jar. - extend WebClient to display all available jobs - extend WebClient to diplay plan and submit each job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2175] Allow multiple jobs in single jar...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/707#issuecomment-113846141 I would like to merge the change. It doesn't break anything existing and could be helpful for our users. Since you probably have to update the PR for the examples anyways, could you also quickly mention the feature in the documentation somewhere? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2167) Add fromHCat() to TableEnvironment
[
https://issues.apache.org/jira/browse/FLINK-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594897#comment-14594897
]
Robert Metzger commented on FLINK-2167:
---
Mh. Are there any error message?
Maybe you have to load the hive-site.xml through the classloader?
Add fromHCat() to TableEnvironment
--
Key: FLINK-2167
URL: https://issues.apache.org/jira/browse/FLINK-2167
Project: Flink
Issue Type: New Feature
Components: Table API
Affects Versions: 0.9
Reporter: Fabian Hueske
Assignee: Timo Walther
Priority: Minor
Labels: starter
Add a {{fromHCat()}} method to the {{TableEnvironment}} to read a {{Table}}
from an HCatalog table.
The implementation could reuse Flink's HCatInputFormat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2175) Allow multiple jobs in single jar file
[ https://issues.apache.org/jira/browse/FLINK-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594899#comment-14594899 ] ASF GitHub Bot commented on FLINK-2175: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/707#discussion_r32889524 --- Diff: flink-examples/flink-java-examples/pom.xml --- @@ -319,7 +319,32 @@ under the License. /includes /configuration /execution - + --- End diff -- This will produce another jar file in the `examples/` directory, right? If so, I'm against this because its very hard for beginners to understand the purpose of this example. Allow multiple jobs in single jar file -- Key: FLINK-2175 URL: https://issues.apache.org/jira/browse/FLINK-2175 Project: Flink Issue Type: Improvement Components: Examples, other, Webfrontend Reporter: Matthias J. Sax Assignee: Matthias J. Sax Priority: Minor Allow to package multiple jobs into a single jar. - extend WebClient to display all available jobs - extend WebClient to diplay plan and submit each job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2175] Allow multiple jobs in single jar...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/707#discussion_r32889524 --- Diff: flink-examples/flink-java-examples/pom.xml --- @@ -319,7 +319,32 @@ under the License. /includes /configuration /execution - + --- End diff -- This will produce another jar file in the `examples/` directory, right? If so, I'm against this because its very hard for beginners to understand the purpose of this example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [streaming] Fast calculation of medians of win...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/684#discussion_r32888195
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/util/FieldAccessorTest.java
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.util;
+
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.BasicArrayTypeInfo;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+// This only tests a fraction of FieldAccessor. The other parts are tested
indirectly by AggregationFunctionTest.
+public class FieldAccessorTest {
+
+ @Test
+ @SuppressWarnings(unchecked)
+ public void arrayFieldAccessorTest() {
+ int[] a = new int[]{3,5};
+ FieldAccessorint[], Integer fieldAccessor =
+ (FieldAccessorint[], Integer) (Object)
+ FieldAccessor.create(1,
PrimitiveArrayTypeInfo.getInfoFor(a.getClass()), null);
+
+ assertEquals(Integer.class,
fieldAccessor.getFieldType().getTypeClass());
+
+ assertEquals((Integer)a[1], fieldAccessor.get(a));
+
+ a = fieldAccessor.set(a, 6);
+ assertEquals((Integer)a[1], fieldAccessor.get(a));
+
+
+
+ Integer[] b = new Integer[]{3,5};
+ FieldAccessorInteger[], Integer fieldAccessor2 =
+ (FieldAccessorInteger[], Integer) (Object)
+ FieldAccessor.create(1,
BasicArrayTypeInfo.getInfoFor(b.getClass()), null);
+
+ assertEquals(Integer.class,
fieldAccessor2.getFieldType().getTypeClass());
+
+ assertEquals((Integer)b[1], fieldAccessor2.get(b));
+
+ b = fieldAccessor2.set(b, 6);
+ assertEquals((Integer)b[1], fieldAccessor2.get(b));
+ }
+
+ @Test
+ @SuppressWarnings(unchecked)
+ public void tupleFieldAccessorOutOfBoundsTest() {
+ try {
+ FieldAccessorTuple2Integer, Integer, Integer
fieldAccessor =
+ (FieldAccessorTuple2Integer,
Integer, Integer) (Object)
+ FieldAccessor.create(2,
TupleTypeInfo.getBasicTupleTypeInfo(Integer.class, Integer.class),
+ null);
+ fail();
+ } catch (IndexOutOfBoundsException e) {
+ // Nothing to do here
+ }
+ }
--- End diff --
Why is there no test for the POJO field accessor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594853#comment-14594853 ] ASF GitHub Bot commented on FLINK-1085: --- GitHub user dabaitu opened a pull request: https://github.com/apache/flink/pull/854 FLINK-1085: Unnecessary failing of GroupReduceCombineDriver I have a unit test failure and it seems it may have been there before my changes: Tests in error: UtilsTest.testUberjarLocator:39 NullPointer It's looking for some uber jar and not finding it. Did I need to compile something else first? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dabaitu/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/854.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #854 commit e693ba2398c170b6d65a2365a4cc0fe091f3319b Author: dabaitu tom@gmail.com Date: 2015-06-20T22:35:48Z FLINK-1085: change exception to warning log and track oversized Record count Unnecessary failing of GroupReduceCombineDriver --- Key: FLINK-1085 URL: https://issues.apache.org/jira/browse/FLINK-1085 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.6.1-incubating, 0.7.0-incubating Reporter: Fabian Hueske Labels: starter With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again. The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer. Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: FLINK-1085: Unnecessary failing of GroupReduce...
GitHub user dabaitu opened a pull request: https://github.com/apache/flink/pull/854 FLINK-1085: Unnecessary failing of GroupReduceCombineDriver I have a unit test failure and it seems it may have been there before my changes: Tests in error: UtilsTest.testUberjarLocator:39 NullPointer It's looking for some uber jar and not finding it. Did I need to compile something else first? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dabaitu/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/854.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #854 commit e693ba2398c170b6d65a2365a4cc0fe091f3319b Author: dabaitu tom@gmail.com Date: 2015-06-20T22:35:48Z FLINK-1085: change exception to warning log and track oversized Record count --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594881#comment-14594881 ] ASF GitHub Bot commented on FLINK-1085: --- Github user dabaitu commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113842215 @rmetzger : 'mvn test' Unnecessary failing of GroupReduceCombineDriver --- Key: FLINK-1085 URL: https://issues.apache.org/jira/browse/FLINK-1085 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.6.1-incubating, 0.7.0-incubating Reporter: Fabian Hueske Labels: starter With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again. The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer. Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1085] Unnecessary failing of GroupReduc...
Github user dabaitu commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113842215 @rmetzger : 'mvn test' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1085] Unnecessary failing of GroupReduc...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113842515 Can you try to run mvn clean verify in the project root? It will use a lot of resources and run for more than 30 minutes but the tests should pass ;) Also, the travis (CI system) build for your pull request passed. So the tests are all good. Lets wait for @fhueske to review your proposed changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594886#comment-14594886 ] ASF GitHub Bot commented on FLINK-1085: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113842515 Can you try to run mvn clean verify in the project root? It will use a lot of resources and run for more than 30 minutes but the tests should pass ;) Also, the travis (CI system) build for your pull request passed. So the tests are all good. Lets wait for @fhueske to review your proposed changes. Unnecessary failing of GroupReduceCombineDriver --- Key: FLINK-1085 URL: https://issues.apache.org/jira/browse/FLINK-1085 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.6.1-incubating, 0.7.0-incubating Reporter: Fabian Hueske Labels: starter With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again. The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer. Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [streaming] Fast calculation of medians of win...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/684#discussion_r32888221
--- Diff:
flink-staging/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedDataStream.scala
---
@@ -339,4 +335,30 @@ class WindowedDataStream[T](javaStream:
JavaWStream[T]) {
javaStream.getDiscretizedStream.getExecutionEnvironment).scalaClean(f)
}
+ /**
+ * Gives the median of the current window at the specified field at
every trigger.
+ * The type of the field can only be Double (as the median of integers
might be a
+ * fractional number).
+ *
+ * The median is updated online as the window changes, and the runtime of
+ * one update is logarithmic with the current window size.
+ *
+ */
+ def median(pos: Int): WindowedDataStream[T] = {
--- End diff --
Can you also update the streaming programming guide with this new method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: New operator state interfaces
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/747#issuecomment-113836818 Mh. Is it always the same test failing with the same message? What is the failure? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-2141) Allow GSA's Gather to perform this operation in more than one direction
[ https://issues.apache.org/jira/browse/FLINK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andra Lungu updated FLINK-2141: --- Assignee: Shivani Ghatge Allow GSA's Gather to perform this operation in more than one direction --- Key: FLINK-2141 URL: https://issues.apache.org/jira/browse/FLINK-2141 Project: Flink Issue Type: New Feature Components: Gelly Affects Versions: 0.9 Reporter: Andra Lungu Assignee: Shivani Ghatge For the time being, a vertex only gathers information from its in-edges. Similarly to the vertex-centric approach, we would like to allow users to gather data from out and all edges as well. This property should be set using a setDirection() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2248) Allow disabling of sdtout logging output
[ https://issues.apache.org/jira/browse/FLINK-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594851#comment-14594851 ] Robert Metzger commented on FLINK-2248: --- The execution config has a method to disable the stdout logging. It would be cool to change that flag from the command line interface as well. Allow disabling of sdtout logging output Key: FLINK-2248 URL: https://issues.apache.org/jira/browse/FLINK-2248 Project: Flink Issue Type: Improvement Reporter: Theodore Vasiloudis Priority: Minor Currently when a job is submitted through the CLI we get in stdout all the log output about each stage in the job. It would useful to have an easy way to disable this output when submitting the job, as most of the time we are only interested in the log output if something goes wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1085] Unnecessary failing of GroupReduc...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113835793 How did you start the tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594858#comment-14594858 ] ASF GitHub Bot commented on FLINK-1085: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-113835793 How did you start the tests? Unnecessary failing of GroupReduceCombineDriver --- Key: FLINK-1085 URL: https://issues.apache.org/jira/browse/FLINK-1085 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.6.1-incubating, 0.7.0-incubating Reporter: Fabian Hueske Labels: starter With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again. The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer. Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-2235) Local Flink cluster allocates too much memory
[
https://issues.apache.org/jira/browse/FLINK-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594861#comment-14594861
]
Robert Metzger edited comment on FLINK-2235 at 6/20/15 11:01 PM:
-
I agree. Lets fix this one for OracleJDK 6 (if it is not causing too much
trouble)
was (Author: rmetzger):
I agree. Lets fix this one for OracleJDK 6.
Local Flink cluster allocates too much memory
-
Key: FLINK-2235
URL: https://issues.apache.org/jira/browse/FLINK-2235
Project: Flink
Issue Type: Bug
Components: Local Runtime, TaskManager
Affects Versions: 0.9
Environment: Oracle JDK: 1.6.0_65-b14-462
Eclipse
Reporter: Maximilian Michels
Priority: Minor
When executing a Flink job locally, the task manager gets initialized with an
insane amount of memory. After a quick look in the code it seems that the
call to {{EnvironmentInformation.getSizeOfFreeHeapMemoryWithDefrag()}}
returns a wrong estimate of the heap memory size.
Moreover, the same user switched to Oracle JDK 1.8 and that made the error
disappear. So I'm guessing this is some Java 1.6 quirk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2235) Local Flink cluster allocates too much memory
[
https://issues.apache.org/jira/browse/FLINK-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594861#comment-14594861
]
Robert Metzger commented on FLINK-2235:
---
I agree. Lets fix this one for OracleJDK 6.
Local Flink cluster allocates too much memory
-
Key: FLINK-2235
URL: https://issues.apache.org/jira/browse/FLINK-2235
Project: Flink
Issue Type: Bug
Components: Local Runtime, TaskManager
Affects Versions: 0.9
Environment: Oracle JDK: 1.6.0_65-b14-462
Eclipse
Reporter: Maximilian Michels
Priority: Minor
When executing a Flink job locally, the task manager gets initialized with an
insane amount of memory. After a quick look in the code it seems that the
call to {{EnvironmentInformation.getSizeOfFreeHeapMemoryWithDefrag()}}
returns a wrong estimate of the heap memory size.
Moreover, the same user switched to Oracle JDK 1.8 and that made the error
disappear. So I'm guessing this is some Java 1.6 quirk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
