[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578778#comment-16578778
]
ASF GitHub Bot commented on FLINK-6437:
---
zentol commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-412622673
well I still like the idea of separate config files, but the JIRA discussion
happened
more than a year ago, _before we even had released the HistoryServer_. Now
we have to think about backwards compatibility and will thus naturally end up
adding complexity. I'm not sure if this is really worth it, especially so since
this issue has never been raised again since the HS was released.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #6500: [FLINK-10066] Keep only archived version of previous executions
asfgit closed pull request #6500: [FLINK-10066] Keep only archived version of
previous executions
URL: https://github.com/apache/flink/pull/6500
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
index 4b1c62fe707..ab8c94c0188 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
@@ -18,6 +18,7 @@
package org.apache.flink.runtime.executiongraph;
import org.apache.flink.runtime.accumulators.StringifiedAccumulatorResult;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
import org.apache.flink.runtime.execution.ExecutionState;
import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
import org.apache.flink.util.ExceptionUtils;
@@ -40,6 +41,8 @@
private final TaskManagerLocation assignedResourceLocation; // for the
archived execution
+ private final AllocationID assignedAllocationID;
+
/* Continuously updated map of user-defined accumulators */
private final StringifiedAccumulatorResult[] userAccumulators;
@@ -48,21 +51,24 @@
private final IOMetrics ioMetrics;
public ArchivedExecution(Execution execution) {
- this.userAccumulators =
execution.getUserAccumulatorsStringified();
- this.attemptId = execution.getAttemptId();
- this.attemptNumber = execution.getAttemptNumber();
- this.stateTimestamps = execution.getStateTimestamps();
- this.parallelSubtaskIndex =
execution.getVertex().getParallelSubtaskIndex();
- this.state = execution.getState();
- this.failureCause =
ExceptionUtils.stringifyException(execution.getFailureCause());
- this.assignedResourceLocation =
execution.getAssignedResourceLocation();
- this.ioMetrics = execution.getIOMetrics();
+ this(
+ execution.getUserAccumulatorsStringified(),
+ execution.getIOMetrics(),
+ execution.getAttemptId(),
+ execution.getAttemptNumber(),
+ execution.getState(),
+
ExceptionUtils.stringifyException(execution.getFailureCause()),
+ execution.getAssignedResourceLocation(),
+ execution.getAssignedAllocationID(),
+ execution.getVertex().getParallelSubtaskIndex(),
+ execution.getStateTimestamps());
}
public ArchivedExecution(
StringifiedAccumulatorResult[] userAccumulators,
IOMetrics ioMetrics,
ExecutionAttemptID attemptId, int attemptNumber,
ExecutionState state, String failureCause,
- TaskManagerLocation assignedResourceLocation, int
parallelSubtaskIndex, long[] stateTimestamps) {
+ TaskManagerLocation assignedResourceLocation,
AllocationID assignedAllocationID, int parallelSubtaskIndex,
+ long[] stateTimestamps) {
this.userAccumulators = userAccumulators;
this.ioMetrics = ioMetrics;
this.failureCause = failureCause;
@@ -72,6 +78,7 @@ public ArchivedExecution(
this.state = state;
this.stateTimestamps = stateTimestamps;
this.parallelSubtaskIndex = parallelSubtaskIndex;
+ this.assignedAllocationID = assignedAllocationID;
}
//
@@ -103,6 +110,10 @@ public TaskManagerLocation getAssignedResourceLocation() {
return assignedResourceLocation;
}
+ public AllocationID getAssignedAllocationID() {
+ return assignedAllocationID;
+ }
+
@Override
public String getFailureCauseAsString() {
return failureCause;
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
index 36669d34be7..04efa048fb6 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
@@ -40,7 +40,7 @@
public ArchivedExecutionVertex(ExecutionVertex vertex)
[GitHub] fhueske commented on a change in pull request #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs
fhueske commented on a change in pull request #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs URL: https://github.com/apache/flink/pull/6535#discussion_r209687761 ## File path: docs/dev/table/functions.md ## @@ -24,7 +24,7 @@ under the License. Flink Table API & SQL provides users with a set of built-in functions for data transformations. This page gives a brief overview of them. If a function that you need is not supported yet, you can implement a user-defined function. -Or if you think the function is general enough, please https://issues.apache.org/jira/secure/CreateIssue!default.jspa;>open a JIRA issue for it. +If you think that the function is general enough, please open a Jira issue for it with a detailed description. Review comment: Oh, please keep the link :-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9977) Refine the docs for Table/SQL built-in functions
[
https://issues.apache.org/jira/browse/FLINK-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578625#comment-16578625
]
ASF GitHub Bot commented on FLINK-9977:
---
fhueske commented on a change in pull request #6535: [FLINK-9977] [table][doc]
Refine the SQL/Table built-in function docs
URL: https://github.com/apache/flink/pull/6535#discussion_r209687761
##
File path: docs/dev/table/functions.md
##
@@ -24,7 +24,7 @@ under the License.
Flink Table API & SQL provides users with a set of built-in functions for data
transformations. This page gives a brief overview of them.
If a function that you need is not supported yet, you can implement a user-defined function.
-Or if you think the function is general enough, please https://issues.apache.org/jira/secure/CreateIssue!default.jspa;>open a
JIRA issue for it.
+If you think that the function is general enough, please open a Jira issue for
it with a detailed description.
Review comment:
Oh, please keep the link :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine the docs for Table/SQL built-in functions
>
>
> Key: FLINK-9977
> URL: https://issues.apache.org/jira/browse/FLINK-9977
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Minor
> Labels: pull-request-available
> Attachments: Java.jpg, SQL.jpg, Scala.jpg
>
>
> There exist some syntax errors or inconsistencies in documents and Scala docs
> of the Table/SQL built-in functions. This issue aims to make some
> improvements to them.
> Also, according to FLINK-10103, we should use single quotes to express
> strings in SQL. For example, CONCAT("AA", "BB", "CC") should be replaced with
> CONCAT('AA', 'BB', 'CC').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl]
add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r209690587
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java
##
@@ -175,7 +183,6 @@ ChannelFuture connect(final InetSocketAddress
serverSocketAddress) {
bootstrap.handler(new ChannelInitializer() {
@Override
public void initChannel(SocketChannel channel) throws
Exception {
-
// SSL handler should be added first in the
pipeline
if (clientSSLContext != null) {
Review comment:
if SSL is disabled, for example
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-9878) IO worker threads BLOCKED on SSL Session Cache while CMS full gc
[ https://issues.apache.org/jira/browse/FLINK-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578643#comment-16578643 ] ASF GitHub Bot commented on FLINK-9878: --- NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#discussion_r209690309 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java ## @@ -52,6 +56,9 @@ private Bootstrap bootstrap; Review comment: out of scope of this PR - there's also more around this package, if you wanted to mark/change these accordingly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IO worker threads BLOCKED on SSL Session Cache while CMS full gc > > > Key: FLINK-9878 > URL: https://issues.apache.org/jira/browse/FLINK-9878 > Project: Flink > Issue Type: Bug > Components: Network >Affects Versions: 1.5.0, 1.5.1, 1.6.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Labels: pull-request-available > Fix For: 1.5.3, 1.6.1, 1.7.0 > > > According to https://github.com/netty/netty/issues/832, there is a JDK issue > during garbage collection when the SSL session cache is not limited. We > should allow the user to configure this and further (advanced) SSL parameters > for fine-tuning to fix this and similar issues. In particular, the following > parameters should be configurable: > - SSL session cache size > - SSL session timeout > - SSL handshake timeout > - SSL close notify flush timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9878) IO worker threads BLOCKED on SSL Session Cache while CMS full gc
[ https://issues.apache.org/jira/browse/FLINK-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578644#comment-16578644 ] ASF GitHub Bot commented on FLINK-9878: --- NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#discussion_r209690309 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java ## @@ -52,6 +56,9 @@ private Bootstrap bootstrap; Review comment: out of scope of this PR - there's also even more around this package, if you wanted to mark/change these accordingly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IO worker threads BLOCKED on SSL Session Cache while CMS full gc > > > Key: FLINK-9878 > URL: https://issues.apache.org/jira/browse/FLINK-9878 > Project: Flink > Issue Type: Bug > Components: Network >Affects Versions: 1.5.0, 1.5.1, 1.6.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Labels: pull-request-available > Fix For: 1.5.3, 1.6.1, 1.7.0 > > > According to https://github.com/netty/netty/issues/832, there is a JDK issue > during garbage collection when the SSL session cache is not limited. We > should allow the user to configure this and further (advanced) SSL parameters > for fine-tuning to fix this and similar issues. In particular, the following > parameters should be configurable: > - SSL session cache size > - SSL session timeout > - SSL handshake timeout > - SSL close notify flush timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10066) Keep only archived version of previous executions
[ https://issues.apache.org/jira/browse/FLINK-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter closed FLINK-10066. -- Resolution: Fixed Merged in: master: 160dc56fdf release-1.6: 74323d50b0 release-1.5: 2217c09c88 > Keep only archived version of previous executions > - > > Key: FLINK-10066 > URL: https://issues.apache.org/jira/browse/FLINK-10066 > Project: Flink > Issue Type: Improvement > Components: JobManager >Affects Versions: 1.4.3, 1.5.2, 1.6.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > Labels: pull-request-available > > Currently, the execution vertex stores a limited amount of previous > executions in a bounded list. This happens primarily for archiving purposes > and to remember previous locations and allocation ids. We remember the whole > execution to eventually convert it into an archived execution. > This seems unnecessary and dangerous as we have observed that this strategy > is prone to memory leaks in the job manager. With a very high vertex count or > parallelism, remembering complete executions can become very memory > intensive. Instead I suggest to eagerly transform the executions into the > archived version before adding them to the list, i.e. only the archived > version is ever still referenced after the execution becomes obsolete. This > gives better control over which information about the execution should really > be kept in memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10133) finished job's jobgraph never been cleaned up in zookeeper for standalone clusters (HA mode with multiple masters)
[
https://issues.apache.org/jira/browse/FLINK-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578792#comment-16578792
]
Elias Levy commented on FLINK-10133:
[~Frefreak] this is likely the same issue as FLINK-10011. If so, mark this one
as a duplicate.
> finished job's jobgraph never been cleaned up in zookeeper for standalone
> clusters (HA mode with multiple masters)
> --
>
> Key: FLINK-10133
> URL: https://issues.apache.org/jira/browse/FLINK-10133
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.5.0, 1.5.2, 1.6.0
>Reporter: Xiangyu Zhu
>Priority: Major
>
> Hi,
> We have 3 servers in our test environment, noted as node1-3. Setup is as
> following:
> * hadoop hdfs: node1 as namenode, node2,3 as datanode
> * zookeeper: node1-3 as a quorum (but also tried node1 alone)
> * flink: node1,2 as masters, node2,3 as slaves
> As my understanding when a job finished the corresponding job's blob data is
> expected to be deleted from hdfs path and node under zookeeper's path `/\{zk
> path root}/\{cluster-id}/jobgraphs/\{job id}` should be deleted after that.
> However we observe that whenever we submitted a job and it finished (via
> `bin/flink run WordCount.jar`), the blob data is gone whereas job id node
> under zookeeper is still there, with a uuid style lock node inside it. From
> the debug node in zookeeper we observed something like "cannot be deleted
> because non empty". Because of this, as long as a job is finished and the
> jobgraph node persists, if restart the clusters or kill one manager (to test
> HA mode), it tries to recover a finished job and couldn't find blob data
> under hdfs, and the whole cluster is down.
> If we use only node1 as master and node2,3 as slaves, the jobgraphs node can
> be deleted successfully. If the jobgraphs is clean, killing one job manager
> makes another stand-by JM raised as leader, so it is only this jobgraphs
> issue preventing HA from working.
> I'm not sure if there's something wrong with our configs because this happens
> every time for finished job (we only tested with wordcount.jar though). I'm
> aware of FLINK-10011 and FLINK-10029, but unlike FLINK-10011 this happens
> every time, rendering HA mode un-useable for us.
> Any idea what might cause this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl]
add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r209682805
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java
##
@@ -163,80 +163,188 @@ public static void setSSLVerifyHostname(Configuration
sslConfig, SSLParameters s
}
/**
-* Creates the SSL Context for the client if SSL is configured.
+* Configuration settings and key/trustmanager instances to set up an
SSL client connection.
+*/
+ public static class SSLClientConfiguration {
Review comment:
good idea - that makes the change even smaller...well, at least the
important parts of the change ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-9878) IO worker threads BLOCKED on SSL Session Cache while CMS full gc
[
https://issues.apache.org/jira/browse/FLINK-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578629#comment-16578629
]
ASF GitHub Bot commented on FLINK-9878:
---
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl]
add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r209682805
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java
##
@@ -163,80 +163,188 @@ public static void setSSLVerifyHostname(Configuration
sslConfig, SSLParameters s
}
/**
-* Creates the SSL Context for the client if SSL is configured.
+* Configuration settings and key/trustmanager instances to set up an
SSL client connection.
+*/
+ public static class SSLClientConfiguration {
Review comment:
good idea - that makes the change even smaller...well, at least the
important parts of the change ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> IO worker threads BLOCKED on SSL Session Cache while CMS full gc
>
>
> Key: FLINK-9878
> URL: https://issues.apache.org/jira/browse/FLINK-9878
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.5.1, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.3, 1.6.1, 1.7.0
>
>
> According to https://github.com/netty/netty/issues/832, there is a JDK issue
> during garbage collection when the SSL session cache is not limited. We
> should allow the user to configure this and further (advanced) SSL parameters
> for fine-tuning to fix this and similar issues. In particular, the following
> parameters should be configurable:
> - SSL session cache size
> - SSL session timeout
> - SSL handshake timeout
> - SSL close notify flush timeout
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9977) Refine the docs for Table/SQL built-in functions
[
https://issues.apache.org/jira/browse/FLINK-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578630#comment-16578630
]
ASF GitHub Bot commented on FLINK-9977:
---
fhueske commented on issue #6535: [FLINK-9977] [table][doc] Refine the
SQL/Table built-in function docs
URL: https://github.com/apache/flink/pull/6535#issuecomment-412593462
Thanks for the update @xccui.
+1 to merge
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine the docs for Table/SQL built-in functions
>
>
> Key: FLINK-9977
> URL: https://issues.apache.org/jira/browse/FLINK-9977
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Minor
> Labels: pull-request-available
> Attachments: Java.jpg, SQL.jpg, Scala.jpg
>
>
> There exist some syntax errors or inconsistencies in documents and Scala docs
> of the Table/SQL built-in functions. This issue aims to make some
> improvements to them.
> Also, according to FLINK-10103, we should use single quotes to express
> strings in SQL. For example, CONCAT("AA", "BB", "CC") should be replaced with
> CONCAT('AA', 'BB', 'CC').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] fhueske commented on issue #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs
fhueske commented on issue #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs URL: https://github.com/apache/flink/pull/6535#issuecomment-412593462 Thanks for the update @xccui. +1 to merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10066) Keep only archived version of previous executions
[
https://issues.apache.org/jira/browse/FLINK-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578712#comment-16578712
]
ASF GitHub Bot commented on FLINK-10066:
asfgit closed pull request #6500: [FLINK-10066] Keep only archived version of
previous executions
URL: https://github.com/apache/flink/pull/6500
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
index 4b1c62fe707..ab8c94c0188 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
@@ -18,6 +18,7 @@
package org.apache.flink.runtime.executiongraph;
import org.apache.flink.runtime.accumulators.StringifiedAccumulatorResult;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
import org.apache.flink.runtime.execution.ExecutionState;
import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
import org.apache.flink.util.ExceptionUtils;
@@ -40,6 +41,8 @@
private final TaskManagerLocation assignedResourceLocation; // for the
archived execution
+ private final AllocationID assignedAllocationID;
+
/* Continuously updated map of user-defined accumulators */
private final StringifiedAccumulatorResult[] userAccumulators;
@@ -48,21 +51,24 @@
private final IOMetrics ioMetrics;
public ArchivedExecution(Execution execution) {
- this.userAccumulators =
execution.getUserAccumulatorsStringified();
- this.attemptId = execution.getAttemptId();
- this.attemptNumber = execution.getAttemptNumber();
- this.stateTimestamps = execution.getStateTimestamps();
- this.parallelSubtaskIndex =
execution.getVertex().getParallelSubtaskIndex();
- this.state = execution.getState();
- this.failureCause =
ExceptionUtils.stringifyException(execution.getFailureCause());
- this.assignedResourceLocation =
execution.getAssignedResourceLocation();
- this.ioMetrics = execution.getIOMetrics();
+ this(
+ execution.getUserAccumulatorsStringified(),
+ execution.getIOMetrics(),
+ execution.getAttemptId(),
+ execution.getAttemptNumber(),
+ execution.getState(),
+
ExceptionUtils.stringifyException(execution.getFailureCause()),
+ execution.getAssignedResourceLocation(),
+ execution.getAssignedAllocationID(),
+ execution.getVertex().getParallelSubtaskIndex(),
+ execution.getStateTimestamps());
}
public ArchivedExecution(
StringifiedAccumulatorResult[] userAccumulators,
IOMetrics ioMetrics,
ExecutionAttemptID attemptId, int attemptNumber,
ExecutionState state, String failureCause,
- TaskManagerLocation assignedResourceLocation, int
parallelSubtaskIndex, long[] stateTimestamps) {
+ TaskManagerLocation assignedResourceLocation,
AllocationID assignedAllocationID, int parallelSubtaskIndex,
+ long[] stateTimestamps) {
this.userAccumulators = userAccumulators;
this.ioMetrics = ioMetrics;
this.failureCause = failureCause;
@@ -72,6 +78,7 @@ public ArchivedExecution(
this.state = state;
this.stateTimestamps = stateTimestamps;
this.parallelSubtaskIndex = parallelSubtaskIndex;
+ this.assignedAllocationID = assignedAllocationID;
}
//
@@ -103,6 +110,10 @@ public TaskManagerLocation getAssignedResourceLocation() {
return assignedResourceLocation;
}
+ public AllocationID getAssignedAllocationID() {
+ return assignedAllocationID;
+ }
+
@Override
public String getFailureCauseAsString() {
return failureCause;
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionVertex.java
index 36669d34be7..04efa048fb6 100644
---
[GitHub] bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition
bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#discussion_r209712274 ## File path: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/PartitionerITCase.java ## @@ -228,9 +229,29 @@ private static void verifyRebalancePartitioning(List> re new Tuple2(2, "c"), new Tuple2(0, "a")); - assertEquals( - new HashSet>(expected), - new HashSet>(rebalancePartitionResult)); + List> expected1 = Arrays.asList( + new Tuple2(1, "a"), + new Tuple2(2, "b"), + new Tuple2(0, "b"), + new Tuple2(1, "a"), + new Tuple2(2, "a"), + new Tuple2(0, "c"), + new Tuple2(1, "a")); + + List> expected2 = Arrays.asList( + new Tuple2(2, "a"), + new Tuple2(0, "b"), + new Tuple2(1, "b"), + new Tuple2(2, "a"), + new Tuple2(0, "a"), + new Tuple2(1, "c"), + new Tuple2(2, "a")); + + assertTrue( Review comment: to be more robust, shall this assert that exactly one case match and the other two unmatch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition
bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#discussion_r209712274 ## File path: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/PartitionerITCase.java ## @@ -228,9 +229,29 @@ private static void verifyRebalancePartitioning(List> re new Tuple2(2, "c"), new Tuple2(0, "a")); - assertEquals( - new HashSet>(expected), - new HashSet>(rebalancePartitionResult)); + List> expected1 = Arrays.asList( + new Tuple2(1, "a"), + new Tuple2(2, "b"), + new Tuple2(0, "b"), + new Tuple2(1, "a"), + new Tuple2(2, "a"), + new Tuple2(0, "c"), + new Tuple2(1, "a")); + + List> expected2 = Arrays.asList( + new Tuple2(2, "a"), + new Tuple2(0, "b"), + new Tuple2(1, "b"), + new Tuple2(2, "a"), + new Tuple2(0, "a"), + new Tuple2(1, "c"), + new Tuple2(2, "a")); + + assertTrue( Review comment: to be more robust for people to modify `expectedX`, shall this assert that exactly one case match and the other two unmatch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-8532) RebalancePartitioner should use Random value for its first partition
[ https://issues.apache.org/jira/browse/FLINK-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578760#comment-16578760 ] ASF GitHub Bot commented on FLINK-8532: --- bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#discussion_r209712274 ## File path: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/PartitionerITCase.java ## @@ -228,9 +229,29 @@ private static void verifyRebalancePartitioning(List> re new Tuple2(2, "c"), new Tuple2(0, "a")); - assertEquals( - new HashSet>(expected), - new HashSet>(rebalancePartitionResult)); + List> expected1 = Arrays.asList( + new Tuple2(1, "a"), + new Tuple2(2, "b"), + new Tuple2(0, "b"), + new Tuple2(1, "a"), + new Tuple2(2, "a"), + new Tuple2(0, "c"), + new Tuple2(1, "a")); + + List> expected2 = Arrays.asList( + new Tuple2(2, "a"), + new Tuple2(0, "b"), + new Tuple2(1, "b"), + new Tuple2(2, "a"), + new Tuple2(0, "a"), + new Tuple2(1, "c"), + new Tuple2(2, "a")); + + assertTrue( Review comment: to be more robust for people to modify `expectedX`, shall this assert that exactly one case match and the other two unmatch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RebalancePartitioner should use Random value for its first partition > > > Key: FLINK-8532 > URL: https://issues.apache.org/jira/browse/FLINK-8532 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Yuta Morisawa >Priority: Minor > Labels: pull-request-available > > In some conditions, RebalancePartitioner doesn't balance data correctly > because it use the same value for selecting next operators. > RebalancePartitioner initializes its partition id using the same value in > every threads, so it indeed balances data, but at one moment the amount of > data in each operator is skew. > Particularly, when the data rate of former operators is equal , data skew > becomes severe. > > > Example: > Consider a simple operator chain. > -> map1 -> rebalance -> map2 -> > Each map operator(map1, map2) contains three subtasks(subtask 1, 2, 3, 4, 5, > 6). > map1 map2 > st1 st4 > st2 st5 > st3 st6 > > At the beginning, every subtasks in map1 sends data to st4 in map2 because > they use the same initial parition id. > Next time the map1 receive data st1,2,3 send data to st5 because they > increment its partition id when they processed former data. > In my environment, it takes twice the time to process data when I use > RebalancePartitioner as long as I use other partitioners(rescale, keyby). > > To solve this problem, in my opinion, RebalancePartitioner should use its own > operator id for the initial value. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8532) RebalancePartitioner should use Random value for its first partition
[ https://issues.apache.org/jira/browse/FLINK-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578757#comment-16578757 ] ASF GitHub Bot commented on FLINK-8532: --- bowenli86 commented on a change in pull request #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#discussion_r209712274 ## File path: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/PartitionerITCase.java ## @@ -228,9 +229,29 @@ private static void verifyRebalancePartitioning(List> re new Tuple2(2, "c"), new Tuple2(0, "a")); - assertEquals( - new HashSet>(expected), - new HashSet>(rebalancePartitionResult)); + List> expected1 = Arrays.asList( + new Tuple2(1, "a"), + new Tuple2(2, "b"), + new Tuple2(0, "b"), + new Tuple2(1, "a"), + new Tuple2(2, "a"), + new Tuple2(0, "c"), + new Tuple2(1, "a")); + + List> expected2 = Arrays.asList( + new Tuple2(2, "a"), + new Tuple2(0, "b"), + new Tuple2(1, "b"), + new Tuple2(2, "a"), + new Tuple2(0, "a"), + new Tuple2(1, "c"), + new Tuple2(2, "a")); + + assertTrue( Review comment: to be more robust, shall this assert that exactly one case match and the other two unmatch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RebalancePartitioner should use Random value for its first partition > > > Key: FLINK-8532 > URL: https://issues.apache.org/jira/browse/FLINK-8532 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Yuta Morisawa >Priority: Minor > Labels: pull-request-available > > In some conditions, RebalancePartitioner doesn't balance data correctly > because it use the same value for selecting next operators. > RebalancePartitioner initializes its partition id using the same value in > every threads, so it indeed balances data, but at one moment the amount of > data in each operator is skew. > Particularly, when the data rate of former operators is equal , data skew > becomes severe. > > > Example: > Consider a simple operator chain. > -> map1 -> rebalance -> map2 -> > Each map operator(map1, map2) contains three subtasks(subtask 1, 2, 3, 4, 5, > 6). > map1 map2 > st1 st4 > st2 st5 > st3 st6 > > At the beginning, every subtasks in map1 sends data to st4 in map2 because > they use the same initial parition id. > Next time the map1 receive data st1,2,3 send data to st5 because they > increment its partition id when they processed former data. > In my environment, it takes twice the time to process data when I use > RebalancePartitioner as long as I use other partitioners(rescale, keyby). > > To solve this problem, in my opinion, RebalancePartitioner should use its own > operator id for the initial value. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578759#comment-16578759
]
ASF GitHub Bot commented on FLINK-6437:
---
StephanEwen commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-412618428
Sure, so the compatibility story needs a more detailed design, and that
needs to be discussed before looking at concrete code. Agreed.
On whether we want this feature or not - the discussion in Jira was in favor
of doing this. Curious what is the reasoning for the push back now, @zentol ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] StephanEwen commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
StephanEwen commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-412618428 Sure, so the compatibility story needs a more detailed design, and that needs to be discussed before looking at concrete code. Agreed. On whether we want this feature or not - the discussion in Jira was in favor of doing this. Curious what is the reasoning for the push back now, @zentol ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#discussion_r209690309 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java ## @@ -52,6 +56,9 @@ private Bootstrap bootstrap; Review comment: out of scope of this PR - there's also more around this package, if you wanted to mark/change these accordingly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#discussion_r209690309 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java ## @@ -52,6 +56,9 @@ private Bootstrap bootstrap; Review comment: out of scope of this PR - there's also even more around this package, if you wanted to mark/change these accordingly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Guibo-Pan commented on issue #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition
Guibo-Pan commented on issue #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#issuecomment-412602068 @yanghua thanks for your suggestion, I have made an update. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10019) Fix Composite getResultType of UDF cannot be chained with other operators
[
https://issues.apache.org/jira/browse/FLINK-10019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578793#comment-16578793
]
Rong Rong commented on FLINK-10019:
---
Dug a little deeper in Calcite and found that in:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/type/InferTypes.java#L68
Seems like if the return type is inferred as a record, or {{isStruct()}}, "it
must have the same number of fields as the number of operands." which clearly
is not the case here since the following expression: {{AS(func(a), "myRow")}}
only passes over the {{func(a)}} for type inference, but not the alias
{{"myRow"}}
> Fix Composite getResultType of UDF cannot be chained with other operators
> -
>
> Key: FLINK-10019
> URL: https://issues.apache.org/jira/browse/FLINK-10019
> Project: Flink
> Issue Type: Bug
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> If explicitly return a CompositeType in {{udf.getResultType}}, will result in
> some failures in chained operators.
> For example: consider a simple UDF,
> {code:scala}
> object Func extends ScalarFunction {
> def eval(row: Row): Row = {
> row
> }
> override def getParameterTypes(signature: Array[Class[_]]):
> Array[TypeInformation[_]] =
> Array(Types.ROW(Types.INT))
> override def getResultType(signature: Array[Class[_]]): TypeInformation[_] =
> Types.ROW(Types.INT)
> }
> {code}
> This should work perfectly since it's just a simple pass through, however
> {code:scala}
> @Test
> def testRowType(): Unit = {
> val data = List(
> Row.of(Row.of(12.asInstanceOf[Integer]), "1")
> )
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromCollection(data)(Types.ROW(Types.ROW(Types.INT),
> Types.STRING))
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val table = stream.toTable(tEnv, 'a, 'b)
> tEnv.registerFunction("func", Func)
> tEnv.registerTable("t", table)
> // This works perfectly
> val result1 = tEnv.sqlQuery("SELECT func(a) FROM t").toAppendStream[Row]
> result1.addSink(new StreamITCase.StringSink[Row])
> // This throws exception
> val result2 = tEnv.sqlQuery("SELECT func(a) as myRow FROM
> t").toAppendStream[Row]
> result2.addSink(new StreamITCase.StringSink[Row])
> env.execute()
> }
> {code}
> Exception code:
> {code:java}
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:41)
> at
> org.apache.calcite.sql.type.InferTypes$2.inferOperandTypes(InferTypes.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:1777)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:459)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:349)
> ...
> {code}
> This is due to the fact that Calcite inferOperandTypes does not expect to
> infer a struct RelDataType.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10019) Fix Composite getResultType of UDF cannot be chained with other operators
[
https://issues.apache.org/jira/browse/FLINK-10019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578793#comment-16578793
]
Rong Rong edited comment on FLINK-10019 at 8/13/18 7:00 PM:
Dug a little deeper in Calcite and found that in:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/type/InferTypes.java#L68
Seems like if the return type is inferred as a record, or {{isStruct()}}, "it
must have the same number of fields as the number of operands." which clearly
is not the case here since the following expression: **{{AS(func(a),
"myRow")}}** only passes over the **{{func(a)}}** for type inference, but not
the alias **{{"myRow"}}**
was (Author: walterddr):
Dug a little deeper in Calcite and found that in:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/type/InferTypes.java#L68
Seems like if the return type is inferred as a record, or {{isStruct()}}, "it
must have the same number of fields as the number of operands." which clearly
is not the case here since the following expression: {{AS(func(a), "myRow")}}
only passes over the {{func(a)}} for type inference, but not the alias
{{"myRow"}}
> Fix Composite getResultType of UDF cannot be chained with other operators
> -
>
> Key: FLINK-10019
> URL: https://issues.apache.org/jira/browse/FLINK-10019
> Project: Flink
> Issue Type: Bug
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> If explicitly return a CompositeType in {{udf.getResultType}}, will result in
> some failures in chained operators.
> For example: consider a simple UDF,
> {code:scala}
> object Func extends ScalarFunction {
> def eval(row: Row): Row = {
> row
> }
> override def getParameterTypes(signature: Array[Class[_]]):
> Array[TypeInformation[_]] =
> Array(Types.ROW(Types.INT))
> override def getResultType(signature: Array[Class[_]]): TypeInformation[_] =
> Types.ROW(Types.INT)
> }
> {code}
> This should work perfectly since it's just a simple pass through, however
> {code:scala}
> @Test
> def testRowType(): Unit = {
> val data = List(
> Row.of(Row.of(12.asInstanceOf[Integer]), "1")
> )
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromCollection(data)(Types.ROW(Types.ROW(Types.INT),
> Types.STRING))
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val table = stream.toTable(tEnv, 'a, 'b)
> tEnv.registerFunction("func", Func)
> tEnv.registerTable("t", table)
> // This works perfectly
> val result1 = tEnv.sqlQuery("SELECT func(a) FROM t").toAppendStream[Row]
> result1.addSink(new StreamITCase.StringSink[Row])
> // This throws exception
> val result2 = tEnv.sqlQuery("SELECT func(a) as myRow FROM
> t").toAppendStream[Row]
> result2.addSink(new StreamITCase.StringSink[Row])
> env.execute()
> }
> {code}
> Exception code:
> {code:java}
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:41)
> at
> org.apache.calcite.sql.type.InferTypes$2.inferOperandTypes(InferTypes.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:1777)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:459)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:349)
> ...
> {code}
> This is due to the fact that Calcite inferOperandTypes does not expect to
> infer a struct RelDataType.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9878) IO worker threads BLOCKED on SSL Session Cache while CMS full gc
[
https://issues.apache.org/jira/browse/FLINK-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578646#comment-16578646
]
ASF GitHub Bot commented on FLINK-9878:
---
NicoK commented on a change in pull request #6355: [FLINK-9878][network][ssl]
add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r209690587
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java
##
@@ -175,7 +183,6 @@ ChannelFuture connect(final InetSocketAddress
serverSocketAddress) {
bootstrap.handler(new ChannelInitializer() {
@Override
public void initChannel(SocketChannel channel) throws
Exception {
-
// SSL handler should be added first in the
pipeline
if (clientSSLContext != null) {
Review comment:
if SSL is disabled, for example
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> IO worker threads BLOCKED on SSL Session Cache while CMS full gc
>
>
> Key: FLINK-9878
> URL: https://issues.apache.org/jira/browse/FLINK-9878
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.5.1, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.3, 1.6.1, 1.7.0
>
>
> According to https://github.com/netty/netty/issues/832, there is a JDK issue
> during garbage collection when the SSL session cache is not limited. We
> should allow the user to configure this and further (advanced) SSL parameters
> for fine-tuning to fix this and similar issues. In particular, the following
> parameters should be configurable:
> - SSL session cache size
> - SSL session timeout
> - SSL handshake timeout
> - SSL close notify flush timeout
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8532) RebalancePartitioner should use Random value for its first partition
[ https://issues.apache.org/jira/browse/FLINK-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578678#comment-16578678 ] ASF GitHub Bot commented on FLINK-8532: --- Guibo-Pan commented on issue #6544: [FLINK-8532] [Streaming] modify RebalancePartitioner to use a random partition as its first partition URL: https://github.com/apache/flink/pull/6544#issuecomment-412602068 @yanghua thanks for your suggestion, I have made an update. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RebalancePartitioner should use Random value for its first partition > > > Key: FLINK-8532 > URL: https://issues.apache.org/jira/browse/FLINK-8532 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Yuta Morisawa >Priority: Minor > Labels: pull-request-available > > In some conditions, RebalancePartitioner doesn't balance data correctly > because it use the same value for selecting next operators. > RebalancePartitioner initializes its partition id using the same value in > every threads, so it indeed balances data, but at one moment the amount of > data in each operator is skew. > Particularly, when the data rate of former operators is equal , data skew > becomes severe. > > > Example: > Consider a simple operator chain. > -> map1 -> rebalance -> map2 -> > Each map operator(map1, map2) contains three subtasks(subtask 1, 2, 3, 4, 5, > 6). > map1 map2 > st1 st4 > st2 st5 > st3 st6 > > At the beginning, every subtasks in map1 sends data to st4 in map2 because > they use the same initial parition id. > Next time the map1 receive data st1,2,3 send data to st5 because they > increment its partition id when they processed former data. > In my environment, it takes twice the time to process data when I use > RebalancePartitioner as long as I use other partitioners(rescale, keyby). > > To solve this problem, in my opinion, RebalancePartitioner should use its own > operator id for the initial value. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9999) Add ISNUMERIC supported in Table API/SQL
[
https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578781#comment-16578781
]
ASF GitHub Bot commented on FLINK-:
---
walterddr commented on a change in pull request #6473: [FLINK-] [table] Add
ISNUMERIC supported in Table API/SQL
URL: https://github.com/apache/flink/pull/6473#discussion_r209719307
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
##
@@ -450,6 +450,63 @@ class ScalarFunctionsTest extends ScalarTypesTestBase {
"")
}
+ @Test
+ def testIsNumeric(): Unit = {
Review comment:
Originally what I meant is, since this function only supports
string/varchar, let's have a test that specifies `ISNUMERIC(1L)` throws
`ValidationException`.
Regarding the usage of this in general, I think this is useful to chained
with many other operators with strict type constrains. such as `CASE WHEN
ISNUMERIC(...) THEN ... ELSE ...`, where the `THEN` clause requires some strict
numeric vaues. That's why I was wondering if we should do a better support
beyond just STRING/VARCHAR type.
I will comment on the JIRA actually. thanks for bring this up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ISNUMERIC supported in Table API/SQL
>
>
> Key: FLINK-
> URL: https://issues.apache.org/jira/browse/FLINK-
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> ISNUMERIC function used to verify a expression is a valid numberic type.
> documentation :
> https://docs.microsoft.com/en-us/sql/t-sql/functions/isnumeric-transact-sql?view=sql-server-2017
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] walterddr commented on a change in pull request #6473: [FLINK-9999] [table] Add ISNUMERIC supported in Table API/SQL
walterddr commented on a change in pull request #6473: [FLINK-] [table] Add
ISNUMERIC supported in Table API/SQL
URL: https://github.com/apache/flink/pull/6473#discussion_r209719307
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
##
@@ -450,6 +450,63 @@ class ScalarFunctionsTest extends ScalarTypesTestBase {
"")
}
+ @Test
+ def testIsNumeric(): Unit = {
Review comment:
Originally what I meant is, since this function only supports
string/varchar, let's have a test that specifies `ISNUMERIC(1L)` throws
`ValidationException`.
Regarding the usage of this in general, I think this is useful to chained
with many other operators with strict type constrains. such as `CASE WHEN
ISNUMERIC(...) THEN ... ELSE ...`, where the `THEN` clause requires some strict
numeric vaues. That's why I was wondering if we should do a better support
beyond just STRING/VARCHAR type.
I will comment on the JIRA actually. thanks for bring this up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] NicoK commented on issue #6355: [FLINK-9878][network][ssl] add more low-level ssl options
NicoK commented on issue #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#issuecomment-412597634 I pushed a rework of this PR which has a lighter footprint on the changes in SSLUtils by using a wrapper around `SSLContext` as @pnowojski suggested. I kept all existing logic though, including the `@Nullable` fields (vs. `Optional`) for these reasons: 1) there are already conflicts when applying this to `release-1.6` and I'd like to keep the footprint small (some of the suggestions already make the diff bigger) 2) there are several `null` checks which would need refactoring 3) this seems to be out of scope of this PR, especially since no nullable field is added (any more) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9878) IO worker threads BLOCKED on SSL Session Cache while CMS full gc
[ https://issues.apache.org/jira/browse/FLINK-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578653#comment-16578653 ] ASF GitHub Bot commented on FLINK-9878: --- NicoK commented on issue #6355: [FLINK-9878][network][ssl] add more low-level ssl options URL: https://github.com/apache/flink/pull/6355#issuecomment-412597634 I pushed a rework of this PR which has a lighter footprint on the changes in SSLUtils by using a wrapper around `SSLContext` as @pnowojski suggested. I kept all existing logic though, including the `@Nullable` fields (vs. `Optional`) for these reasons: 1) there are already conflicts when applying this to `release-1.6` and I'd like to keep the footprint small (some of the suggestions already make the diff bigger) 2) there are several `null` checks which would need refactoring 3) this seems to be out of scope of this PR, especially since no nullable field is added (any more) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IO worker threads BLOCKED on SSL Session Cache while CMS full gc > > > Key: FLINK-9878 > URL: https://issues.apache.org/jira/browse/FLINK-9878 > Project: Flink > Issue Type: Bug > Components: Network >Affects Versions: 1.5.0, 1.5.1, 1.6.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Labels: pull-request-available > Fix For: 1.5.3, 1.6.1, 1.7.0 > > > According to https://github.com/netty/netty/issues/832, there is a JDK issue > during garbage collection when the SSL session cache is not limited. We > should allow the user to configure this and further (advanced) SSL parameters > for fine-tuning to fix this and similar issues. In particular, the following > parameters should be configurable: > - SSL session cache size > - SSL session timeout > - SSL handshake timeout > - SSL close notify flush timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-412622673 well I still like the idea of separate config files, but the JIRA discussion happened more than a year ago, _before we even had released the HistoryServer_. Now we have to think about backwards compatibility and will thus naturally end up adding complexity. I'm not sure if this is really worth it, especially so since this issue has never been raised again since the HS was released. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10135) The JobManager doesn't report the cluster-level metrics
Joey Echeverria created FLINK-10135:
---
Summary: The JobManager doesn't report the cluster-level metrics
Key: FLINK-10135
URL: https://issues.apache.org/jira/browse/FLINK-10135
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 1.5.0
Reporter: Joey Echeverria
In [the documentation for
metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
in the Flink 1.5.0 release, it says that the following metrics are reported by
the JobManager:
{noformat}
numRegisteredTaskManagers
numRunningJobs
taskSlotsAvailable
taskSlotsTotal
{noformat}
In the job manager REST endpoint
({{http://:8081/jobmanager/metrics}}), those metrics don't appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun commented on issue #6339: [FLINK-9859] [runtime] More Akka config
TisonKun commented on issue #6339: [FLINK-9859] [runtime] More Akka config URL: https://github.com/apache/flink/pull/6339#issuecomment-412743884 ping @tillrohrmann :-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9859) More Akka config
[ https://issues.apache.org/jira/browse/FLINK-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579207#comment-16579207 ] ASF GitHub Bot commented on FLINK-9859: --- TisonKun commented on issue #6339: [FLINK-9859] [runtime] More Akka config URL: https://github.com/apache/flink/pull/6339#issuecomment-412743884 ping @tillrohrmann :-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > More Akka config > > > Key: FLINK-9859 > URL: https://issues.apache.org/jira/browse/FLINK-9859 > Project: Flink > Issue Type: Improvement > Components: Local Runtime >Affects Versions: 1.5.1 >Reporter: 陈梓立 >Assignee: 陈梓立 >Priority: Major > Labels: pull-request-available > Fix For: 1.5.3 > > > Add more akka config options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10127) Add TypeInformation and serializers for JDK8 Instant
[
https://issues.apache.org/jira/browse/FLINK-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578935#comment-16578935
]
ASF GitHub Bot commented on FLINK-10127:
alexeyt820 opened a new pull request #6549: [FLINK-10127] Add Instant to basic
types
URL: https://github.com/apache/flink/pull/6549
## What is the purpose of the change
This pull requests add JDK8 Instant type as basic type to Flink type system
## Brief change log
- *InstantSerializer* added
- *InstantComparator* added
- *BasicTypeInfo* modified to include *INSTANT_TYPE_INFO*
- "Types" modified to include *INSTANT*
## Verifying this change
This change added tests and can be verified as follows:
- *Added unit tests for InstantSerializer and InstantComparator*
- *Modified BasicTypeInfoTest to include Instant*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: yes
- The runtime per-record code paths (performance sensitive): don't know
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: don't know
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
-
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/types_serialization.html#flinks-typeinformation-class
should be modified to include Instant
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add TypeInformation and serializers for JDK8 Instant
> -
>
> Key: FLINK-10127
> URL: https://issues.apache.org/jira/browse/FLINK-10127
> Project: Flink
> Issue Type: Improvement
>Reporter: Alexey Trenikhin
>Priority: Minor
> Labels: pull-request-available
>
> Currently Flink's basic types include all Java primitives and their boxed
> form, plus {{void}}, {{String}}, {{Date}}, {{BigDecimal}}, and
> {{BigInteger}}. New JDK8 Instance type should be added as well
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10127) Add TypeInformation and serializers for JDK8 Instant
[
https://issues.apache.org/jira/browse/FLINK-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10127:
---
Labels: pull-request-available (was: )
> Add TypeInformation and serializers for JDK8 Instant
> -
>
> Key: FLINK-10127
> URL: https://issues.apache.org/jira/browse/FLINK-10127
> Project: Flink
> Issue Type: Improvement
>Reporter: Alexey Trenikhin
>Priority: Minor
> Labels: pull-request-available
>
> Currently Flink's basic types include all Java primitives and their boxed
> form, plus {{void}}, {{String}}, {{Date}}, {{BigDecimal}}, and
> {{BigInteger}}. New JDK8 Instance type should be added as well
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10135) The JobManager doesn't report the cluster-level metrics
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang reassigned FLINK-10135:
Assignee: vinoyang
> The JobManager doesn't report the cluster-level metrics
> ---
>
> Key: FLINK-10135
> URL: https://issues.apache.org/jira/browse/FLINK-10135
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.5.0
>Reporter: Joey Echeverria
>Assignee: vinoyang
>Priority: Major
>
> In [the documentation for
> metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
> in the Flink 1.5.0 release, it says that the following metrics are reported
> by the JobManager:
> {noformat}
> numRegisteredTaskManagers
> numRunningJobs
> taskSlotsAvailable
> taskSlotsTotal
> {noformat}
> In the job manager REST endpoint
> ({{http://:8081/jobmanager/metrics}}), those metrics don't
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] alexeyt820 opened a new pull request #6549: [FLINK-10127] Add Instant to basic types
alexeyt820 opened a new pull request #6549: [FLINK-10127] Add Instant to basic types URL: https://github.com/apache/flink/pull/6549 ## What is the purpose of the change This pull requests add JDK8 Instant type as basic type to Flink type system ## Brief change log - *InstantSerializer* added - *InstantComparator* added - *BasicTypeInfo* modified to include *INSTANT_TYPE_INFO* - "Types" modified to include *INSTANT* ## Verifying this change This change added tests and can be verified as follows: - *Added unit tests for InstantSerializer and InstantComparator* - *Modified BasicTypeInfoTest to include Instant* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: yes - The runtime per-record code paths (performance sensitive): don't know - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: don't know - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/types_serialization.html#flinks-typeinformation-class should be modified to include Instant This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xccui commented on issue #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs
xccui commented on issue #6535: [FLINK-9977] [table][doc] Refine the SQL/Table built-in function docs URL: https://github.com/apache/flink/pull/6535#issuecomment-412717935 Thanks for the review, @fhueske. Will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9977) Refine the docs for Table/SQL built-in functions
[
https://issues.apache.org/jira/browse/FLINK-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579094#comment-16579094
]
ASF GitHub Bot commented on FLINK-9977:
---
xccui commented on issue #6535: [FLINK-9977] [table][doc] Refine the SQL/Table
built-in function docs
URL: https://github.com/apache/flink/pull/6535#issuecomment-412717935
Thanks for the review, @fhueske. Will merge this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine the docs for Table/SQL built-in functions
>
>
> Key: FLINK-9977
> URL: https://issues.apache.org/jira/browse/FLINK-9977
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Minor
> Labels: pull-request-available
> Attachments: Java.jpg, SQL.jpg, Scala.jpg
>
>
> There exist some syntax errors or inconsistencies in documents and Scala docs
> of the Table/SQL built-in functions. This issue aims to make some
> improvements to them.
> Also, according to FLINK-10103, we should use single quotes to express
> strings in SQL. For example, CONCAT("AA", "BB", "CC") should be replaced with
> CONCAT('AA', 'BB', 'CC').
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579156#comment-16579156
]
ASF GitHub Bot commented on FLINK-6437:
---
yanghua commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-412736043
hi @StephanEwen @zentol ,
Currently, the history server runs as a standalone JVM process, which means
it is a separate component. I think the configuration split is reasonable. If I
only need to start the history server then I don't rely on other
configurations, and I don't need to pay attention to other configurations.
Regarding the discussion of compatibility, I basically agree with @zentol
's point of view.
Basic on the existing implementation, I think we can add more log warnings
in the new method of `GlobalConfiguration`. The general idea is as follows:
* Judge the new configuration file, if it does not exist, I will add a log
to inform the user that he is using the old configuration;
* If the new configuration and the old configuration about history server
exist at the same time, then we will warn the user that there are two
configurations and we will base on the `flink-historyserver-conf.yaml`;
In addition, I will give a log warning when the configuration file is loaded
by the `HistoryServer#main` method, for example :
> the current configuration about history server in `flink-conf.yaml` is
only for backward compatibility, it is recommended that they enable the new
configuration file `flink-historyserver-conf.yaml`.
In addition, I will comment out the configuration items in the
`flink-historyserver-conf.yaml`, considering that we have done so many guides,
I believe users will be more likely to accept the correct guidelines.
Even if it is based on a configuration already in `flink-conf.yaml`, the
current implementation will still work. But if we say that there are
configurations in both files, we seem to have no better way than to give a
warning "based on `flink-historyserver-conf.yaml`".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579157#comment-16579157
]
ASF GitHub Bot commented on FLINK-6437:
---
yanghua edited a comment on issue #6542: [FLINK-6437][History Server] Move
history server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-412736043
hi @StephanEwen @zentol ,
Currently, the history server runs as a single JVM process, which means it
is a separate component. I think the configuration split is reasonable. If I
only need to start the history server then I don't rely on other
configurations, and I don't need to pay attention to other configurations.
Regarding the discussion of compatibility, I basically agree with @zentol
's point of view.
Basic on the existing implementation, I think we can add more log warnings
in the new method of `GlobalConfiguration`. The general idea is as follows:
* Judge the new configuration file, if it does not exist, I will add a log
to inform the user that he is using the old configuration;
* If the new configuration and the old configuration about history server
exist at the same time, then we will warn the user that there are two
configurations and we will base on the `flink-historyserver-conf.yaml`;
In addition, I will give a log warning when the configuration file is loaded
by the `HistoryServer#main` method, for example :
> the current configuration about history server in `flink-conf.yaml` is
only for backward compatibility, it is recommended that they enable the new
configuration file `flink-historyserver-conf.yaml`.
In addition, I will comment out the configuration items in the
`flink-historyserver-conf.yaml`, considering that we have done so many guides,
I believe users will be more likely to accept the correct guidelines.
Even if it is based on a configuration already in `flink-conf.yaml`, the
current implementation will still work. But if we say that there are
configurations in both files, we seem to have no better way than to give a
warning "based on `flink-historyserver-conf.yaml`".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua edited a comment on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
yanghua edited a comment on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-412736043 hi @StephanEwen @zentol , Currently, the history server runs as a single JVM process, which means it is a separate component. I think the configuration split is reasonable. If I only need to start the history server then I don't rely on other configurations, and I don't need to pay attention to other configurations. Regarding the discussion of compatibility, I basically agree with @zentol 's point of view. Basic on the existing implementation, I think we can add more log warnings in the new method of `GlobalConfiguration`. The general idea is as follows: * Judge the new configuration file, if it does not exist, I will add a log to inform the user that he is using the old configuration; * If the new configuration and the old configuration about history server exist at the same time, then we will warn the user that there are two configurations and we will base on the `flink-historyserver-conf.yaml`; In addition, I will give a log warning when the configuration file is loaded by the `HistoryServer#main` method, for example : > the current configuration about history server in `flink-conf.yaml` is only for backward compatibility, it is recommended that they enable the new configuration file `flink-historyserver-conf.yaml`. In addition, I will comment out the configuration items in the `flink-historyserver-conf.yaml`, considering that we have done so many guides, I believe users will be more likely to accept the correct guidelines. Even if it is based on a configuration already in `flink-conf.yaml`, the current implementation will still work. But if we say that there are configurations in both files, we seem to have no better way than to give a warning "based on `flink-historyserver-conf.yaml`". This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
yanghua commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-412736043 hi @StephanEwen @zentol , Currently, the history server runs as a standalone JVM process, which means it is a separate component. I think the configuration split is reasonable. If I only need to start the history server then I don't rely on other configurations, and I don't need to pay attention to other configurations. Regarding the discussion of compatibility, I basically agree with @zentol 's point of view. Basic on the existing implementation, I think we can add more log warnings in the new method of `GlobalConfiguration`. The general idea is as follows: * Judge the new configuration file, if it does not exist, I will add a log to inform the user that he is using the old configuration; * If the new configuration and the old configuration about history server exist at the same time, then we will warn the user that there are two configurations and we will base on the `flink-historyserver-conf.yaml`; In addition, I will give a log warning when the configuration file is loaded by the `HistoryServer#main` method, for example : > the current configuration about history server in `flink-conf.yaml` is only for backward compatibility, it is recommended that they enable the new configuration file `flink-historyserver-conf.yaml`. In addition, I will comment out the configuration items in the `flink-historyserver-conf.yaml`, considering that we have done so many guides, I believe users will be more likely to accept the correct guidelines. Even if it is based on a configuration already in `flink-conf.yaml`, the current implementation will still work. But if we say that there are configurations in both files, we seem to have no better way than to give a warning "based on `flink-historyserver-conf.yaml`". This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10134) UTF-16 support for TextInputFormat
David Dreyfus created FLINK-10134:
-
Summary: UTF-16 support for TextInputFormat
Key: FLINK-10134
URL: https://issues.apache.org/jira/browse/FLINK-10134
Project: Flink
Issue Type: Bug
Components: Core
Affects Versions: 1.4.2
Reporter: David Dreyfus
It does not appear that Flink supports a charset encoding of "UTF-16". It
particular, it doesn't appear that Flink consumes the Byte Order Mark (BOM) to
establish whether a UTF-16 file is UTF-16LE or UTF-16BE.
TextInputFormat.setCharset("UTF-16") calls DelimitedInputFormat.setCharset(),
which sets TextInputFormat.charsetName and then modifies the previously set
delimiterString to construct the proper byte string encoding of the the
delimiter. This same charsetName is also used in TextInputFormat.readRecord()
to interpret the bytes read from the file.
There are two problems that this implementation would seem to have when using
UTF-16.
# delimiterString.getBytes(getCharset()) in DelimitedInputFormat.java will
return a Big Endian byte sequence including the Byte Order Mark (BOM). The
actual text file will not contain a BOM at each line ending, so the delimiter
will never be read. Moreover, if the actual byte encoding of the file is Little
Endian, the bytes will be interpreted incorrectly.
# TextInputFormat.readRecord() will not see a BOM each time it decodes a byte
sequence with the String(bytes, offset, numBytes, charset) call. Therefore, it
will assume Big Endian, which may not always be correct. [1]
[https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/io/TextInputFormat.java#L95]
While there are likely many solutions, I would think that all of them would
have to start by reading the BOM from the file when a Split is opened and then
using that BOM to modify the specified encoding to a BOM specific one when the
caller doesn't specify one, and to overwrite the caller's specification if the
BOM is in conflict with the caller's specification. That is, if the BOM
indicates Little Endian and the caller indicates UTF-16BE, Flink should rewrite
the charsetName as UTF-16LE.
I hope this makes sense and that I haven't been testing incorrectly or
misreading the code.
I've verified the problem on version 1.4.2. I believe the problem exists on all
versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10133) finished job's jobgraph never been cleaned up in zookeeper for standalone clusters (HA mode with multiple masters)
[
https://issues.apache.org/jira/browse/FLINK-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579248#comment-16579248
]
Xiangyu Zhu commented on FLINK-10133:
-
[~Wosinsan] [~elevy] I have uploaded the logs with some sensitive info
modified. If the log looks ok to you then this issue can be closed as
duplicate. Thanks!
> finished job's jobgraph never been cleaned up in zookeeper for standalone
> clusters (HA mode with multiple masters)
> --
>
> Key: FLINK-10133
> URL: https://issues.apache.org/jira/browse/FLINK-10133
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.5.0, 1.5.2, 1.6.0
>Reporter: Xiangyu Zhu
>Priority: Major
> Attachments: client.log, namenode.log, standalonesession.log,
> zookeeper.log
>
>
> Hi,
> We have 3 servers in our test environment, noted as node1-3. Setup is as
> following:
> * hadoop hdfs: node1 as namenode, node2,3 as datanode
> * zookeeper: node1-3 as a quorum (but also tried node1 alone)
> * flink: node1,2 as masters, node2,3 as slaves
> As my understanding when a job finished the corresponding job's blob data is
> expected to be deleted from hdfs path and node under zookeeper's path `/\{zk
> path root}/\{cluster-id}/jobgraphs/\{job id}` should be deleted after that.
> However we observe that whenever we submitted a job and it finished (via
> `bin/flink run WordCount.jar`), the blob data is gone whereas job id node
> under zookeeper is still there, with a uuid style lock node inside it. From
> the debug node in zookeeper we observed something like "cannot be deleted
> because non empty". Because of this, as long as a job is finished and the
> jobgraph node persists, if restart the clusters or kill one manager (to test
> HA mode), it tries to recover a finished job and couldn't find blob data
> under hdfs, and the whole cluster is down.
> If we use only node1 as master and node2,3 as slaves, the jobgraphs node can
> be deleted successfully. If the jobgraphs is clean, killing one job manager
> makes another stand-by JM raised as leader, so it is only this jobgraphs
> issue preventing HA from working.
> I'm not sure if there's something wrong with our configs because this happens
> every time for finished job (we only tested with wordcount.jar though). I'm
> aware of FLINK-10011 and FLINK-10029, but unlike FLINK-10011 this happens
> every time, rendering HA mode un-useable for us.
> Any idea what might cause this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10133) finished job's jobgraph never been cleaned up in zookeeper for standalone clusters (HA mode with multiple masters)
[
https://issues.apache.org/jira/browse/FLINK-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangyu Zhu updated FLINK-10133:
Attachment: client.log
namenode.log
zookeeper.log
standalonesession.log
> finished job's jobgraph never been cleaned up in zookeeper for standalone
> clusters (HA mode with multiple masters)
> --
>
> Key: FLINK-10133
> URL: https://issues.apache.org/jira/browse/FLINK-10133
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.5.0, 1.5.2, 1.6.0
>Reporter: Xiangyu Zhu
>Priority: Major
> Attachments: client.log, namenode.log, standalonesession.log,
> zookeeper.log
>
>
> Hi,
> We have 3 servers in our test environment, noted as node1-3. Setup is as
> following:
> * hadoop hdfs: node1 as namenode, node2,3 as datanode
> * zookeeper: node1-3 as a quorum (but also tried node1 alone)
> * flink: node1,2 as masters, node2,3 as slaves
> As my understanding when a job finished the corresponding job's blob data is
> expected to be deleted from hdfs path and node under zookeeper's path `/\{zk
> path root}/\{cluster-id}/jobgraphs/\{job id}` should be deleted after that.
> However we observe that whenever we submitted a job and it finished (via
> `bin/flink run WordCount.jar`), the blob data is gone whereas job id node
> under zookeeper is still there, with a uuid style lock node inside it. From
> the debug node in zookeeper we observed something like "cannot be deleted
> because non empty". Because of this, as long as a job is finished and the
> jobgraph node persists, if restart the clusters or kill one manager (to test
> HA mode), it tries to recover a finished job and couldn't find blob data
> under hdfs, and the whole cluster is down.
> If we use only node1 as master and node2,3 as slaves, the jobgraphs node can
> be deleted successfully. If the jobgraphs is clean, killing one job manager
> makes another stand-by JM raised as leader, so it is only this jobgraphs
> issue preventing HA from working.
> I'm not sure if there's something wrong with our configs because this happens
> every time for finished job (we only tested with wordcount.jar though). I'm
> aware of FLINK-10011 and FLINK-10029, but unlike FLINK-10011 this happens
> every time, rendering HA mode un-useable for us.
> Any idea what might cause this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10074) Allowable number of checkpoint failures
[ https://issues.apache.org/jira/browse/FLINK-10074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579269#comment-16579269 ] Thomas Weise commented on FLINK-10074: -- I think configuring the behavior as a count of allowable consecutive failures would work well. Would this replace the existing setFailOnCheckpointingErrors (will that setting become irrelevant when the user already sets the count)? [https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/environment/CheckpointConfig.html] Regarding what happens once the job was allowed to fail and recovers only to fail again: Shouldn't the counter only be reset after the next successful checkpoint vs. on restart? > Allowable number of checkpoint failures > > > Key: FLINK-10074 > URL: https://issues.apache.org/jira/browse/FLINK-10074 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Reporter: Thomas Weise >Assignee: vinoyang >Priority: Major > > For intermittent checkpoint failures it is desirable to have a mechanism to > avoid restarts. If, for example, a transient S3 error prevents checkpoint > completion, the next checkpoint may very well succeed. The user may wish to > not incur the expense of restart under such scenario and this could be > expressed with a failure threshold (number of subsequent checkpoint > failures), possibly combined with a list of exceptions to tolerate. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)
[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579276#comment-16579276 ] Fred Teunissen commented on FLINK-8500: --- I've rebased this PR with the latest master branch yesterday evening. > Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher) > --- > > Key: FLINK-8500 > URL: https://issues.apache.org/jira/browse/FLINK-8500 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.4.0 >Reporter: yanxiaobin >Priority: Major > Fix For: 1.7.0 > > Attachments: image-2018-01-30-14-58-58-167.png, > image-2018-01-31-10-48-59-633.png > > > The method deserialize of KeyedDeserializationSchema needs a parameter > 'kafka message timestamp' (from ConsumerRecord) .In some business scenarios, > this is useful！ > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10018) Want to have stop command with checkpointing
[ https://issues.apache.org/jira/browse/FLINK-10018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577875#comment-16577875 ] Keshav Lodhi commented on FLINK-10018: -- My scenarios are: * I have implemented *StoppableFunction* to stop flink job. This is what i have done https://issues.apache.org/jira/browse/FLINK-3404. * So whenever i am stopping the flink job without checkpoints, it is getting stopped but the problem i am facing with the checkpoints. * With the checkpoints job is not getting stopped, rather it is getting restarted again and again. * We have tried *stop* from flink web UI (After implementing *StoppableFunction*) as well as from the CLI, but no luck. !Flink stop command.png! > Want to have stop command with checkpointing > -- > > Key: FLINK-10018 > URL: https://issues.apache.org/jira/browse/FLINK-10018 > Project: Flink > Issue Type: New Feature >Reporter: Keshav Lodhi >Priority: Major > Labels: features > Attachments: Flink stop command.png > > > Hi Team, > I want have stop command to stop flink job which is having the > checkpoints/savepoints, But currently it is not there as i have seen in flink > official documents. Currently cancel command is present with > checkpoints/savepoints. > I have gone through this official document: > "https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html; > So i want to know, is there any reason for not giving this feature. > Thank you -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] GJL opened a new pull request #6545: [hotfix][kubernetes] Fix broken hyperlink in README.md
GJL opened a new pull request #6545: [hotfix][kubernetes] Fix broken hyperlink in README.md URL: https://github.com/apache/flink/pull/6545 Fix broken hyperlink. @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann closed pull request #6545: [hotfix][kubernetes] Fix broken hyperlink in README.md
tillrohrmann closed pull request #6545: [hotfix][kubernetes] Fix broken
hyperlink in README.md
URL: https://github.com/apache/flink/pull/6545
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-container/kubernetes/README.md
b/flink-container/kubernetes/README.md
index 9e0449616e2..5fd2286d667 100644
--- a/flink-container/kubernetes/README.md
+++ b/flink-container/kubernetes/README.md
@@ -20,7 +20,7 @@ The files contain the following variables:
- `${FLINK_JOB_PARALLELISM}`: Degree of parallelism with which to start the
Flink job and the number of required task managers
One way to substitute the variables is to use `envsubst`.
-See [here]((https://stackoverflow.com/a/23622446/4815083)) for a guide to
install it on Mac OS X.
+See [here](https://stackoverflow.com/a/23622446/4815083) for a guide to
install it on Mac OS X.
In non HA mode, you should first start the job cluster service:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-9891) Flink cluster is not shutdown in YARN mode when Flink client is stopped
[
https://issues.apache.org/jira/browse/FLINK-9891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577908#comment-16577908
]
Till Rohrmann commented on FLINK-9891:
--
Would it be an option to use `yarn application -kill` to terminate the Flink
cluster instead of adding the shut down hook [~packet]? The only thing you need
to retrieve is the corresponding application id of the running Flink cluster.
> Flink cluster is not shutdown in YARN mode when Flink client is stopped
> ---
>
> Key: FLINK-9891
> URL: https://issues.apache.org/jira/browse/FLINK-9891
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.5.1
>Reporter: Sergey Krasovskiy
>Assignee: Shuyi Chen
>Priority: Major
> Labels: pull-request-available
>
> We are not using session mode and detached mode. The command to run Flink job
> on YARN is:
> {code:java}
> /bin/flink run -m yarn-cluster -yn 1 -yqu flink -yjm 768 -ytm
> 2048 -j ./flink-quickstart-java-1.0-SNAPSHOT.jar -c org.test.WordCount
> {code}
> Flink CLI logs:
> {code:java}
> Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/flink-streaming/flink-streaming-1.5.1-1.5.1-bin-hadoop27-scala_2.11-1531485329/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/2.4.2.10-1/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2018-07-18 12:47:03,747 INFO
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service
> address: http://hmaster-1.ipbl.rgcloud.net:8188/ws/v1/timeline/
> 2018-07-18 12:47:04,222 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli -
> No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-07-18 12:47:04,222 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli -
> No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-07-18 12:47:04,248 WARN
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Neither the
> HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink
> YARN Client needs one of these to be set to properly load the Hadoop
> configuration for accessing YARN.
> 2018-07-18 12:47:04,409 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster specification:
> ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=2048,
> numberTaskManagers=1, slotsPerTaskManager=1}
> 2018-07-18 12:47:04,783 WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit
> local reads feature cannot be used because libhadoop cannot be loaded.
> 2018-07-18 12:47:04,788 WARN
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - The configuration
> directory
> ('/opt/flink-streaming/flink-streaming-1.5.1-1.5.1-bin-hadoop27-scala_2.11-1531485329/conf')
> contains both LOG4J and Logback configuration files. Please delete or rename
> one of them.
> 2018-07-18 12:47:07,846 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application
> master application_1531474158783_10814
> 2018-07-18 12:47:08,073 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application
> application_1531474158783_10814
> 2018-07-18 12:47:08,074 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for the cluster
> to be allocated
> 2018-07-18 12:47:08,076 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying cluster,
> current state ACCEPTED
> 2018-07-18 12:47:12,864 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN application has
> been deployed successfully.
> {code}
> Job Manager logs:
> {code:java}
> 2018-07-18 12:47:09,913 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>
> 2018-07-18 12:47:09,915 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting
> YarnSessionClusterEntrypoint (Version: 1.5.1, Rev:3488f8b, Date:10.07.2018 @
> 11:51:27 GMT)
> ...
> {code}
> Issues:
> # Flink job is running as a Flink session
> # Ctrl+C or 'stop' doesn't stop a job and YARN cluster
> # Cancel job via Job Maanager web ui doesn't stop Flink cluster. To kill the
> cluster we need to run: yarn application -kill
> We also tried to run a flink job with 'mode: legacy' and we have the same
> issues:
> # Add property 'mode: legacy' to
[GitHub] zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-412430748 I don't think we should do this _now_. It seems like there's little to gain at this point, since we would still have to support configuration via `flink-conf.yaml` to not break every existing setup (like this PR does). Given that there's no overlap between HistoryServer and cluster options there's also little chance of accidentally messing up the configuration for the other one. Besides that this PR is untested (yet again!). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[
https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577911#comment-16577911
]
ASF GitHub Bot commented on FLINK-10123:
tillrohrmann commented on a change in pull request #6539: [FLINK-10123] Use
ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
URL: https://github.com/apache/flink/pull/6539#discussion_r209512211
##
File path: flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package akka.actor
Review comment:
We need to use the same package as the Akka classes since `ActorSystemImpl`
is package private `private[akka]`. It is a bit ugly but it should work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
> --
>
> Key: FLINK-10123
> URL: https://issues.apache.org/jira/browse/FLINK-10123
> Project: Flink
> Issue Type: Improvement
> Components: REST
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Instead of using the {{DefaultThreadFactory}} in the
> {{RestServerEndpoint}}/{{RestClient}} we should use the
> {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}}
> per default as the uncaught exception handler. This should guard against
> uncaught exceptions by simply terminating the JVM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9964) Add a CSV table format factory
[
https://issues.apache.org/jira/browse/FLINK-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577862#comment-16577862
]
buptljy commented on FLINK-9964:
[~twalthr] Could you review my PR ?
> Add a CSV table format factory
> --
>
> Key: FLINK-9964
> URL: https://issues.apache.org/jira/browse/FLINK-9964
> Project: Flink
> Issue Type: Sub-task
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: buptljy
>Priority: Major
> Labels: pull-request-available
>
> We should add a RFC 4180 compliant CSV table format factory to read and write
> data into Kafka and other connectors. This requires a
> {{SerializationSchemaFactory}} and {{DeserializationSchemaFactory}}. How we
> want to represent all data types and nested types is still up for discussion.
> For example, we could flatten and deflatten nested types as it is done
> [here|http://support.gnip.com/articles/json2csv.html]. We can also have a
> look how tools such as the Avro to CSV tool perform the conversion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10126) There should be a Scala DataSource
[ https://issues.apache.org/jira/browse/FLINK-10126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577867#comment-16577867 ] vinoyang commented on FLINK-10126: -- Hi [~wind_ljy] I just looked at it roughly and found that the method of the same name of scala calls the warp method to wrap the DataSource into a DataSet. I haven't looked at why it does this. And is it possible to provide a way to return directly to the DataSource. As far as I understand it, the API behavior of the two languages should be as consistent as possible. If we can simply return the same signature method, then the problem will not be too complicated, but if not, it needs to be rethought. > There should be a Scala DataSource > -- > > Key: FLINK-10126 > URL: https://issues.apache.org/jira/browse/FLINK-10126 > Project: Flink > Issue Type: Improvement >Reporter: Alexis Sarda-Espinosa >Assignee: vinoyang >Priority: Minor > Labels: datasource, scala > > In Java, an ExecutionEnvironment's createInput method returns a DataSource, > whereas the Scala version returns a DataSet. There is no Scala DataSource > wrapper, and the Scala DataSet does not provide the Java DataSource methods, > such as getSplitDataProperties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10104) Job super slow to start
[
https://issues.apache.org/jira/browse/FLINK-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577887#comment-16577887
]
Gary Yao commented on FLINK-10104:
--
Hi [~fsimond],
I assume you are using Hortonworks HDP 2.5. I was not able to reproduce your
symptoms on their VM. Then I had a deeper look at the logs, in which I see many
occurrences of:
{noformat}
No open TaskExecutor connection . Ignoring close TaskExecutor
connection.
{noformat}
This is logged in {{ResourceManager#closeTaskManagerConnection}} [1] but
unfortunately we do not log the exception. I suspect that the method is called
from {{YarnResourceManager#onContainersCompleted}} [2]. This method is a
callback
invoked by YARN when a container completes. Because there is only a single
TaskManager log in your file (the one that succeeded to run the job), I assume
that the containers are stopped for reasons that are outside of Flink's
control (maybe a problem related to your YARN setup).
I would suggest the following things for further troubleshooting:
* Add improved logging to Flink, and build a custom Flink distribution [3]. For
example, log the {{ContainerStatus}} instances in {{onContainersCompleted}}.
The {{ContainerStatus}} has a diagnostics string that can be helpful.
* If the improved logging does not help, check YARN logs for hints on why the
containers exited.
* Try deploying using the Apache Hadoop distribution.
Best,
Gary
[1]
https://github.com/apache/flink/blob/release-1.5.2/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L797
[2]
https://github.com/apache/flink/blob/release-1.5.2/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java#L339
[3] https://ci.apache.org/projects/flink/flink-docs-master/start/building.html
> Job super slow to start
> ---
>
> Key: FLINK-10104
> URL: https://issues.apache.org/jira/browse/FLINK-10104
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.2
>Reporter: Florian
>Priority: Major
> Attachments: flink2.log
>
>
> Following a discussion on another topic with [~GJL] (
> [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html
>
> )|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html]
> It seems that there is a bug as my job is very slow to start.
> I am using Flink to process messages from an input topic, and to redirect
> them to two output topics, and when I start the job, I have to wait between 5
> and 10 minutes before I get anything into the output topic. With version
> 1.4.2, it was much faster.
> I run the job on Yarn, and, as asked by Gary, I attached the results of yarn
> logs -applicationId
>
> Also, as you can notice from the logs, the reported version is 0.1
> Rev:1a9b648. I have no clue why, as I downloaded the official Flink 1.5.2
> distribution
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10104) Job super slow to start
[
https://issues.apache.org/jira/browse/FLINK-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577887#comment-16577887
]
Gary Yao edited comment on FLINK-10104 at 8/13/18 6:38 AM:
---
Hi [~fsimond],
I assume you are using Hortonworks HDP 2.5. I was not able to reproduce your
symptoms on their VM. Then I had a deeper look at the logs, in which I see many
occurrences of:
{noformat}
No open TaskExecutor connection . Ignoring close TaskExecutor
connection.
{noformat}
This is logged in {{ResourceManager#closeTaskManagerConnection}} [1] but
unfortunately we do not log the exception. I suspect that the method is called
from {{YarnResourceManager#onContainersCompleted}} [2]. This method is a
callback
invoked by YARN when a container completes. Because there is only a single
TaskManager log in your file (the one that succeeded to run the job), I assume
that the containers are stopped for reasons that are outside of Flink's
control (maybe a problem related to your YARN setup).
I would suggest the following for further troubleshooting:
* Add improved logging to Flink, and build a custom Flink distribution [3].
For example, log the {{ContainerStatus}} instances in
{{onContainersCompleted}}. The {{ContainerStatus}} has a diagnostics string
that can be helpful.
* If the improved logging does not help, check YARN logs for hints on why the
containers exited.
* Try deploying using the Apache Hadoop distribution.
Best,
Gary
[1]
[https://github.com/apache/flink/blob/release-1.5.2/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L797]
[2]
[https://github.com/apache/flink/blob/release-1.5.2/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java#L339]
[3] [https://ci.apache.org/projects/flink/flink-docs-master/start/building.html]
was (Author: gjy):
Hi [~fsimond],
I assume you are using Hortonworks HDP 2.5. I was not able to reproduce your
symptoms on their VM. Then I had a deeper look at the logs, in which I see many
occurrences of:
{noformat}
No open TaskExecutor connection . Ignoring close TaskExecutor
connection.
{noformat}
This is logged in {{ResourceManager#closeTaskManagerConnection}} [1] but
unfortunately we do not log the exception. I suspect that the method is called
from {{YarnResourceManager#onContainersCompleted}} [2]. This method is a
callback
invoked by YARN when a container completes. Because there is only a single
TaskManager log in your file (the one that succeeded to run the job), I assume
that the containers are stopped for reasons that are outside of Flink's
control (maybe a problem related to your YARN setup).
I would suggest the following things for further troubleshooting:
* Add improved logging to Flink, and build a custom Flink distribution [3]. For
example, log the {{ContainerStatus}} instances in {{onContainersCompleted}}.
The {{ContainerStatus}} has a diagnostics string that can be helpful.
* If the improved logging does not help, check YARN logs for hints on why the
containers exited.
* Try deploying using the Apache Hadoop distribution.
Best,
Gary
[1]
https://github.com/apache/flink/blob/release-1.5.2/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L797
[2]
https://github.com/apache/flink/blob/release-1.5.2/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java#L339
[3] https://ci.apache.org/projects/flink/flink-docs-master/start/building.html
> Job super slow to start
> ---
>
> Key: FLINK-10104
> URL: https://issues.apache.org/jira/browse/FLINK-10104
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.2
>Reporter: Florian
>Priority: Major
> Attachments: flink2.log
>
>
> Following a discussion on another topic with [~GJL] (
> [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html
>
> )|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html]
> It seems that there is a bug as my job is very slow to start.
> I am using Flink to process messages from an input topic, and to redirect
> them to two output topics, and when I start the job, I have to wait between 5
> and 10 minutes before I get anything into the output topic. With version
> 1.4.2, it was much faster.
> I run the job on Yarn, and, as asked by Gary, I attached the results of yarn
> logs -applicationId
>
> Also, as you can notice from the logs, the reported version is 0.1
> Rev:1a9b648. I have no clue why, as I downloaded the official Flink 1.5.2
> distribution
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10128) Timeout on transfering jars on travis-ci
[
https://issues.apache.org/jira/browse/FLINK-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577895#comment-16577895
]
Dawid Wysakowicz edited comment on FLINK-10128 at 8/13/18 7:03 AM:
---
This is a travis problem. All builds with {{sudo: required}} (which is the case
for flink) are currently having network problems. Closing.
See: https://www.traviscistatus.com/incidents/hgmftqkw1sdl
was (Author: dawidwys):
This is a travis problem. All builds with {{sudo: required}} (which is the case
for flink) are currently having network problems.
See: https://www.traviscistatus.com/incidents/hgmftqkw1sdl
> Timeout on transfering jars on travis-ci
>
>
> Key: FLINK-10128
> URL: https://issues.apache.org/jira/browse/FLINK-10128
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Reporter: buptljy
>Priority: Minor
>
> 07:55:06.549 [ERROR] Failed to execute goal on project flink-parent: Could
> not resolve dependencies for project
> org.apache.flink:flink-parent:pom:1.7-SNAPSHOT: Could not transfer artifact
> com.google.code.findbugs:jsr305:jar:1.3.9 from/to central
> (http://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:80
> [repo.maven.apache.org/151.101.184.215] failed: Connection timed out
> (Connection timed out) -> [Help 1]
> [Travis Page|https://travis-ci.org/apache/flink/jobs/415047040]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-10128) Timeout on transfering jars on travis-ci
[ https://issues.apache.org/jira/browse/FLINK-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-10128. Resolution: Not A Problem > Timeout on transfering jars on travis-ci > > > Key: FLINK-10128 > URL: https://issues.apache.org/jira/browse/FLINK-10128 > Project: Flink > Issue Type: Bug > Components: Build System >Reporter: buptljy >Priority: Minor > > 07:55:06.549 [ERROR] Failed to execute goal on project flink-parent: Could > not resolve dependencies for project > org.apache.flink:flink-parent:pom:1.7-SNAPSHOT: Could not transfer artifact > com.google.code.findbugs:jsr305:jar:1.3.9 from/to central > (http://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:80 > [repo.maven.apache.org/151.101.184.215] failed: Connection timed out > (Connection timed out) -> [Help 1] > [Travis Page|https://travis-ci.org/apache/flink/jobs/415047040] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10107) SQL Client end-to-end test fails for releases
[
https://issues.apache.org/jira/browse/FLINK-10107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577905#comment-16577905
]
ASF GitHub Bot commented on FLINK-10107:
zentol edited a comment on issue #6534: [FLINK-10107] [sql-client] Relocate
Flink Kafka connectors for SQL JARs
URL: https://github.com/apache/flink/pull/6534#issuecomment-412429668
> This PR tries to solve the problem of having SQL jars that interfere with
each other.
If it's not between different kafka jars, which jars are actually affected
by this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> SQL Client end-to-end test fails for releases
> -
>
> Key: FLINK-10107
> URL: https://issues.apache.org/jira/browse/FLINK-10107
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It seems that SQL JARs for Kafka 0.10 and Kafka 0.9 have conflicts that only
> occur for releases and not SNAPSHOT builds. This might be due to their file
> name. Depending on the file name either 0.9 is loaded before 0.10 and vice
> versa.
> One of the following errors occured:
> {code}
> 2018-08-08 18:28:51,636 ERROR
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.ClientUtils -
> Failed to close coordinator
> java.lang.NoClassDefFoundError:
> org/apache/flink/kafka09/shaded/org/apache/kafka/common/requests/OffsetCommitResponse
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:473)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:357)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:439)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:319)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.ClientUtils.closeQuietly(ClientUtils.java:63)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1277)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1258)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.run(KafkaConsumerThread.java:286)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.kafka09.shaded.org.apache.kafka.common.requests.OffsetCommitResponse
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:120)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> {code}
> {code}
> java.lang.NoSuchFieldError: producer
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer010.invoke(FlinkKafkaProducer010.java:369)
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
> at
>
[jira] [Commented] (FLINK-5232) Add a Thread default uncaught exception handler on the JobManager
[
https://issues.apache.org/jira/browse/FLINK-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577899#comment-16577899
]
ASF GitHub Bot commented on FLINK-5232:
---
tillrohrmann commented on issue #6334: [FLINK-5232] Add a Thread default
uncaught exception handler on the JobManager
URL: https://github.com/apache/flink/pull/6334#issuecomment-412427923
I'll close this PR when merging #6539
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a Thread default uncaught exception handler on the JobManager
> -
>
> Key: FLINK-5232
> URL: https://issues.apache.org/jira/browse/FLINK-5232
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> When some JobManager threads die because of uncaught exceptions, we should
> bring down the JobManager. If a thread dies from an uncaught exception, there
> is a high chance that the JobManager becomes dysfunctional.
> The only sfae thing is to rely on the JobManager being restarted by YARN /
> Mesos / Kubernetes / etc.
> I suggest to add this code to the JobManager launch:
> {code}
> Thread.setDefaultUncaughtExceptionHandler(new UncaughtExceptionHandler() {
> @Override
> public void uncaughtException(Thread t, Throwable e) {
> try {
> LOG.error("Thread {} died due to an uncaught exception. Killing
> process.", t.getName());
> } finally {
> Runtime.getRuntime().halt(-1);
> }
> }
> });
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10050) Support 'allowedLateness' in CoGroupedStreams
[ https://issues.apache.org/jira/browse/FLINK-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eugen yushin updated FLINK-10050: - Affects Version/s: 1.6.0 > Support 'allowedLateness' in CoGroupedStreams > - > > Key: FLINK-10050 > URL: https://issues.apache.org/jira/browse/FLINK-10050 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.5.1, 1.6.0 >Reporter: eugen yushin >Priority: Major > Labels: ready-to-commit, windows > > WindowedStream has a support of 'allowedLateness' feature, while > CoGroupedStreams are not. At the mean time, WindowedStream is an inner part > of CoGroupedStreams and all main functionality (like evictor/trigger/...) is > simply delegated to WindowedStream. > There's no chance to operate with late arriving data from previous steps in > cogroups (and joins). Consider the following flow: > a. read data from source1 -> aggregate data with allowed lateness > b. read data from source2 -> aggregate data with allowed lateness > c. cogroup/join streams a and b, and compare aggregated values > Step c doesn't accept any late data from steps a/b due to lack of > `allowedLateness` API call in CoGroupedStreams.java. > Scope: add method `WithWindow.allowedLateness` to Java API > (flink-streaming-java) and extend scala API (flink-streaming-scala). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10107) SQL Client end-to-end test fails for releases
[
https://issues.apache.org/jira/browse/FLINK-10107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577904#comment-16577904
]
ASF GitHub Bot commented on FLINK-10107:
zentol commented on issue #6534: [FLINK-10107] [sql-client] Relocate Flink
Kafka connectors for SQL JARs
URL: https://github.com/apache/flink/pull/6534#issuecomment-412429668
> This PR tries to solve the problem of having SQL jars that interfere with
each other.
If it's not between different kafka jars, which jars are actually affected
by this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> SQL Client end-to-end test fails for releases
> -
>
> Key: FLINK-10107
> URL: https://issues.apache.org/jira/browse/FLINK-10107
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It seems that SQL JARs for Kafka 0.10 and Kafka 0.9 have conflicts that only
> occur for releases and not SNAPSHOT builds. This might be due to their file
> name. Depending on the file name either 0.9 is loaded before 0.10 and vice
> versa.
> One of the following errors occured:
> {code}
> 2018-08-08 18:28:51,636 ERROR
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.ClientUtils -
> Failed to close coordinator
> java.lang.NoClassDefFoundError:
> org/apache/flink/kafka09/shaded/org/apache/kafka/common/requests/OffsetCommitResponse
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:473)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:357)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:439)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:319)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.ClientUtils.closeQuietly(ClientUtils.java:63)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1277)
> at
> org.apache.flink.kafka09.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1258)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.run(KafkaConsumerThread.java:286)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.kafka09.shaded.org.apache.kafka.common.requests.OffsetCommitResponse
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:120)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> {code}
> {code}
> java.lang.NoSuchFieldError: producer
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer010.invoke(FlinkKafkaProducer010.java:369)
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
> at
>
[GitHub] zentol edited a comment on issue #6534: [FLINK-10107] [sql-client] Relocate Flink Kafka connectors for SQL JARs
zentol edited a comment on issue #6534: [FLINK-10107] [sql-client] Relocate Flink Kafka connectors for SQL JARs URL: https://github.com/apache/flink/pull/6534#issuecomment-412429668 > This PR tries to solve the problem of having SQL jars that interfere with each other. If it's not between different kafka jars, which jars are actually affected by this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zentol commented on issue #6534: [FLINK-10107] [sql-client] Relocate Flink Kafka connectors for SQL JARs
zentol commented on issue #6534: [FLINK-10107] [sql-client] Relocate Flink Kafka connectors for SQL JARs URL: https://github.com/apache/flink/pull/6534#issuecomment-412429668 > This PR tries to solve the problem of having SQL jars that interfere with each other. If it's not between different kafka jars, which jars are actually affected by this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10018) Want to have stop command with checkpointing
[ https://issues.apache.org/jira/browse/FLINK-10018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keshav Lodhi updated FLINK-10018: - Attachment: Flink stop command.png > Want to have stop command with checkpointing > -- > > Key: FLINK-10018 > URL: https://issues.apache.org/jira/browse/FLINK-10018 > Project: Flink > Issue Type: New Feature >Reporter: Keshav Lodhi >Priority: Major > Labels: features > Attachments: Flink stop command.png > > > Hi Team, > I want have stop command to stop flink job which is having the > checkpoints/savepoints, But currently it is not there as i have seen in flink > official documents. Currently cancel command is present with > checkpoints/savepoints. > I have gone through this official document: > "https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html; > So i want to know, is there any reason for not giving this feature. > Thank you -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10126) There should be a Scala DataSource
[ https://issues.apache.org/jira/browse/FLINK-10126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577877#comment-16577877 ] buptljy commented on FLINK-10126: - [~yanghua] Ok, it makes sense to me. > There should be a Scala DataSource > -- > > Key: FLINK-10126 > URL: https://issues.apache.org/jira/browse/FLINK-10126 > Project: Flink > Issue Type: Improvement >Reporter: Alexis Sarda-Espinosa >Assignee: vinoyang >Priority: Minor > Labels: datasource, scala > > In Java, an ExecutionEnvironment's createInput method returns a DataSource, > whereas the Scala version returns a DataSet. There is no Scala DataSource > wrapper, and the Scala DataSet does not provide the Java DataSource methods, > such as getSplitDataProperties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10129) Flink job IDs are not getting deleted automatically from zookeeper metadata after canceling flink job in flink HA cluster
Keshav Lodhi created FLINK-10129: Summary: Flink job IDs are not getting deleted automatically from zookeeper metadata after canceling flink job in flink HA cluster Key: FLINK-10129 URL: https://issues.apache.org/jira/browse/FLINK-10129 Project: Flink Issue Type: Bug Reporter: Keshav Lodhi Hi Team, Here is, what i am looking for: * We have flink HA dockerized cluster with (3 zookeepers, 2 job-managers, 3 task-managers) * So whenever we are cancelling the flink job, it is getting cancelled but it is not deleting the cancelled job ID from the zookeeper metadata (Inside flink/jobgraph folder in zookeeper) automatically. * So whenever any one of the job-manager goes down/restarted , it doesn't come up and throws exception like "Could not find this job id xx". * The current work around is to remove the canceled job ID from the zookeeper metadata manually. (But this is not the recommended solution). Please advise. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10128) Timeout on transfering jars on travis-ci
[
https://issues.apache.org/jira/browse/FLINK-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577895#comment-16577895
]
Dawid Wysakowicz commented on FLINK-10128:
--
This is a travis problem. All builds with {{sudo: required}} (which is the case
for flink) are currently having network problems.
See: https://www.traviscistatus.com/incidents/hgmftqkw1sdl
> Timeout on transfering jars on travis-ci
>
>
> Key: FLINK-10128
> URL: https://issues.apache.org/jira/browse/FLINK-10128
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Reporter: buptljy
>Priority: Minor
>
> 07:55:06.549 [ERROR] Failed to execute goal on project flink-parent: Could
> not resolve dependencies for project
> org.apache.flink:flink-parent:pom:1.7-SNAPSHOT: Could not transfer artifact
> com.google.code.findbugs:jsr305:jar:1.3.9 from/to central
> (http://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:80
> [repo.maven.apache.org/151.101.184.215] failed: Connection timed out
> (Connection timed out) -> [Help 1]
> [Travis Page|https://travis-ci.org/apache/flink/jobs/415047040]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on issue #6334: [FLINK-5232] Add a Thread default uncaught exception handler on the JobManager
tillrohrmann commented on issue #6334: [FLINK-5232] Add a Thread default uncaught exception handler on the JobManager URL: https://github.com/apache/flink/pull/6334#issuecomment-412427923 I'll close this PR when merging #6539 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9919) Remove unit from fullRestarts metric docs
[ https://issues.apache.org/jira/browse/FLINK-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577912#comment-16577912 ] ASF GitHub Bot commented on FLINK-9919: --- zentol closed pull request #6543: [FLINK-9919] Remove unit from fullRestarts metric docs URL: https://github.com/apache/flink/pull/6543 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/monitoring/metrics.md b/docs/monitoring/metrics.md index 554e1c5b1f2..05427379ada 100644 --- a/docs/monitoring/metrics.md +++ b/docs/monitoring/metrics.md @@ -1133,7 +1133,7 @@ Thus, in order to infer the metric identifier: fullRestarts - The total number of full restarts since this job was submitted (in milliseconds). + The total number of full restarts since this job was submitted. Gauge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove unit from fullRestarts metric docs > - > > Key: FLINK-9919 > URL: https://issues.apache.org/jira/browse/FLINK-9919 > Project: Flink > Issue Type: Improvement > Components: Documentation, Metrics >Affects Versions: 1.4.0, 1.5.0, 1.6.0 >Reporter: Chesnay Schepler >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The > [fullRestarts](https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#availability) > metric documentation says that the unit for the metric is milliseconds, yet > it is a simple count. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9919) Remove unit from fullRestarts metric docs
[ https://issues.apache.org/jira/browse/FLINK-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-9919. --- Resolution: Fixed Fix Version/s: 1.7.0 master: 5e6ae8d2a86d657ca9f6ea290490ed0bb00dd8eb > Remove unit from fullRestarts metric docs > - > > Key: FLINK-9919 > URL: https://issues.apache.org/jira/browse/FLINK-9919 > Project: Flink > Issue Type: Improvement > Components: Documentation, Metrics >Affects Versions: 1.4.0, 1.5.0, 1.6.0 >Reporter: Chesnay Schepler >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The > [fullRestarts](https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#availability) > metric documentation says that the unit for the metric is milliseconds, yet > it is a simple count. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tillrohrmann commented on a change in pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
tillrohrmann commented on a change in pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#discussion_r209512211 ## File path: flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package akka.actor Review comment: We need to use the same package as the Akka classes since `ActorSystemImpl` is package private `private[akka]`. It is a bit ugly but it should work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577910#comment-16577910
]
ASF GitHub Bot commented on FLINK-6437:
---
zentol commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-412430748
I don't think we should do this _now_. It seems like there's little to gain
at this point, since we would still have to support configuration via
`flink-conf.yaml` to not break every existing setup (like this PR does).
Given that there's no overlap between HistoryServer and cluster options
there's also little chance of accidentally messing up the configuration for the
other one.
Besides that this PR is untested (yet again!).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10104) Job super slow to start
[ https://issues.apache.org/jira/browse/FLINK-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577989#comment-16577989 ] Florian edited comment on FLINK-10104 at 8/13/18 10:22 AM: --- Hi [~gjy], Thanks a lot for your time and your investigation. I think you are right, we are running Hortonworks HDP 2.5 Unfortunately, I don't have more details on the problem, but it seems that it is working fine with Flink 1.6.0. Just noticed it has been release so I wanted to give it a try, and it works just fine. However, I still have this strange issue saying that I am using version 0.1, now, with version 1.6.0 : Starting YarnJobClusterEntrypoint (Version: 0.1, Rev:ff472b4, Date:07.08.2018 @ 15:31:13 CEST) I also noticed that I already had this problem with 1.4.2: Starting YARN ApplicationMaster / ResourceManager / JobManager (Version: 0.1, Rev:04e4c85, Date:28.02.2018 @ 20:53:14 CST) was (Author: fsimond): Hi [~gjy], Thanky a lot for your time and your investigation. I think you are right, we are running Hortonworks HDP 2.5 Unfortunately, I don't have more details on the problem, but it seems that it is working fine with Flink 1.6.0. Just noticed it has been release so I wanted to give it a try, and it works just fine. However, I still have this strange issue saying that I am using version 0.1, now, with version 1.6.0 : Starting YarnJobClusterEntrypoint (Version: 0.1, Rev:ff472b4, Date:07.08.2018 @ 15:31:13 CEST) I also noticed that I already had this problem with 1.4.2: Starting YARN ApplicationMaster / ResourceManager / JobManager (Version: 0.1, Rev:04e4c85, Date:28.02.2018 @ 20:53:14 CST) > Job super slow to start > --- > > Key: FLINK-10104 > URL: https://issues.apache.org/jira/browse/FLINK-10104 > Project: Flink > Issue Type: Bug >Affects Versions: 1.5.2 >Reporter: Florian >Priority: Major > Attachments: flink2.log > > > Following a discussion on another topic with [~GJL] ( > [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html > > )|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-build-the-program-from-JAR-file-td22102.html] > It seems that there is a bug as my job is very slow to start. > I am using Flink to process messages from an input topic, and to redirect > them to two output topics, and when I start the job, I have to wait between 5 > and 10 minutes before I get anything into the output topic. With version > 1.4.2, it was much faster. > I run the job on Yarn, and, as asked by Gary, I attached the results of yarn > logs -applicationId > > Also, as you can notice from the logs, the reported version is 0.1 > Rev:1a9b648. I have no clue why, as I downloaded the official Flink 1.5.2 > distribution > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
yanghua commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412493074 @StefanRRichter OK, I respect this decision, but since Flink already provides the Python API, I think a better option is to match Flink's Python API with a specific version of the DataStream API, and then we give a declaration on the documentation: Python api only supports which version, and will not be updated or upgraded. Instead of supporting a semi-finished product that cannot be explained. I want to enhance it, mainly because multiple users ask questions about the Python API on the mailing list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412493850 @yanghua I somewhat disagree. It is not a regression that we would need to fix, and why give users incentives to start/continue using an API that will soon be replaced? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-10006) Improve logging in BarrierBuffer
[
https://issues.apache.org/jira/browse/FLINK-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber resolved FLINK-10006.
-
Resolution: Fixed
Fix Version/s: 1.7.0
1.6.1
1.5.3
Fixed via:
- master: ed61c3a46b074abdb1c33606aa464a70e46a4dc8
- release-1.6: 0fa308e30e5d6eec70e687a24c65d3f2d7a5e512
- release-1.5: 8193d5dc68289760ad68cf0b6b237fd86b0fd906
> Improve logging in BarrierBuffer
>
>
> Key: FLINK-10006
> URL: https://issues.apache.org/jira/browse/FLINK-10006
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.5.2, 1.6.0, 1.7.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.3, 1.6.1, 1.7.0
>
>
> Almost all log messages of {{BarrierBuffer}} do not contain the task name and
> are therefore of little use if either multiple slots are executed on a single
> TM or multiple checkpoints run in parallel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10131) Improve logging around ResultSubpartition
Nico Kruber created FLINK-10131:
---
Summary: Improve logging around ResultSubpartition
Key: FLINK-10131
URL: https://issues.apache.org/jira/browse/FLINK-10131
Project: Flink
Issue Type: Improvement
Components: Logging, Network
Affects Versions: 1.6.0, 1.5.2, 1.7.0
Reporter: Nico Kruber
Assignee: Nico Kruber
All log messages in {{PipelinedSubpartition}} and {{SpillableSubpartition}}
should contain the task name to improve debugging based on log files.
Furthermore, subpartition log messages should at least contain their
subpartition index to make them distinguishable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol commented on issue #6536: Fix Javadoc links in documentation
zentol commented on issue #6536: Fix Javadoc links in documentation URL: https://github.com/apache/flink/pull/6536#issuecomment-412477213 ah, you mean the snapshot javadocs aren't working for the snapshot branches as well. Still, can't we fix this trivially in `update_branch_version.sh` by extending it? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10066) Keep only archived version of previous executions
[ https://issues.apache.org/jira/browse/FLINK-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578138#comment-16578138 ] ASF GitHub Bot commented on FLINK-10066: StefanRRichter commented on issue #6500: [FLINK-10066] Keep only archived version of previous executions URL: https://github.com/apache/flink/pull/6500#issuecomment-412489962 CC @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Keep only archived version of previous executions > - > > Key: FLINK-10066 > URL: https://issues.apache.org/jira/browse/FLINK-10066 > Project: Flink > Issue Type: Improvement > Components: JobManager >Affects Versions: 1.4.3, 1.5.2, 1.6.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > Labels: pull-request-available > > Currently, the execution vertex stores a limited amount of previous > executions in a bounded list. This happens primarily for archiving purposes > and to remember previous locations and allocation ids. We remember the whole > execution to eventually convert it into an archived execution. > This seems unnecessary and dangerous as we have observed that this strategy > is prone to memory leaks in the job manager. With a very high vertex count or > parallelism, remembering complete executions can become very memory > intensive. Instead I suggest to eagerly transform the executions into the > archived version before adding them to the list, i.e. only the archived > version is ever still referenced after the execution becomes obsolete. This > gives better control over which information about the execution should really > be kept in memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412491024 @yanghua I had a few discussions with PMC members and the plans for Flink's Python support are centered around Apache Beam. While Flink's own Python API is not yet deprecated in a strict sense, I think it does not make much sense anymore to enhance it. For this reason, my suggestion would be to close the PRs and JIRAs about enhancing the Flink Python API and not put any more efforts into it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10012) Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
[ https://issues.apache.org/jira/browse/FLINK-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578151#comment-16578151 ] ASF GitHub Bot commented on FLINK-10012: StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412491024 @yanghua I had a few discussions with PMC members and the plans for Flink's Python support are centered around Apache Beam. While Flink's own Python API is not yet deprecated in a strict sense, I think it does not make much sense anymore to enhance it. For this reason, my suggestion would be to close the PRs and JIRAs about enhancing the Flink Python API and not put any more efforts into it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API > -- > > Key: FLINK-10012 > URL: https://issues.apache.org/jira/browse/FLINK-10012 > Project: Flink > Issue Type: Sub-task > Components: Python API >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10012) Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
[ https://issues.apache.org/jira/browse/FLINK-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578157#comment-16578157 ] ASF GitHub Bot commented on FLINK-10012: yanghua commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412493074 @StefanRRichter OK, I respect this decision, but since Flink already provides the Python API, I think a better option is to match Flink's Python API with a specific version of the DataStream API, and then we give a declaration on the documentation: Python api only supports which version, and will not be updated or upgraded. Instead of supporting a semi-finished product that cannot be explained. I want to enhance it, mainly because multiple users ask questions about the Python API on the mailing list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API > -- > > Key: FLINK-10012 > URL: https://issues.apache.org/jira/browse/FLINK-10012 > Project: Flink > Issue Type: Sub-task > Components: Python API >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10012) Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API
[ https://issues.apache.org/jira/browse/FLINK-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578165#comment-16578165 ] ASF GitHub Bot commented on FLINK-10012: StefanRRichter commented on issue #6475: [FLINK-10012] Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API URL: https://github.com/apache/flink/pull/6475#issuecomment-412493850 @yanghua I somewhat disagree. It is not a regression that we would need to fix, and why give users incentives to start/continue using an API that will soon be replaced? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add setStreamTimeCharacteristic/getStreamTimeCharacteristic for Python API > -- > > Key: FLINK-10012 > URL: https://issues.apache.org/jira/browse/FLINK-10012 > Project: Flink > Issue Type: Sub-task > Components: Python API >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] azagrebin commented on a change in pull request #5304: [FLINK-8290]Modify clientId to groupId in flink-connector-kafka-0.8
azagrebin commented on a change in pull request #5304: [FLINK-8290]Modify
clientId to groupId in flink-connector-kafka-0.8
URL: https://github.com/apache/flink/pull/5304#discussion_r209553402
##
File path:
flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/SimpleConsumerThread.java
##
@@ -123,6 +124,7 @@ public SimpleConsumerThread(
this.fetchSize = getInt(config, "fetch.message.max.bytes",
1048576);
this.bufferSize = getInt(config, "socket.receive.buffer.bytes",
65536);
this.reconnectLimit = getInt(config,
"flink.simple-consumer-reconnectLimit", 3);
+ this.clientId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
Review comment:
I think we can just replace this line in the current PR:
```
this.clientId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
```
with these 2 lines:
```
String groupId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
this.clientId = config.getProperty("client.id", groupId);
```
`Group id` will be default value for `client id` then, as in Kafka docs, but
user can change both of them independently if user ever needs it. It should
also help in logs with multiple topics.
`"flink-kafka-consumer-legacy-" + broker.id()` will be just default value
for `group.id`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-8290) Modify clientId to groupId in flink-connector-kafka-0.8
[
https://issues.apache.org/jira/browse/FLINK-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578043#comment-16578043
]
ASF GitHub Bot commented on FLINK-8290:
---
azagrebin commented on a change in pull request #5304: [FLINK-8290]Modify
clientId to groupId in flink-connector-kafka-0.8
URL: https://github.com/apache/flink/pull/5304#discussion_r209553402
##
File path:
flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/SimpleConsumerThread.java
##
@@ -123,6 +124,7 @@ public SimpleConsumerThread(
this.fetchSize = getInt(config, "fetch.message.max.bytes",
1048576);
this.bufferSize = getInt(config, "socket.receive.buffer.bytes",
65536);
this.reconnectLimit = getInt(config,
"flink.simple-consumer-reconnectLimit", 3);
+ this.clientId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
Review comment:
I think we can just replace this line in the current PR:
```
this.clientId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
```
with these 2 lines:
```
String groupId = config.getProperty("group.id",
"flink-kafka-consumer-legacy-" + broker.id());
this.clientId = config.getProperty("client.id", groupId);
```
`Group id` will be default value for `client id` then, as in Kafka docs, but
user can change both of them independently if user ever needs it. It should
also help in logs with multiple topics.
`"flink-kafka-consumer-legacy-" + broker.id()` will be just default value
for `group.id`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Modify clientId to groupId in flink-connector-kafka-0.8
> ---
>
> Key: FLINK-8290
> URL: https://issues.apache.org/jira/browse/FLINK-8290
> Project: Flink
> Issue Type: Improvement
>Reporter: xymaqingxiang
>Assignee: xymaqingxiang
>Priority: Major
> Labels: pull-request-available
>
> Now the Clientid that consumes the all topics are
> constant("flink-kafka-consumer-legacy-" + broker.id()), and it is not easy
> for us to look at kafka's log, so I recommend that it be modified to groupid.
> We can modify the SimpleConsumerThread.java file, as shown below:
> {code:java}
> private final String clientId;
> ...
> this.clientId = config.getProperty("group.id", "flink-kafka-consumer-legacy-"
> + broker.id());
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9913) Improve output serialization only once in RecordWriter
[
https://issues.apache.org/jira/browse/FLINK-9913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578048#comment-16578048
]
ASF GitHub Bot commented on FLINK-9913:
---
zhijiangW commented on issue #6417: [FLINK-9913][runtime] Improve output
serialization only once in RecordWriter
URL: https://github.com/apache/flink/pull/6417#issuecomment-412469613
@pnowojski , thanks for your reviews!
I basically agree with your idea of separating current `RecordSerializer`
further, but consider specific implementation, there are still some issues to
be confirmed. After we reach the agreement, I will continue with the test
issues. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve output serialization only once in RecordWriter
> --
>
> Key: FLINK-9913
> URL: https://issues.apache.org/jira/browse/FLINK-9913
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.6.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Currently the {{RecordWriter}} emits output into multi channels via
> {{ChannelSelector}} or broadcasts output to all channels directly. Each
> channel has a separate {{RecordSerializer}} for serializing outputs, that
> means the output will be serialized as many times as the number of selected
> channels.
> As we know, data serialization is a high cost operation, so we can get good
> benefits by improving the serialization only once.
> I would suggest the following changes for realizing it.
> # Only one {{RecordSerializer}} is created in {{RecordWriter}} for all the
> channels.
> # The output is serialized into the intermediate data buffer only once for
> different channels.
> # The intermediate serialization results are copied into different
> {{BufferBuilder}}s for different channels.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhijiangW commented on issue #6417: [FLINK-9913][runtime] Improve output serialization only once in RecordWriter
zhijiangW commented on issue #6417: [FLINK-9913][runtime] Improve output serialization only once in RecordWriter URL: https://github.com/apache/flink/pull/6417#issuecomment-412469613 @pnowojski , thanks for your reviews! I basically agree with your idea of separating current `RecordSerializer` further, but consider specific implementation, there are still some issues to be confirmed. After we reach the agreement, I will continue with the test issues. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10006) Improve logging in BarrierBuffer
[
https://issues.apache.org/jira/browse/FLINK-10006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-10006:
Component/s: Logging
> Improve logging in BarrierBuffer
>
>
> Key: FLINK-10006
> URL: https://issues.apache.org/jira/browse/FLINK-10006
> Project: Flink
> Issue Type: Improvement
> Components: Logging, Network
>Affects Versions: 1.5.2, 1.6.0, 1.7.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.3, 1.6.1, 1.7.0
>
>
> Almost all log messages of {{BarrierBuffer}} do not contain the task name and
> are therefore of little use if either multiple slots are executed on a single
> TM or multiple checkpoints run in parallel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol commented on issue #6536: Fix Javadoc links in documentation
zentol commented on issue #6536: Fix Javadoc links in documentation URL: https://github.com/apache/flink/pull/6536#issuecomment-412476867 `update_release_branch` doesn't exist. `update_branch_version.sh` is applied to master, but the commit is also contained in the `release-X.Y` branches, and as seen in 8d180d5fa69a68a7023b052f83e51ca6a3d35256 it does correctly modify the versions. What exactly isn't working? If it doesn't work, why not fix it instead of introducing manual steps yet again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10129) Flink job IDs are not getting deleted automatically from zookeeper metadata after canceling flink job in flink HA cluster
[ https://issues.apache.org/jira/browse/FLINK-10129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578134#comment-16578134 ] Till Rohrmann commented on FLINK-10129: --- Hi [~keshav.lo...@widas.in], thanks for reporting this issue. I think this issue could be the same as FLINK-10011. If this should be the case, then we can close this issue as a duplicate. > Flink job IDs are not getting deleted automatically from zookeeper metadata > after canceling flink job in flink HA cluster > -- > > Key: FLINK-10129 > URL: https://issues.apache.org/jira/browse/FLINK-10129 > Project: Flink > Issue Type: Bug >Reporter: Keshav Lodhi >Priority: Blocker > > Hi Team, > Here is, what i am looking for: > * We have flink HA dockerized cluster with (3 zookeepers, 2 job-managers, 3 > task-managers) > * So whenever we are cancelling the flink job, it is getting cancelled but > it is not deleting the cancelled job ID from the zookeeper metadata (Inside > flink/jobgraph folder in zookeeper) automatically. > * So whenever any one of the job-manager goes down/restarted , it doesn't > come up and throws exception like "Could not find this job id xx". > * The current work around is to remove the canceled job ID from the > zookeeper metadata manually. (But this is not the recommended solution). > > Please advise. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] StefanRRichter commented on issue #6500: [FLINK-10066] Keep only archived version of previous executions
StefanRRichter commented on issue #6500: [FLINK-10066] Keep only archived version of previous executions URL: https://github.com/apache/flink/pull/6500#issuecomment-412489962 CC @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-2995) Set default number of retries to larger than 0
[ https://issues.apache.org/jira/browse/FLINK-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned FLINK-2995: --- Assignee: vinoyang > Set default number of retries to larger than 0 > -- > > Key: FLINK-2995 > URL: https://issues.apache.org/jira/browse/FLINK-2995 > Project: Flink > Issue Type: Wish > Components: Core >Affects Versions: 0.10.0 >Reporter: Stephan Ewen >Assignee: vinoyang >Priority: Minor > > Right now, the default number of retries is 0, meaning fault tolerance must > be activated explicitly. > In both streaming and batch, we can provide a smoother experience when fault > tolerance is activated by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twalthr commented on a change in pull request #6337: [FLINK-9853] [table] Add HEX support
twalthr commented on a change in pull request #6337: [FLINK-9853] [table] Add
HEX support
URL: https://github.com/apache/flink/pull/6337#discussion_r209582385
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
##
@@ -392,6 +392,93 @@ class ScalarFunctionsTest extends ScalarTypesTestBase {
"äää1234512345")
}
+ @Test
+ def testHex(): Unit = {
+testAllApis(
+ 100.hex(),
+ "100.hex()",
+ "HEX(100)",
+ "64")
+
+testAllApis(
+ 'f2.hex(),
+ "f2.hex()",
+ "HEX(f2)",
+ "2a")
Review comment:
MySQL returns upper-case letters. Should we do this as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] twalthr commented on a change in pull request #6337: [FLINK-9853] [table] Add HEX support
twalthr commented on a change in pull request #6337: [FLINK-9853] [table] Add
HEX support
URL: https://github.com/apache/flink/pull/6337#discussion_r209582133
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/mathExpressions.scala
##
@@ -422,3 +422,21 @@ case class Bin(child: Expression) extends UnaryExpression
{
relBuilder.call(ScalarSqlFunctions.BIN, child.toRexNode)
}
}
+
+case class Hex(child: Expression) extends UnaryExpression {
+ override private[flink] def resultType: TypeInformation[_] =
BasicTypeInfo.STRING_TYPE_INFO
+
+ override private[flink] def validateInput(): ValidationResult =
child.resultType match {
+case _: IntegerTypeInfo[_] =>
Review comment:
Rely on `TypeCheckUtils.isIntegerFamily` and `TypeCheckUtils#isString`
instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-9853) add hex support in table api and sql
[
https://issues.apache.org/jira/browse/FLINK-9853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578170#comment-16578170
]
ASF GitHub Bot commented on FLINK-9853:
---
twalthr commented on a change in pull request #6337: [FLINK-9853] [table] Add
HEX support
URL: https://github.com/apache/flink/pull/6337#discussion_r209582385
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
##
@@ -392,6 +392,93 @@ class ScalarFunctionsTest extends ScalarTypesTestBase {
"äää1234512345")
}
+ @Test
+ def testHex(): Unit = {
+testAllApis(
+ 100.hex(),
+ "100.hex()",
+ "HEX(100)",
+ "64")
+
+testAllApis(
+ 'f2.hex(),
+ "f2.hex()",
+ "HEX(f2)",
+ "2a")
Review comment:
MySQL returns upper-case letters. Should we do this as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> add hex support in table api and sql
>
>
> Key: FLINK-9853
> URL: https://issues.apache.org/jira/browse/FLINK-9853
> Project: Flink
> Issue Type: Improvement
> Components: Table API SQL
>Reporter: xueyu
>Priority: Major
> Labels: pull-request-available
>
> like in mysql, HEX could take int or string arguments, For a integer argument
> N, it returns a hexadecimal string representation of the value of N. For a
> string argument str, it returns a hexadecimal string representation of str
> where each byte of each character in str is converted to two hexadecimal
> digits.
> Syntax:
> HEX(100) = 64
> HEX('This is a test String.') = '546869732069732061207465737420537472696e672e'
> See more: [link
> MySQL|https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_hex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
