[GitHub] [flink] flinkbot edited a comment on pull request #17622: [FLINK-24407][doc-zh]Pulsar connector chinese document link to Pulsar document location incorrectly.
flinkbot edited a comment on pull request #17622: URL: https://github.com/apache/flink/pull/17622#issuecomment-956197495 ## CI report: * 0815a002a1ee08873b6270214a14caec145ac83a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28019) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25267:
-
Description:
My Stateful Functions job is running on Kubernetes (minikube on my local env)
and has these settings:
* Using StateFun v3.1.0
* Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
* Checkpointing mode is EXACTLY_ONCE
* State backend is rocksdb and incremental checkpointing is enabled
When I kill the jobmanager (master) pod, minikube starts another pod and this
new pod fails when it tries to load last checkpoint:
{code:java}
...
2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job myStatefunApp
().
2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using restart back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for myStatefunApp ().
2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
[] - Initialized DefaultCompletedCheckpointStore in
'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
with /checkpoints/.
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Running initialization on master for job myStatefunApp
().
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Successfully ran initialization on master in 0 ms.
2021-12-11 14:25:26,617 INFO
org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built
1 pipelined regions in 1 ms
2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using job/cluster config to configure application-defined state
backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
predefined options: DEFAULT.
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
application-defined options factory:
DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using application-defined state backend:
EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Checkpoint storage is set to 'filesystem': (checkpoints
"hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
2021-12-11 14:25:26,712 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
Recovering checkpoints from
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,724 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found
1 checkpoints in
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to fetch 1 checkpoints from storage.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to retrieve checkpoint 2.
2021-12-11 14:25:26,931 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
from Checkpoint 2 @ 1639232587220 for
located at
hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp//chk-2.
2021-12-11 14:25:27,012 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job
failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:436)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28026) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25175) Introduce TableDescriptor.forManaged
[
https://issues.apache.org/jira/browse/FLINK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee reassigned FLINK-25175:
Assignee: Jingsong Lee
> Introduce TableDescriptor.forManaged
>
>
> Key: FLINK-25175
> URL: https://issues.apache.org/jira/browse/FLINK-25175
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.15.0
>
>
> Introduce table api for managed table
> {code:java}
> @PublicEvolving
> public class TableDescriptor {
>
> /** Creates a new {@link Builder} for a managed dynamic table. */
> public static Builder forManaged() {
> return new Builder();
> }
>
> ...
> } {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-25273) Some doubts about the FLINK-22848
[ https://issues.apache.org/jira/browse/FLINK-25273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-25273. -- Resolution: Not A Bug Hi and thanks for opening this ticket. It's better to bring this up for discussion on the Dev mailing list. See https://flink.apache.org/community.html on how to do this. You could also comment on the Jira ticket you're referring to, but I think the Dev mailing list is better suited. > Some doubts about the FLINK-22848 > - > > Key: FLINK-25273 > URL: https://issues.apache.org/jira/browse/FLINK-25273 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Jianhui Dong >Priority: Major > > I have been in contact with Flink and Calcite for a while, and I have some > questions about this issue: https://issues.apache.org/jira/browse/FLINK-22848. > First of all, the discussion about this issue mentioned that since calcite > did not support the syntax analysis of set a=b (without quotes), regular > expressions were used. However, I checked the commit history some days ago, > and I found that calcite should already support SET syntax parsing (see > SqlSetOption) in v1.14 or even earlier. but its problem is that it would > recognize the `true/false/unknown/null` tokens as keywords, causing the > parsing to be worse than expected, but this problem can be solved by > restricting the syntax, like use '' in the issue FLINK-22848. > Then I investigated the earliest version of flink that introduced calcite, > flink should have introduced Calcite 1.16 in 1.5 at the earliest version. At > that time, calcite should already support the syntax of SET a=b (without > quotes), so I would like to find out What exactly does the "not supported" > mentioned in the issue FLINK-22848 means? Maybe you can give a more specific > case. > In addition, I also have some doubts about the results of the discussion of > the issue. I think it is indeed a more elegant solution to use the SQL parser > to analyze it, but When calcite has built-in support for SET grammar, why do > we need to extend the SET grammar to re-support it? Even this change may > cause backward-incompatible. > In my personal opinion of view, is that we can solve this problem by adding > special restrictions on the above tokens on the basis of native Calcite > analysis, such as in the user manual because values such as `unknown` and > `null` are meaningless in the production environment. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #37: [FLINK-24955] Add Estimator and Transformer for One Hot Encoder
yunfengzhou-hub commented on a change in pull request #37:
URL: https://github.com/apache/flink-ml/pull/37#discussion_r767476362
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/linalg/typeinfo/SparseVectorSerializer.java
##
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.linalg.typeinfo;
+
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.ml.linalg.SparseVector;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+/** Specialized serializer for {@link SparseVector}. */
+public final class SparseVectorSerializer extends
TypeSerializerSingleton {
+
+private static final long serialVersionUID = 1L;
+
+private static final double[] EMPTY_DOUBLE_ARRAY = new double[0];
+
+private static final int[] EMPTY_INT_ARRAY = new int[0];
+
+private static final SparseVectorSerializer INSTANCE = new
SparseVectorSerializer();
Review comment:
OK. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17622: [FLINK-24407][doc-zh]Pulsar connector chinese document link to Pulsar document location incorrectly.
flinkbot edited a comment on pull request #17622: URL: https://github.com/apache/flink/pull/17622#issuecomment-956197495 ## CI report: * ba19443440c7613e48bacf4ca692ef207936a8b2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28018) * 0815a002a1ee08873b6270214a14caec145ac83a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28019) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25273) Some doubts about the FLINK-22848
Jianhui Dong created FLINK-25273: Summary: Some doubts about the FLINK-22848 Key: FLINK-25273 URL: https://issues.apache.org/jira/browse/FLINK-25273 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Jianhui Dong I have been in contact with Flink and Calcite for a while, and I have some questions about this issue: https://issues.apache.org/jira/browse/FLINK-22848. First of all, the discussion about this issue mentioned that since calcite did not support the syntax analysis of set a=b (without quotes), regular expressions were used. However, I checked the commit history some days ago, and I found that calcite should already support SET syntax parsing (see SqlSetOption) in v1.14 or even earlier. but its problem is that it would recognize the `true/false/unknown/null` tokens as keywords, causing the parsing to be worse than expected, but this problem can be solved by restricting the syntax, like use '' in the issue FLINK-22848. Then I investigated the earliest version of flink that introduced calcite, flink should have introduced Calcite 1.16 in 1.5 at the earliest version. At that time, calcite should already support the syntax of SET a=b (without quotes), so I would like to find out What exactly does the "not supported" mentioned in the issue FLINK-22848 means? Maybe you can give a more specific case. In addition, I also have some doubts about the results of the discussion of the issue. I think it is indeed a more elegant solution to use the SQL parser to analyze it, but When calcite has built-in support for SET grammar, why do we need to extend the SET grammar to re-support it? Even this change may cause backward-incompatible. In my personal opinion of view, is that we can solve this problem by adding special restrictions on the above tokens on the basis of native Calcite analysis, such as in the user manual because values such as `unknown` and `null` are meaningless in the production environment. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24295) Too many requestPartitionState may jam the JobManager during task deployment
[
https://issues.apache.org/jira/browse/FLINK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458183#comment-17458183
]
huntercc commented on FLINK-24295:
--
I set an empirical value `3` as the default value, which seems to be working
well. But I'm also not sure it will be a best value for any scenario.
> Too many requestPartitionState may jam the JobManager during task deployment
>
>
> Key: FLINK-24295
> URL: https://issues.apache.org/jira/browse/FLINK-24295
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Major
>
> After the optimization of the phase 2 we've done in FLINK-21110, the speed of
> task deployment has accelerated. However, we find that during the task
> deployment, there may be too many {{requestPartitionState}} RPC calls from
> TaskManagers that would jam the JobManager.
> Why would there be too many {{requestPartitionState}} RPC calls? After the
> optimization, the JobManager can submit tasks to TaskManagers quickly. If
> JobManager calls {{submitTask}} faster than the speed of dealing with
> {{submitTask}} by TaskManagers, there may be a scenario that some
> TaskManagers deploy tasks faster than other TaskManagers.
> When a downstream task is deployed, it would try to request partitions from
> upstream tasks, which may be located at a remote TaskManager. If the upstream
> tasks are not deployed, it would request the partition state from JobManager.
> In the worst case, the complexity of the computation and memory would be
> O(N^2).
> In our test with a streaming job, which has two vertices with the 8,000
> parallelism and connected with all-to-all edges, in the worst case, there
> will be 32,000,000 {{requestPartitionState}} RPC calls in the JobManager.
> Each RPC call requires 1 KiB space in the heap memory of the JobManager. The
> overall space cost of {{requestPartitionState}} will be 32 GiB, which is a
> heavy burden for GC to deal with.
> In our test, the size of the heap memory of JobManager is 8 GiB. During the
> task deployment the JobManager gets more full GCs. The JobManager gets stuck
> since it is filled with full GCs and has no time to deal with the incoming
> RPC calls.
> The worst thing is that there's no log outputted for this RPC call. When a
> user find the JobManager is get slower or get stuck, he/she won't be able to
> find out why.
> Why does this case rarely happen before? Before the optimization, it takes a
> long time to calculate TaskDeploymentDescriptors and send them to
> TaskManagers. JobManager calls {{submitTask}} more slowly than the speed of
> dealing with {{submitTask}} by TaskManagers in most cases. Since the
> deployment of tasks are topologically sorted, the upstream tasks is deployed
> before the downstream tasks, and this case rarely happens.
> In my opinion, the solution of this issue needs more discussion. According to
> the discussion in the pull request
> ([https://github.com/apache/flink/pull/6680]), it's not safe to remove this
> RPC call, because we cannot always make sure the assumption that an upstream
> task failure will always fail the downstream consumers is always right.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25229) Introduce flink-table-api-bridge-base
[
https://issues.apache.org/jira/browse/FLINK-25229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-25229:
-
Summary: Introduce flink-table-api-bridge-base (was: Introduce
flink-table-api-bridge-common)
> Introduce flink-table-api-bridge-base
> -
>
> Key: FLINK-25229
> URL: https://issues.apache.org/jira/browse/FLINK-25229
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Francesco Guardiani
>Assignee: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> This package should deduplicate code from api-java-bridge and
> api-scala-bridge, notably:
> * the various operations provided by both {{ScalaDataStreamQueryOperation}}
> and {{JavaDataStreamQueryOperation}} (which are essentially the same code)
> * some code in {{StreamTableEnvironmentImpl}} and {{StreamStatementSetImpl}}
> The end goal is that planner should remove the runtime (not test) dependency
> on flink-scala-api and flink-scala-api-bridge
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25229) Introduce flink-table-api-bridge-common
[
https://issues.apache.org/jira/browse/FLINK-25229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-25229.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master: e4ae2ef81e9ecbda10c4dcc5776584b07c2f5e6b
> Introduce flink-table-api-bridge-common
> ---
>
> Key: FLINK-25229
> URL: https://issues.apache.org/jira/browse/FLINK-25229
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Francesco Guardiani
>Assignee: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> This package should deduplicate code from api-java-bridge and
> api-scala-bridge, notably:
> * the various operations provided by both {{ScalaDataStreamQueryOperation}}
> and {{JavaDataStreamQueryOperation}} (which are essentially the same code)
> * some code in {{StreamTableEnvironmentImpl}} and {{StreamStatementSetImpl}}
> The end goal is that planner should remove the runtime (not test) dependency
> on flink-scala-api and flink-scala-api-bridge
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] twalthr closed pull request #18065: [FLINK-25229][table] Introduce flink-table-api-bridge-common
twalthr closed pull request #18065: URL: https://github.com/apache/flink/pull/18065 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * 26b6b9c6159d8bc45e892462f7104f2c37c33cc8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25695) * 3b77ef5449acad53555add69956a0cb884f0adf8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28024) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25272) Flink sql syntax analysis is inconsistent with the sql grammer declared now
[ https://issues.apache.org/jira/browse/FLINK-25272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianhui Dong updated FLINK-25272: - Description: Syntax analysis in the code(https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/includes/parserImpls.ftl) is inconsistent with the grammar declared on the module(https://github.com/apache/flink/edit/master/docs/content/docs/dev/table/sql/create.md). The document states that all CREATE statement formats are `CREATE TABLE [IF NOT EXISTS]`, there is no `TEMPORAL` keywords, but in flink-sql-parser/src/main/codegen/includes/parserImpls.ftl::SqlCreateExtended, it will recorganize `TEMPORAL` token and set `isTemporary` with true if the keyword exists. Besides, I think the field `isTemporary` is outdated too, every table in Flink is a temporal table as the document wrote. In fact, when they are stored in the catalog, they are all converted to `CatalogSchemaTable` which inherits the `TemporalTable` interface. BTW, the view and the function have the same condition. IMO, I think we should remove the `TEMPORAL` token in parserImpls.ftl and also remove the `isTemporary` field which now always is false. This issue may be a little confused cause my English is not very well. If you have any questions, you can add comments under the issue or contact me at lam...@apache.org and I could also provide a pull request for this. was: Syntax analysis in the code(https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/includes/parserImpls.ftl) is inconsistent with the grammar declared on the module(https://github.com/apache/flink/edit/master/docs/content/docs/dev/table/sql/create.md). The document states that all CREATE statement formats are `CREATE TABLE [IF NOT EXISTS]`, there is no `TEMPORAL` keywords, but in flink-sql-parser/src/main/codegen/includes/parserImpls.ftl::SqlCreateExtended, it will recorganize `TEMPORAL` token and set `isTemporary` with true if the keyword exists. Besides, I think the field `isTemporary` is outdated too, every table in Flink is temporal table as the document wrote. In fact, when they are stored in the catalog, they are all converted to `CatalogSchemaTable` which inherits the `TemporalTable` interface. BTW, the view and the function has the same condition. IMO, I think we should remove the `TEMPORAL` token in parserImpls.ftl and also remove the `isTemporary` field which now always is false. This issue may be a little confused cause my English is not very well. If you have any questions, you can add comments under the issue or contact me with lam...@apache.org. > Flink sql syntax analysis is inconsistent with the sql grammer declared now > --- > > Key: FLINK-25272 > URL: https://issues.apache.org/jira/browse/FLINK-25272 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Jianhui Dong >Priority: Minor > > Syntax analysis in the > code(https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/includes/parserImpls.ftl) > is inconsistent with the grammar declared on the > module(https://github.com/apache/flink/edit/master/docs/content/docs/dev/table/sql/create.md). > The document states that all CREATE statement formats are `CREATE TABLE [IF > NOT EXISTS]`, there is no `TEMPORAL` keywords, but in > flink-sql-parser/src/main/codegen/includes/parserImpls.ftl::SqlCreateExtended, > it will recorganize `TEMPORAL` token and set `isTemporary` with true if the > keyword exists. > Besides, I think the field `isTemporary` is outdated too, every table in > Flink is a temporal table as the document wrote. In fact, when they are > stored in the catalog, they are all converted to `CatalogSchemaTable` which > inherits the `TemporalTable` interface. > BTW, the view and the function have the same condition. IMO, I think we > should remove the `TEMPORAL` token in parserImpls.ftl and also remove the > `isTemporary` field which now always is false. > This issue may be a little confused cause my English is not very well. If you > have any questions, you can add comments under the issue or contact me at > lam...@apache.org and I could also provide a pull request for this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * 26b6b9c6159d8bc45e892462f7104f2c37c33cc8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25695) * 3b77ef5449acad53555add69956a0cb884f0adf8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25272) Flink sql syntax analysis is inconsistent with the sql grammer declared now
Jianhui Dong created FLINK-25272: Summary: Flink sql syntax analysis is inconsistent with the sql grammer declared now Key: FLINK-25272 URL: https://issues.apache.org/jira/browse/FLINK-25272 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Jianhui Dong Syntax analysis in the code(https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/includes/parserImpls.ftl) is inconsistent with the grammar declared on the module(https://github.com/apache/flink/edit/master/docs/content/docs/dev/table/sql/create.md). The document states that all CREATE statement formats are `CREATE TABLE [IF NOT EXISTS]`, there is no `TEMPORAL` keywords, but in flink-sql-parser/src/main/codegen/includes/parserImpls.ftl::SqlCreateExtended, it will recorganize `TEMPORAL` token and set `isTemporary` with true if the keyword exists. Besides, I think the field `isTemporary` is outdated too, every table in Flink is temporal table as the document wrote. In fact, when they are stored in the catalog, they are all converted to `CatalogSchemaTable` which inherits the `TemporalTable` interface. BTW, the view and the function has the same condition. IMO, I think we should remove the `TEMPORAL` token in parserImpls.ftl and also remove the `isTemporary` field which now always is false. This issue may be a little confused cause my English is not very well. If you have any questions, you can add comments under the issue or contact me with lam...@apache.org. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] xinbinhuang commented on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
xinbinhuang commented on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-992183226 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28023) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread
flinkbot edited a comment on pull request #17988: URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654 ## CI report: * 140a48b370a6ceeec78c59636b169428a6649ccf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28014) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-19743) Add Source metrics definitions
[
https://issues.apache.org/jira/browse/FLINK-19743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-19743.
-
Fix Version/s: 1.14.0
(was: 1.12.6)
(was: 1.15.0)
Resolution: Duplicate
> Add Source metrics definitions
> --
>
> Key: FLINK-19743
> URL: https://issues.apache.org/jira/browse/FLINK-19743
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.14.0
>
>
> Add the metrics defined in
> [FLIP-33|https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics]
> to \{{OperatorMetricsGroup}} and {{SourceReaderContext}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-11682) Add abstract source and sink metric classes.
[ https://issues.apache.org/jira/browse/FLINK-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-11682: Fix Version/s: 1.14.0 > Add abstract source and sink metric classes. > > > Key: FLINK-11682 > URL: https://issues.apache.org/jira/browse/FLINK-11682 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > This is ticket will introduce a new flink-connectors-common module to > host the common classes shared by the connectors. In this case, standard > source and sink metrics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-11682) Add abstract source and sink metric classes.
[ https://issues.apache.org/jira/browse/FLINK-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-11682. - Resolution: Duplicate > Add abstract source and sink metric classes. > > > Key: FLINK-11682 > URL: https://issues.apache.org/jira/browse/FLINK-11682 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > > This is ticket will introduce a new flink-connectors-common module to > host the common classes shared by the connectors. In this case, standard > source and sink metrics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-11683) Implement standard metrics in Kafka connectors.
[ https://issues.apache.org/jira/browse/FLINK-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-11683. - Fix Version/s: 1.14.0 Resolution: Fixed > Implement standard metrics in Kafka connectors. > --- > > Key: FLINK-11683 > URL: https://issues.apache.org/jira/browse/FLINK-11683 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > Implement the standard metrics in Kafka connectors. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24491) ExecutionGraphInfo may not be archived when the dispatcher terminates
[
https://issues.apache.org/jira/browse/FLINK-24491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhilong Hong updated FLINK-24491:
-
Component/s: Runtime / Coordination
(was: Runtime / Configuration)
> ExecutionGraphInfo may not be archived when the dispatcher terminates
> -
>
> Key: FLINK-24491
> URL: https://issues.apache.org/jira/browse/FLINK-24491
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Zhilong Hong
>Priority: Major
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> When a job finishes, its JobManagerRunnerResult will be processed in the
> callback of {{Dispatcher#runJob}}. In the callback, ExecutionGraphInfo will
> be archived by HistoryServerArchivist asynchronously. However, the
> CompletableFuture of the archiving is ignored. The job may be removed before
> the archiving is finished. For the batch job running in the
> per-job/application mode, the dispatcher will terminate itself once the job
> is finished. In this case, ExecutionGraphInfo may not be archived when the
> dispatcher terminates.
> If the ExecutionGraphInfo is lost, users are not able to know whether the
> batch job is finished normally or not. They have to refer to the logs for the
> result.
> The session mode is not affected, since the dispatcher won't terminate itself
> once the job is finished. The HistoryServerArchivist gets enough time to
> archive the ExcutionGraphInfo.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] wenlong88 commented on a change in pull request #18042: [FLINK-25076][table-planner] Improve operator name for sql job
wenlong88 commented on a change in pull request #18042: URL: https://github.com/apache/flink/pull/18042#discussion_r767460228 ## File path: flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/nodes/exec/operator/BatchOperatorNameTest.xml ## @@ -0,0 +1,2099 @@ + + + + + +
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17622: [FLINK-24407][doc-zh]Pulsar connector chinese document link to Pulsar document location incorrectly.
flinkbot edited a comment on pull request #17622: URL: https://github.com/apache/flink/pull/17622#issuecomment-956197495 ## CI report: * 0338a82a3ec94de07e82ed9c72e0ef367ff67e2b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28015) * ba19443440c7613e48bacf4ca692ef207936a8b2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28018) * 0815a002a1ee08873b6270214a14caec145ac83a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28019) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r767459284
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param outputType The type information of the output element.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
Review comment:
After some offline discussion, we agree to remove this method for now
and add it back when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-25271) ApplicationDispatcherBootstrapITCase. testDispatcherRecoversAfterLosingAndRegainingLeadership failed on azure
Yun Gao created FLINK-25271:
---
Summary: ApplicationDispatcherBootstrapITCase.
testDispatcherRecoversAfterLosingAndRegainingLeadership failed on azure
Key: FLINK-25271
URL: https://issues.apache.org/jira/browse/FLINK-25271
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.14.0
Reporter: Yun Gao
{code:java}
Dec 12 04:22:42 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
Dec 12 04:22:42 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
Dec 12 04:22:42 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
Dec 12 04:22:42 at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Dec 12 04:22:42 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Dec 12 04:22:42 at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
Dec 12 04:22:42 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Dec 12 04:22:42
Dec 12 04:22:42 [INFO]
Dec 12 04:22:42 [INFO] Results:
Dec 12 04:22:42 [INFO]
Dec 12 04:22:42 [ERROR] Errors:
Dec 12 04:22:42 [ERROR]
ApplicationDispatcherBootstrapITCase.testDispatcherRecoversAfterLosingAndRegainingLeadership:124->awaitJobStatus:135->lambda$awaitJobStatus$2:138
» IllegalState
Dec 12 04:22:42 [INFO]
Dec 12 04:22:42 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0
Dec 12 04:22:42 [INFO]
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27993=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=9350
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24295) Too many requestPartitionState may jam the JobManager during task deployment
[
https://issues.apache.org/jira/browse/FLINK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458167#comment-17458167
]
Zhilong Hong commented on FLINK-24295:
--
Thank you for your replies, [~trohrmann] and [~huntercc]. I'm wondering what
should the default value of retry times be.
> Too many requestPartitionState may jam the JobManager during task deployment
>
>
> Key: FLINK-24295
> URL: https://issues.apache.org/jira/browse/FLINK-24295
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Zhilong Hong
>Priority: Major
>
> After the optimization of the phase 2 we've done in FLINK-21110, the speed of
> task deployment has accelerated. However, we find that during the task
> deployment, there may be too many {{requestPartitionState}} RPC calls from
> TaskManagers that would jam the JobManager.
> Why would there be too many {{requestPartitionState}} RPC calls? After the
> optimization, the JobManager can submit tasks to TaskManagers quickly. If
> JobManager calls {{submitTask}} faster than the speed of dealing with
> {{submitTask}} by TaskManagers, there may be a scenario that some
> TaskManagers deploy tasks faster than other TaskManagers.
> When a downstream task is deployed, it would try to request partitions from
> upstream tasks, which may be located at a remote TaskManager. If the upstream
> tasks are not deployed, it would request the partition state from JobManager.
> In the worst case, the complexity of the computation and memory would be
> O(N^2).
> In our test with a streaming job, which has two vertices with the 8,000
> parallelism and connected with all-to-all edges, in the worst case, there
> will be 32,000,000 {{requestPartitionState}} RPC calls in the JobManager.
> Each RPC call requires 1 KiB space in the heap memory of the JobManager. The
> overall space cost of {{requestPartitionState}} will be 32 GiB, which is a
> heavy burden for GC to deal with.
> In our test, the size of the heap memory of JobManager is 8 GiB. During the
> task deployment the JobManager gets more full GCs. The JobManager gets stuck
> since it is filled with full GCs and has no time to deal with the incoming
> RPC calls.
> The worst thing is that there's no log outputted for this RPC call. When a
> user find the JobManager is get slower or get stuck, he/she won't be able to
> find out why.
> Why does this case rarely happen before? Before the optimization, it takes a
> long time to calculate TaskDeploymentDescriptors and send them to
> TaskManagers. JobManager calls {{submitTask}} more slowly than the speed of
> dealing with {{submitTask}} by TaskManagers in most cases. Since the
> deployment of tasks are topologically sorted, the upstream tasks is deployed
> before the downstream tasks, and this case rarely happens.
> In my opinion, the solution of this issue needs more discussion. According to
> the discussion in the pull request
> ([https://github.com/apache/flink/pull/6680]), it's not safe to remove this
> RPC call, because we cannot always make sure the assumption that an upstream
> task failure will always fail the downstream consumers is always right.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25270) Azure failed due to stopped hearing from agent AlibabaCI006-agent01
[
https://issues.apache.org/jira/browse/FLINK-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458165#comment-17458165
]
Yun Gao commented on FLINK-25270:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27983=logs=fc5181b0-e452-5c8f-68de-1097947f6483
> Azure failed due to stopped hearing from agent AlibabaCI006-agent01
> ---
>
> Key: FLINK-25270
> URL: https://issues.apache.org/jira/browse/FLINK-25270
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent01. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Started: 周六 at 下午1:37
> Duration: <1s
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27982=logs=f2b08047-82c3-520f-51ee-a30fd6254285
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25270) Azure failed due to stopped hearing from agent AlibabaCI006-agent01
[
https://issues.apache.org/jira/browse/FLINK-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458166#comment-17458166
]
Yun Gao commented on FLINK-25270:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27984=logs=b1e44b80-6687-5cc5-6529-292f7212c609
> Azure failed due to stopped hearing from agent AlibabaCI006-agent01
> ---
>
> Key: FLINK-25270
> URL: https://issues.apache.org/jira/browse/FLINK-25270
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent01. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Started: 周六 at 下午1:37
> Duration: <1s
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27982=logs=f2b08047-82c3-520f-51ee-a30fd6254285
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25270) Azure failed due to stopped hearing from agent AlibabaCI006-agent01
[
https://issues.apache.org/jira/browse/FLINK-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25270:
Affects Version/s: 1.13.3
> Azure failed due to stopped hearing from agent AlibabaCI006-agent01
> ---
>
> Key: FLINK-25270
> URL: https://issues.apache.org/jira/browse/FLINK-25270
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent01. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Started: 周六 at 下午1:37
> Duration: <1s
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27982=logs=f2b08047-82c3-520f-51ee-a30fd6254285
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25270) Azure failed due to stopped hearing from agent AlibabaCI006-agent01
[
https://issues.apache.org/jira/browse/FLINK-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25270:
Affects Version/s: 1.15.0
> Azure failed due to stopped hearing from agent AlibabaCI006-agent01
> ---
>
> Key: FLINK-25270
> URL: https://issues.apache.org/jira/browse/FLINK-25270
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.3, 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent01. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Started: 周六 at 下午1:37
> Duration: <1s
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27982=logs=f2b08047-82c3-520f-51ee-a30fd6254285
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25267:
-
Description:
My Stateful Functions job is running on Kubernetes (minikube on my local env)
and has these settings:
* Using StateFun v3.1.0
* Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
* Checkpointing mode is EXACTLY_ONCE
* State backend is rocksdb and incremental checkpointing is enabled
When I kill the jobmanager (master) pod, minikube starts another pod and this
new pod fails when it tries to load last checkpoint:
{code:java}
...
2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job myStatefunApp
().
2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using restart back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for myStatefunApp ().
2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
[] - Initialized DefaultCompletedCheckpointStore in
'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
with /checkpoints/.
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Running initialization on master for job myStatefunApp
().
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Successfully ran initialization on master in 0 ms.
2021-12-11 14:25:26,617 INFO
org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built
1 pipelined regions in 1 ms
2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using job/cluster config to configure application-defined state
backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
predefined options: DEFAULT.
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
application-defined options factory:
DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using application-defined state backend:
EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Checkpoint storage is set to 'filesystem': (checkpoints
"hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
2021-12-11 14:25:26,712 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
Recovering checkpoints from
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,724 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found
1 checkpoints in
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to fetch 1 checkpoints from storage.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to retrieve checkpoint 2.
2021-12-11 14:25:26,931 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
from Checkpoint 2 @ 1639232587220 for
located at
hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp//chk-2.
2021-12-11 14:25:27,012 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job
failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:436)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
[jira] [Created] (FLINK-25270) Azure failed due to stopped hearing from agent AlibabaCI006-agent01
Yun Gao created FLINK-25270:
---
Summary: Azure failed due to stopped hearing from agent
AlibabaCI006-agent01
Key: FLINK-25270
URL: https://issues.apache.org/jira/browse/FLINK-25270
Project: Flink
Issue Type: Bug
Components: Build System / Azure Pipelines
Affects Versions: 1.14.0
Reporter: Yun Gao
{code:java}
##[error]We stopped hearing from agent AlibabaCI006-agent01. Verify the agent
machine is running and has a healthy network connection. Anything that
terminates an agent process, starves it for CPU, or blocks its network access
can cause this error. For more information, see:
https://go.microsoft.com/fwlink/?linkid=846610
Started: 周六 at 下午1:37
Duration: <1s
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27982=logs=f2b08047-82c3-520f-51ee-a30fd6254285
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25267:
-
Description:
My Stateful Functions job is running on Kubernetes (minikube on my local env)
and has these settings:
* Using StateFun v3.1.0
* Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
* Checkpointing mode is EXACTLY_ONCE
* State backend is rocksdb and incremental checkpointing is enabled
When I kill the jobmanager (master) pod, minikube starts another pod and this
new pod fails when it tries to load last checkpoint:
{code:java}
...
2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job myStatefunApp
().
2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using restart back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for myStatefunApp ().
2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
[] - Initialized DefaultCompletedCheckpointStore in
'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
with /checkpoints/.
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Running initialization on master for job myStatefunApp
().
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Successfully ran initialization on master in 0 ms.
2021-12-11 14:25:26,617 INFO
org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built
1 pipelined regions in 1 ms
2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using job/cluster config to configure application-defined state
backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
predefined options: DEFAULT.
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
application-defined options factory:
DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using application-defined state backend:
EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Checkpoint storage is set to 'filesystem': (checkpoints
"hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
2021-12-11 14:25:26,712 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
Recovering checkpoints from
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,724 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found
1 checkpoints in
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to fetch 1 checkpoints from storage.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to retrieve checkpoint 2.
2021-12-11 14:25:26,931 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
from Checkpoint 2 @ 1639232587220 for
located at
hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp//chk-2.
2021-12-11 14:25:27,012 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job
failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:436)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * b5c67380fb1c5e0dd6e757a848b916a4c2b4db71 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25269) Quickstarts Scala nightly end-to-end test failed on azure due to could not connect to resource manager
Yun Gao created FLINK-25269:
---
Summary: Quickstarts Scala nightly end-to-end test failed on azure
due to could not connect to resource manager
Key: FLINK-25269
URL: https://issues.apache.org/jira/browse/FLINK-25269
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Dec 10 13:12:17 at akka.actor.Actor.aroundReceive(Actor.scala:537)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at akka.actor.Actor.aroundReceive$(Actor.scala:535)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at akka.actor.ActorCell.invoke(ActorCell.scala:548)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at akka.dispatch.Mailbox.run(Mailbox.scala:231)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
[flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 at
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_292]
Dec 10 13:12:17 at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[?:1.8.0_292]
Dec 10 13:12:17 at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[?:1.8.0_292]
Dec 10 13:12:17 at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
[?:1.8.0_292]
Dec 10 13:12:17 2021-12-10 13:12:07,296 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Starting the resource manager.
Dec 10 13:12:17 2021-12-10 13:12:09,426 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registering TaskManager with ResourceID 10.1.0.79:35275-d26a40
(akka.tcp://flink@10.1.0.79:35275/user/rpc/taskmanager_0) at ResourceManager
Dec 10 13:12:17 2021-12-10 13:12:09,496 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registering TaskManager with ResourceID 10.1.0.79:35275-d26a40
(akka.tcp://flink@10.1.0.79:35275/user/rpc/taskmanager_0) at ResourceManager
Dec 10 13:12:17 2021-12-10 13:12:13,189 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
JobGraph submission 'Elasticsearch7.x end to end sink test example'
(effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,190 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting
job 'Elasticsearch7.x end to end sink test example'
(effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,225 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC
endpoint for org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/rpc/jobmanager_2 .
Dec 10 13:12:17 2021-12-10 13:12:13,252 INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing
job 'Elasticsearch7.x end to end sink test example'
(effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,324 INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart
back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for Elasticsearch7.x end to end sink test example
(effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,428 INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job Elasticsearch7.x end to end sink test example
(effb86be1cfd961e947988200a724de7).
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) * 85b0ac98a2e575635ab621ec474dffd3bea53f18 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25264) Does the binlog_row_image of MySQL must be in full mode when Flink CDC is used?
[
https://issues.apache.org/jira/browse/FLINK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458161#comment-17458161
]
Martijn Visser commented on FLINK-25264:
[~jark] [~Leonard] Any thoughts on this? Or should this be raised as an issue
at https://github.com/ververica/flink-cdc-connectors ?
> Does the binlog_row_image of MySQL must be in full mode when Flink CDC is
> used?
> ---
>
> Key: FLINK-25264
> URL: https://issues.apache.org/jira/browse/FLINK-25264
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.3
>Reporter: coco
>Priority: Major
> Labels: Flink-CDC, MySQL
> Fix For: 1.13.3
>
>
> When I use Flink CDC to connect to MySQL, I encounter the following problems:
> Connection gracefully closed
> Uncaught exception in the SplitEnumerator for Source Source:
> TableSourceScan(table=[[default_catalog, default_database,
> t_alarm_op_record_202112]], fields=[CSN, OPERATION, OPERATOR, UTCTIME,
> CONTENT]) while starting the SplitEnumerator.. Triggering job failover.
> org.apache.flink.table.api.ValidationException: {color:#FF}*The MySQL
> server is configured with binlog_row_image MINIMAL rather than FULL, which is
> required for this connector to work properly. Change the MySQL configuration
> to use a binlog_row_image=FULL and restart the connector.*{color}
> at
> com.ververica.cdc.connectors.mysql.MySqlValidator.checkBinlogRowImage(MySqlValidator.java:132)
> ~[flink-connector-mysql-cdc-2.0.0.jar:2.0.0]
> ...
>
> Check the MySQL configuration.
> Mysql> select @@global.binlog_row_image,@@session.binlog_row_image;
> It is indeed MINIMAL.
> Does the binlog_row_image of MySQL must be in full mode when Flink CDC is
> used?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #37: [FLINK-24955] Add Estimator and Transformer for One Hot Encoder
lindong28 commented on a change in pull request #37:
URL: https://github.com/apache/flink-ml/pull/37#discussion_r767432991
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/linalg/typeinfo/SparseVectorSerializer.java
##
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.linalg.typeinfo;
+
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.ml.linalg.SparseVector;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+/** Specialized serializer for {@link SparseVector}. */
+public final class SparseVectorSerializer extends
TypeSerializerSingleton {
+
+private static final long serialVersionUID = 1L;
+
+private static final double[] EMPTY_DOUBLE_ARRAY = new double[0];
+
+private static final int[] EMPTY_INT_ARRAY = new int[0];
+
+private static final SparseVectorSerializer INSTANCE = new
SparseVectorSerializer();
Review comment:
Could you change this variable to public and let
`SparseVectorTypeInfo::createSerializer(...)`? This would be consistent with
the existing `ValueTypeInfo::createSerializer` in Flink.
Could you update `DenseVectorTypeInfo::createSerializer(...)` as well?
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/linalg/typeinfo/SparseVectorTypeInfo.java
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.linalg.typeinfo;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.ml.linalg.SparseVector;
+
+/** A {@link TypeInformation} for the {@link SparseVector} type. */
+public class SparseVectorTypeInfo extends TypeInformation {
+@Override
Review comment:
Could you add `public static final SparseVectorTypeInfo INSTANCE = new
SparseVectorTypeInfo()` for consistency with other TypeInformation subclasses?
##
File path: flink-ml-core/src/test/java/org/apache/flink/ml/api/StageTest.java
##
@@ -388,5 +388,13 @@ public void testValidators() {
ParamValidator notNull = ParamValidators.notNull();
Assert.assertTrue(notNull.validate(5));
Assert.assertFalse(notNull.validate(null));
+
+ParamValidator nonEmptyArray =
ParamValidators.nonEmptyArray();
+Assert.assertTrue(nonEmptyArray.validate(new String[] {"1"}));
+Assert.assertFalse(nonEmptyArray.validate(null));
+Assert.assertFalse(nonEmptyArray.validate(new String[0]));
+
+ParamValidator nonEmptyArrayInteger =
ParamValidators.nonEmptyArray();
Review comment:
hmm.. should its type be `ParamValidator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25267:
-
Description:
My Stateful Functions job is running on Kubernetes (minikube on my local env)
and has these settings:
* Using StateFun v3.1.0
* Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
* Checkpointing mode is EXACTLY_ONCE
* State backend is rocksdb and incremental checkpointing is enabled
When I kill the jobmanager (master) pod, minikube starts another pod and this
new pod fails when it tries to load last checkpoint:
{code:java}
...
2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job myStatefunApp
().
2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using restart back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for myStatefunApp ().
2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
[] - Initialized DefaultCompletedCheckpointStore in
'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
with /checkpoints/.
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Running initialization on master for job myStatefunApp
().
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Successfully ran initialization on master in 0 ms.
2021-12-11 14:25:26,617 INFO
org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built
1 pipelined regions in 1 ms
2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using job/cluster config to configure application-defined state
backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
predefined options: DEFAULT.
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
application-defined options factory:
DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using application-defined state backend:
EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Checkpoint storage is set to 'filesystem': (checkpoints
"hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
2021-12-11 14:25:26,712 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
Recovering checkpoints from
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,724 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found
1 checkpoints in
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to fetch 1 checkpoints from storage.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to retrieve checkpoint 2.
2021-12-11 14:25:26,931 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
from Checkpoint 2 @ 1639232587220 for
located at
hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp//chk-2.
2021-12-11 14:25:27,012 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job
failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:436)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
[jira] [Updated] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25267:
-
Description:
My Stateful Functions job is running on Kubernetes (minikube on my local env)
and has these settings:
* Using StateFun v3.1.0
* Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
* Checkpointing mode is EXACTLY_ONCE
* State backend is rocksdb and incremental checkpointing is enabled
When I kill the jobmanager (master) pod, minikube starts another pod and this
new pod fails when it tries to load last checkpoint:
{code:java}
...
2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job myStatefunApp
().
2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using restart back off time strategy
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
backoffTimeMS=1000) for myStatefunApp ().
2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
[] - Initialized DefaultCompletedCheckpointStore in
'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
with /checkpoints/.
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Running initialization on master for job myStatefunApp
().
2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Successfully ran initialization on master in 0 ms.
2021-12-11 14:25:26,617 INFO
org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built
1 pipelined regions in 1 ms
2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using job/cluster config to configure application-defined state
backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
predefined options: DEFAULT.
2021-12-11 14:25:26,627 INFO
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] - Using
application-defined options factory:
DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Using application-defined state backend:
EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
writeBatchSize=2097152}
2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Checkpoint storage is set to 'filesystem': (checkpoints
"hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
2021-12-11 14:25:26,712 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
Recovering checkpoints from
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,724 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found
1 checkpoints in
ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to fetch 1 checkpoints from storage.
2021-12-11 14:25:26,725 INFO
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying
to retrieve checkpoint 2.
2021-12-11 14:25:26,931 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
from Checkpoint 2 @ 1639232587220 for
located at
hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp//chk-2.
2021-12-11 14:25:27,012 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job
failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:436)
~[flink-dist_2.12-1.13.2.jar:1.13.2]
at
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28022) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17873: [FLINK-25009][CLI] Output slotSharingGroup as part of JsonGraph
flinkbot edited a comment on pull request #17873: URL: https://github.com/apache/flink/pull/17873#issuecomment-975923151 ## CI report: * 5ee33b6fada620c83fad2b0ab09d2354b3bca583 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26874) * 8fbd048baecf16ef1a7b4a7befb4c4a2b5cc1af4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25266) StreamingKafkaITCase fails with incorrect number of messages
[
https://issues.apache.org/jira/browse/FLINK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458160#comment-17458160
]
Yun Gao commented on FLINK-25266:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27973=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=11139
> StreamingKafkaITCase fails with incorrect number of messages
>
>
> Key: FLINK-25266
> URL: https://issues.apache.org/jira/browse/FLINK-25266
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Major
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27973=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a
> {code}
> Dec 10 17:21:01 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 1,
> Time elapsed: 163.429 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase
> Dec 10 17:21:01 [ERROR] StreamingKafkaITCase.testKafka Time elapsed: 163.421
> s <<< ERROR!
> Dec 10 17:21:01 java.io.IOException: Could not read expected number of
> messages.
> Dec 10 17:21:01 at
> org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.readMessage(LocalStandaloneKafkaResource.java:368)
> Dec 10 17:21:01 at
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:186)
> Dec 10 17:21:01 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Dec 10 17:21:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Dec 10 17:21:01 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Dec 10 17:21:01 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 10 17:21:01 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Dec 10 17:21:01 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Dec 10 17:21:01 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Dec 10 17:21:01 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Dec 10 17:21:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Dec 10 17:21:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Dec 10 17:21:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Dec 10 17:21:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Dec 10 17:21:01 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Dec 10 17:21:01 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r767446298
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/util/ReadWriteUtils.java
##
@@ -329,33 +342,42 @@ public static void updateExistingParams(Stage stage,
Map, Object> pa
* @param model The model data stream.
* @param path The parent directory of the model data file.
* @param modelEncoder The encoder to encode the model data.
+ * @param modelIndex The index of the table to save.
* @param The class type of the model data.
*/
public static void saveModelData(
-DataStream model, String path, Encoder modelEncoder) {
+DataStream model, String path, Encoder modelEncoder, int
modelIndex) {
FileSink sink =
FileSink.forRowFormat(
-new
org.apache.flink.core.fs.Path(getDataPath(path)), modelEncoder)
+new
org.apache.flink.core.fs.Path(getDataPath(path, modelIndex)),
+modelEncoder)
.withRollingPolicy(OnCheckpointRollingPolicy.build())
.withBucketAssigner(new BasePathBucketAssigner<>())
.build();
model.sinkTo(sink);
}
/**
- * Loads the model data from the given path using the model decoder.
+ * Loads the model table with index `modelIndex` from the given path using
the model decoder.
*
* @param env A StreamExecutionEnvironment instance.
* @param path The parent directory of the model data file.
* @param modelDecoder The decoder used to decode the model data.
+ * @param modelIndex The index of the table to load.
* @param The class type of the model data.
* @return The loaded model data.
*/
-public static DataStream loadModelData(
-StreamExecutionEnvironment env, String path, SimpleStreamFormat
modelDecoder) {
+public static Table loadModelData(
Review comment:
I thought there would be cases for the model data that contains multiple
tables. After some offline discussion, we aggree to remove `modelIndex` for now
and support multiple tables when we come to the real use case.
We also aggree to let `loadModelData ` to return a DataStream for two
reasons: (1) save and load could be symmetric (2) `saveModelData` and
`loadModelData` are utility functions for developers, who are often using
`DataStream`s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25174) Introduce ManagedTableFactory
[
https://issues.apache.org/jira/browse/FLINK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee reassigned FLINK-25174:
Assignee: Jingsong Lee
> Introduce ManagedTableFactory
> -
>
> Key: FLINK-25174
> URL: https://issues.apache.org/jira/browse/FLINK-25174
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.15.0
>
>
> We need an interface to discover the managed table factory implementation for
> managed table:
> {code:java}
> /**
> * Base interface for configuring a managed dynamic table connector. The
> managed table factory is
> * used when there is no {@link FactoryUtil#CONNECTOR} option.
> */
> @Internal
> public interface ManagedTableFactory extends DynamicTableFactory {
>
> @Override
> default String factoryIdentifier() {
> return "";
> }
>
> /**
> * Enrich options from catalog and session information.
> *
> * @return new options of this table.
> */
> Map enrichOptions(Context context);
>
> /** Notifies the listener that a table creation occurred. */
> void onCreateTable(Context context);
>
> /** Notifies the listener that a table drop occurred. */
> void onDropTable(Context context);
> } {code}
> A catalog that supports built-in dynamic table needs to implement the method
> in the Catalog (The GenericInMemoryCatalog and HiveCatalog will implement
> this method):
> {code:java}
> /**
> * If return true, the Table without specified connector will be translated
> to the Flink managed table.
> * See {@link CatalogBaseTable.TableKind#MANAGED}
> */
> default boolean supportsManagedTable {
> return false;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r76701
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param outputType The type information of the output element.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
Review comment:
We let the users explicitly provide the `outputType` here because
`TypeExtractor` may not be able infer the output type in some cases. (e.g.,
when the output type is a `Row`, we cannot infer each field of the row).
Note that this design pattern is already used in DataStream. Please checkout
the code example for `DataStream#map`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) * 8e2252bcb560bb6168f69f51d004e5ac849b8729 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28021) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * e31d3b92263e728724977d271d8ae4305d5055de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27632) * aa2f6cbac7a1495b57ff086b3a0188901ad198b9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zuston edited a comment on pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
zuston edited a comment on pull request #18087: URL: https://github.com/apache/flink/pull/18087#issuecomment-992139897 Could you help check this features? @wangyang0918 @KarmaGYZ @XComp @tillrohrmann Now this PR is just draft, if this feature could be invovled in Flink, i will optimize. And maybe jobmanager(Yarn application) node label should also be supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25268) Support task manager node-label in Yarn deployment
[ https://issues.apache.org/jira/browse/FLINK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458159#comment-17458159 ] Junfan Zhang commented on FLINK-25268: -- Could you help check this features? [~wangyang0918] [~guoyangze] Now this PR is just draft, if this feature could be invovled in Flink, i will optimize. And maybe jobmanager(Yarn application) node label should also be supported. > Support task manager node-label in Yarn deployment > -- > > Key: FLINK-25268 > URL: https://issues.apache.org/jira/browse/FLINK-25268 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > Labels: pull-request-available > > Now Flink only support application level node label, it's necessary to > introduce task manager level node-label on Yarn deployment. > h2. Why we need it? > Sometimes we will implement Flink to support deep learning payload using GPU, > so if having this feature, job manager and task managers could use different > nodes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] liufangqi commented on pull request #18059: [FLINK-25224][filesystem] Bump Hadoop version to 2.8.4
liufangqi commented on pull request #18059: URL: https://github.com/apache/flink/pull/18059#issuecomment-992141557 @deadwind4 Could you help merging this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liufangqi commented on pull request #18059: [FLINK-25224][filesystem] Bump Hadoop version to 2.8.4
liufangqi commented on pull request #18059: URL: https://github.com/apache/flink/pull/18059#issuecomment-992140494 > LGTM. THX！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zuston commented on pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
zuston commented on pull request #18087: URL: https://github.com/apache/flink/pull/18087#issuecomment-992139897 Could you help check this features? @wangyang0918 @KarmaGYZ Now this PR is just draft, if this feature could be invovled in Flink, i will optimize. And maybe jobmanager(Yarn application) node label should also be supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
flinkbot edited a comment on pull request #18087: URL: https://github.com/apache/flink/pull/18087#issuecomment-992135476 ## CI report: * f34d532366bda2f53429bf6ebc46d43e49b498b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28020) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
flinkbot commented on pull request #18087: URL: https://github.com/apache/flink/pull/18087#issuecomment-992136125 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f34d532366bda2f53429bf6ebc46d43e49b498b5 (Mon Dec 13 06:03:14 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25268).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
flinkbot commented on pull request #18087: URL: https://github.com/apache/flink/pull/18087#issuecomment-992135476 ## CI report: * f34d532366bda2f53429bf6ebc46d43e49b498b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25268) Support task manager node-label in Yarn deployment
[ https://issues.apache.org/jira/browse/FLINK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25268: --- Labels: pull-request-available (was: ) > Support task manager node-label in Yarn deployment > -- > > Key: FLINK-25268 > URL: https://issues.apache.org/jira/browse/FLINK-25268 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > Labels: pull-request-available > > Now Flink only support application level node label, it's necessary to > introduce task manager level node-label on Yarn deployment. > h2. Why we need it? > Sometimes we will implement Flink to support deep learning payload using GPU, > so if having this feature, job manager and task managers could use different > nodes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] zuston opened a new pull request #18087: [FLINK-25268] Support task manager node-label in Yarn deployment
zuston opened a new pull request #18087: URL: https://github.com/apache/flink/pull/18087 ## What is the purpose of the change Now Flink only support application level node label, it's necessary to introduce task manager level node-label on Yarn deployment. ### Why we need it? Sometimes we will implement Flink to support deep learning payload using GPU, so if having this feature, job manager and task managers could use different nodes. ## Brief change log Introduce node-label of task manager level in Yarn deployment. ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #37: [FLINK-24955] Add Estimator and Transformer for One Hot Encoder
lindong28 commented on a change in pull request #37:
URL: https://github.com/apache/flink-ml/pull/37#discussion_r767430090
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/linalg/SparseVector.java
##
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.api.common.typeinfo.TypeInfo;
+import org.apache.flink.ml.linalg.typeinfo.SparseVectorTypeInfoFactory;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Arrays;
+import java.util.Objects;
+
+/** A sparse vector of double values. */
+@TypeInfo(SparseVectorTypeInfoFactory.class)
+public class SparseVector implements Vector {
+public final int n;
+public final int[] indices;
+public final double[] values;
+
+public SparseVector(int n, int[] indices, double[] values) {
+this.n = n;
+this.indices = indices;
+this.values = values;
+checkSizeAndIndicesRange();
Review comment:
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-24555) Incorrectly put empty list to flink configuration
[
https://issues.apache.org/jira/browse/FLINK-24555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24555:
---
Labels: pull-request-available (was: )
> Incorrectly put empty list to flink configuration
> -
>
> Key: FLINK-24555
> URL: https://issues.apache.org/jira/browse/FLINK-24555
> Project: Flink
> Issue Type: Bug
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
>
> h2. Why
> As I found some warn logs in our production flink jobs, like as follows(the
> detail logs attached at the end). Besides we dont set the
> {{pipeline.classpaths}} param conf.
> {code:java}
> 2021-10-14 17:46:41 WARN - [main] -
> org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying
> to split key and value in configuration file
> /data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_01/flink-conf.yaml:11:
> "pipeline.classpaths: "
> {code}
> I dig the flink code and found it put the empty list into configuration by
> {{ConfigUtils.encodeCollectionToConfig}}.
> h2. How
> So it's better to ignore the empty collection to put in the configuration.
> Because these configuration will be written to the flink-conf.yaml and Flink
> jobmanager will read but fail to read the corresponding key's value as the
> warn log described.
> h2. Appendix
> {code:java}
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: internal.jobgraph-path, job.graph
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property:
> restart-strategy.failure-rate.max-failures-per-interval, 3
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: jobmanager.execution.failover-strategy, region
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: high-availability.cluster-id,
> application_1633896099002_1043646
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: execution.runtime-mode, AUTOMATIC
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: security.kerberos.fetch.delegation-token, false
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: parallelism.default, 2
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-10-14 17:46:41 WARN - [main] -
> org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying
> to split key and value in configuration file
> /data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_01/flink-conf.yaml:11:
> "pipeline.classpaths: "
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: restart-strategy.failure-rate.failure-rate-interval,
> 1 d
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: yarn.application.name,
> app_StreamEngine_Prod_jiandan_beat_rec_test_all
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: yarn.application.queue, talos.job_streaming
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: taskmanager.memory.process.size, 1728m
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zuston closed pull request #17489: [FLINK-24555] Incorrectly put empty list to flink configuration
zuston closed pull request #17489: URL: https://github.com/apache/flink/pull/17489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25268) Support task manager node-label in Yarn deployment
[ https://issues.apache.org/jira/browse/FLINK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junfan Zhang updated FLINK-25268: - Description: Now Flink only support application level node label, it's necessary to introduce task manager level node-label on Yarn deployment. h2. Why we need it? Sometimes we will implement Flink to support deep learning payload using GPU, so if having this feature, job manager and task managers could use different nodes. was:Now Flink only support application level node label, it's necessary to introduce task manager level node-label on Yarn deployment. > Support task manager node-label in Yarn deployment > -- > > Key: FLINK-25268 > URL: https://issues.apache.org/jira/browse/FLINK-25268 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > Now Flink only support application level node label, it's necessary to > introduce task manager level node-label on Yarn deployment. > h2. Why we need it? > Sometimes we will implement Flink to support deep learning payload using GPU, > so if having this feature, job manager and task managers could use different > nodes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25268) Support task manager node-label in Yarn deployment
Junfan Zhang created FLINK-25268: Summary: Support task manager node-label in Yarn deployment Key: FLINK-25268 URL: https://issues.apache.org/jira/browse/FLINK-25268 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang Now Flink only support application level node label, it's necessary to introduce task manager level node-label on Yarn deployment. -- This message was sent by Atlassian Jira (v8.20.1#820001)
