(flink-shaded) annotated tag release-16.2-rc1 updated (ecfcb42 -> 833bf52)
This is an automated email from the ASF dual-hosted git repository. guoweijie pushed a change to annotated tag release-16.2-rc1 in repository https://gitbox.apache.org/repos/asf/flink-shaded.git *** WARNING: tag release-16.2-rc1 was modified! *** from ecfcb42 (commit) to 833bf52 (tag) tagging ecfcb429c9f94e72dca1f08adebbfc046b7be6fe (commit) replaces release-16.1 by Weijie Guo on Mon Nov 13 14:33:26 2023 +0800 - Log - release-16.2-rc1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEjVaubnCCaZpIcHUOpOjEwF7mhh8FAmVRwzYACgkQpOjEwF7m hh97Lw/+PKRxjw29ZNZxp6TMXk88Su1sxc4+XlZRLPKOZk/6xcmMJd5Jq8PqBgmu TzrWpeMwr+CpPHjkK8rqdXgZcaSDzvV/ohl0THV4+w3rEqMOHHv00y4DLTzaPBKw LjM8V0vDzPegedwdFuiTgSbN/bRHUyUUbvQVXvV69y2xAZlWJqV7lpliPVp1ly6Y dTAHYtcH3Tc/hKSMHv2ixQtVauRgptmrmaE60jlQ6/fpGq3aJT50JWVpTGbBgioT q1i1rgjRTINooNXxjFFrw9qBtaN7wZXizkRkfGXgVWO3zKQ+z4lI1ztcnRaUyD/0 Q3IFx7+WdRksmK292OPRZY+XJ2+dO8zWSTm5Lk/S9M53E8otO354dChnf6F7g8ow /AlBv5TD8gURzmO2iryKPenTgt3Pnb9OGxJ21XmOmse3waKKVFkRpITjFtSN3XXr 5LPKDNg5mmbNSNtI3RGNLNXcZTSwdIYz+3wpr08TR6fQNhgZqrvrjYnlyM3hNRE8 /aMjWvSk4hmwHGS8UoULIL6KT3ssza9ou3Jl/ucS4m7EY81UvFjMnY8parEb+fm1 8hNaWGHdNv+p+myscPimgfSB7eMhfMLL7POEhhx+LYUtt+7Z792/h/QwQCzuuN6X PIM09YNyns0KrOJJSiXIsicM0uFPh6ehjI3AlZdaO0sqJBTcpWg= =uvMo -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
svn commit: r65248 - in /dev/flink/flink-shaded-16.2-rc1: ./ flink-shaded-16.2-src.tgz flink-shaded-16.2-src.tgz.asc flink-shaded-16.2-src.tgz.sha512
Author: guoweijie Date: Mon Nov 13 06:51:46 2023 New Revision: 65248 Log: Add flink-shaded 16.2-rc1 Added: dev/flink/flink-shaded-16.2-rc1/ dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz (with props) dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.asc dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.sha512 Added: dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz == Binary file - no diff available. Propchange: dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz -- svn:mime-type = application/octet-stream Added: dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.asc == --- dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.asc (added) +++ dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.asc Mon Nov 13 06:51:46 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEjVaubnCCaZpIcHUOpOjEwF7mhh8FAmVRw5QACgkQpOjEwF7m +hh8oDxAAweWo34ijCNvcG79XcjNiowStF0YeeRzaJfYIRJegMHD0bAtAi2XqT6Gy +enSkvXt82Ho7DzC8dVKPTHlQwakMjwKDJGz7SEiSENNKA+IU9yk/izOq8+1iPBq9 +MkXNI4HruKUNolcOrMNghxkxVRvfEk4Rh2goIPAFgwHMjaxwgXaXXDfTJMIHLHtB +Uu6xZLXotgySJBeJZjpZgPxmmRBtLDuSAK9mm6I8yQwMMMxp9CMipi5UJZylEHgN +1ME7PPipfL56v/rV5Ek2B8PF4+VEXZNouZvNcYiDEGpO/BqK9LfWoLTXG3k0W8Ua +o7QjN1Iz2PI2dEEGhGuNfWUH2UOqqJ6biObLnTRpVS6/3aSDv0foG0onOOKlj+IN +C8DdZYgUYKm86v9oBVB/g1H4JEqX/pBTzlnsfqpjfEftkyBuUGozXsne23vKvnFR +CbajpHJT2I5DmPV4nOpsSRHTHUXq/MeSwnUDVY0i/+uFtLKLyz089YX9YOhKA3kG +CHzq357kTPptvoVhMF1L7aS+KrcR+HqQaGAk7VKf6GYZyoVrjb2JoUgch/lRvV8n +5g8SwV5VQl1FfQ7P14T3J15rq1efpGEPxrM8mJJ3+sZAAFqLUtjn/uELD/8WN+BV +gsSqBryDSjlxm+VYqfcw9FIYsJb+EPz8FEmY77spDzyGrMHgnh0= +=WmC7 +-END PGP SIGNATURE- Added: dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.sha512 == --- dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.sha512 (added) +++ dev/flink/flink-shaded-16.2-rc1/flink-shaded-16.2-src.tgz.sha512 Mon Nov 13 06:51:46 2023 @@ -0,0 +1 @@ +537ff7568bbf912f753d7db998fafda44ba668364c0cc7e4fd3477ccce96bca0e9f3b1cbeaf5ef883d6cc0ce8ffc7f1a65880d67419b91fcddcffb57b40b6d50 flink-shaded-16.2-src.tgz
svn commit: r65247 - /dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_.tgz.sha512
Author: tangyun Date: Mon Nov 13 06:47:16 2023 New Revision: 65247 Log: Delete uncessary files. Removed: dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_.tgz.sha512
svn commit: r65241 - in /dev/flink/flink-1.17.2-rc1: ./ python/
Author: tangyun Date: Mon Nov 13 06:32:26 2023 New Revision: 65241 Log: Add flink-1.17.2-rc1 Added: dev/flink/flink-1.17.2-rc1/ dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_.tgz.sha512 dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_2.12.tgz (with props) dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_2.12.tgz.asc dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_2.12.tgz.sha512 dev/flink/flink-1.17.2-rc1/flink-1.17.2-src.tgz (with props) dev/flink/flink-1.17.2-rc1/flink-1.17.2-src.tgz.asc dev/flink/flink-1.17.2-rc1/flink-1.17.2-src.tgz.sha512 dev/flink/flink-1.17.2-rc1/python/ dev/flink/flink-1.17.2-rc1/python/apache-flink-1.17.2.tar.gz (with props) dev/flink/flink-1.17.2-rc1/python/apache-flink-1.17.2.tar.gz.asc dev/flink/flink-1.17.2-rc1/python/apache-flink-1.17.2.tar.gz.sha512 dev/flink/flink-1.17.2-rc1/python/apache-flink-libraries-1.17.2.tar.gz (with props) dev/flink/flink-1.17.2-rc1/python/apache-flink-libraries-1.17.2.tar.gz.asc dev/flink/flink-1.17.2-rc1/python/apache-flink-libraries-1.17.2.tar.gz.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_11_0_arm64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_11_0_arm64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-macosx_11_0_arm64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-manylinux1_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-manylinux1_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp310-cp310-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-manylinux1_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp37-cp37m-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_11_0_arm64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_11_0_arm64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-macosx_11_0_arm64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-manylinux1_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-manylinux1_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp38-cp38-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_11_0_arm64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_11_0_arm64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-macosx_11_0_arm64.whl.sha512 dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-manylinux1_x86_64.whl (with props) dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-manylinux1_x86_64.whl.asc dev/flink/flink-1.17.2-rc1/python/apache_flink-1.17.2-cp39-cp39-manylinux1_x86_64.whl.sha512 Added: dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_.tgz.sha512 == (empty) Added: dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_2.12.tgz == Binary file - no diff available. Propchange: dev/flink/flink-1.17.2-rc1/flink-1.17.2-bin-scala_2.12.tgz -- svn:mime-type = application/octet-stream Added:
(flink) annotated tag release-1.17.2-rc1 updated (c0027e5777f -> edda9800b85)
This is an automated email from the ASF dual-hosted git repository. tangyun pushed a change to annotated tag release-1.17.2-rc1 in repository https://gitbox.apache.org/repos/asf/flink.git *** WARNING: tag release-1.17.2-rc1 was modified! *** from c0027e5777f (commit) to edda9800b85 (tag) tagging c0027e5777f9d77970fdb99bcc158d65ea48d514 (commit) replaces pre-apache-rename by Yun Tang on Fri Nov 10 10:03:16 2023 +0800 - Log - release-1.17.2-rc1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEELg4atdOdVeYIBx+595XAKk0kgrMFAmVNj2QACgkQ95XAKk0k grNyWBAAmzFdGbq76UNRQVeEGmDapozN4E2aXKGrWFYsCg4EirHN2FocLlcsNT1J xSUcGZkr70TF3W2aSz668svY9QSK9P1bCW7TdjaYzwCi3e9hN7SvsIitGEzYY54A +j5gKRomI+fJHwhkOd1NcFrpdTCpaUNSm/jYx2wVWwDcA1+7kRDfbuXeA9CS0bPK E3m0486UmpkIVvG4FRXs3/Q+iPs4cCjvX4z/3YEBNTA60FPytzfFENBsHmkd7C2Q rK9ws6DIfVPcNxgBwezhxGU9jH7xyMAKpYRipBb5ww/PBQHIfMRFeAjkB2kBjbMt qSLBMmMqFaP91xxtrS9o8IFPvJKeMBNAq3v1Fq+1OmClg9Jt1QNchTPfzho1c+up a/+dkXerxLEAF/xsikYQuvJNJX9rJuQDHZrSeUhMbYH8AUk0tVvNCpnzshDj6keA 2hQi7TMayITGjG/0biuaqlHaI45XkmDwglF5NRBgMhmBqGjCj2oFzQ3BoqyL1u84 qQTeY1S0qDlVsLQbO/iTqUYwaKnXK1A9bHv3FpaBuYtL5VP+t0mF65gW4MvCcqRi ZwFk+QY9I58eNENcm29SgwCrjNBAI+INi3i3olyHeXcsuMkwsZz0TxRY3eo66FNU DV8t3Fxt7x0ih5dF2j3LY4rqEhSLWA465e5XqI3T6I1TXZ5EBRQ= =//qZ -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
(flink-kubernetes-operator) branch main updated: [FLINK-33527][autoscaler] Clear all physical states after autoscaler is disabled
This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new bf5bb94b [FLINK-33527][autoscaler] Clear all physical states after
autoscaler is disabled
bf5bb94b is described below
commit bf5bb94bc87295e5ecb8ace8f579a22dca440e2d
Author: Rui Fan <1996fan...@gmail.com>
AuthorDate: Mon Nov 13 11:44:43 2023 +0800
[FLINK-33527][autoscaler] Clear all physical states after autoscaler is
disabled
---
.../apache/flink/autoscaler/JobAutoScalerImpl.java | 45 --
.../autoscaler/metrics/ScalingHistoryUtils.java| 49 ---
.../flink/autoscaler/JobAutoScalerImplTest.java| 72 ++
3 files changed, 139 insertions(+), 27 deletions(-)
diff --git
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
index 69c646f5..663a8669 100644
---
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
+++
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
@@ -29,10 +29,12 @@ import org.apache.flink.autoscaler.realizer.ScalingRealizer;
import org.apache.flink.autoscaler.state.AutoScalerStateStore;
import org.apache.flink.configuration.PipelineOptions;
import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.util.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.time.Clock;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
@@ -41,6 +43,7 @@ import java.util.concurrent.ConcurrentHashMap;
import static
org.apache.flink.autoscaler.config.AutoScalerOptions.AUTOSCALER_ENABLED;
import static
org.apache.flink.autoscaler.metrics.AutoscalerFlinkMetrics.initRecommendedParallelism;
import static
org.apache.flink.autoscaler.metrics.AutoscalerFlinkMetrics.resetRecommendedParallelism;
+import static
org.apache.flink.autoscaler.metrics.ScalingHistoryUtils.trimScalingHistory;
/** The default implementation of {@link JobAutoScaler}. */
public class JobAutoScalerImpl>
@@ -57,6 +60,8 @@ public class JobAutoScalerImpl>
private final ScalingRealizer scalingRealizer;
private final AutoScalerStateStore stateStore;
+private Clock clock = Clock.systemDefaultZone();
+
@VisibleForTesting
final Map>>
lastEvaluatedMetrics = new ConcurrentHashMap<>();
@@ -86,7 +91,7 @@ public class JobAutoScalerImpl>
try {
if (!ctx.getConfiguration().getBoolean(AUTOSCALER_ENABLED)) {
LOG.debug("Autoscaler is disabled");
-clearParallelismOverrides(ctx);
+clearStatesAfterAutoscalerDisabled(ctx);
return;
}
@@ -116,10 +121,37 @@ public class JobAutoScalerImpl>
stateStore.removeInfoFromCache(jobKey);
}
-private void clearParallelismOverrides(Context ctx) throws Exception {
-var parallelismOverrides = stateStore.getParallelismOverrides(ctx);
-if (parallelismOverrides.isPresent()) {
+private void clearStatesAfterAutoscalerDisabled(Context ctx) throws
Exception {
+var needFlush = false;
+var parallelismOverridesOpt = stateStore.getParallelismOverrides(ctx);
+if (parallelismOverridesOpt.isPresent()) {
+needFlush = true;
stateStore.removeParallelismOverrides(ctx);
+}
+
+var collectedMetricsOpt = stateStore.getCollectedMetrics(ctx);
+if (collectedMetricsOpt.isPresent()) {
+needFlush = true;
+stateStore.removeCollectedMetrics(ctx);
+}
+
+var scalingHistoryOpt = stateStore.getScalingHistory(ctx);
+if (scalingHistoryOpt.isPresent()) {
+var scalingHistory = scalingHistoryOpt.get();
+var trimmedScalingHistory =
+trimScalingHistory(clock.instant(),
ctx.getConfiguration(), scalingHistory);
+if (trimmedScalingHistory.isEmpty()) {
+// All scaling histories are trimmed.
+needFlush = true;
+stateStore.removeScalingHistory(ctx);
+} else if (!scalingHistory.equals(trimmedScalingHistory)) {
+// Some scaling histories are trimmed.
+needFlush = true;
+stateStore.storeScalingHistory(ctx, trimmedScalingHistory);
+}
+}
+
+if (needFlush) {
stateStore.flush(ctx);
}
}
@@ -211,4 +243,9 @@ public class JobAutoScalerImpl>
ctx.getJobKey(),
id -> new
AutoscalerFlinkMetrics(ctx.getMetricGroup().addGroup("AutoScaler")));
}
+
+@VisibleForTesting
+void setClock(Clock clock) {
+this.clock =
(flink-kubernetes-operator) branch main updated: [FLINK-33526] Autoscaler config improvement + cleanup
This is an automated email from the ASF dual-hosted git repository.
fanrui pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new cbcc6b67 [FLINK-33526] Autoscaler config improvement + cleanup
cbcc6b67 is described below
commit cbcc6b67c98ddfad8bd6141edfd1a6e8c2ff00f5
Author: Gyula Fora
AuthorDate: Sun Nov 12 22:12:53 2023 +0100
[FLINK-33526] Autoscaler config improvement + cleanup
---
.../generated/auto_scaler_configuration.html | 8 +++
.../org/apache/flink/autoscaler/JobAutoScaler.java | 18
.../apache/flink/autoscaler/JobAutoScalerImpl.java | 1 +
.../flink/autoscaler/config/AutoScalerOptions.java | 10 -
.../MetricsCollectionAndEvaluationTest.java| 2 +-
.../flink/kubernetes/operator/FlinkOperator.java | 3 +--
.../AbstractFlinkResourceReconciler.java | 25 +++---
.../reconciler/deployment/ReconcilerFactory.java | 4
.../TestingFlinkDeploymentController.java | 1 -
9 files changed, 34 insertions(+), 38 deletions(-)
diff --git a/docs/layouts/shortcodes/generated/auto_scaler_configuration.html
b/docs/layouts/shortcodes/generated/auto_scaler_configuration.html
index 0b5f33a8..c3a1d798 100644
--- a/docs/layouts/shortcodes/generated/auto_scaler_configuration.html
+++ b/docs/layouts/shortcodes/generated/auto_scaler_configuration.html
@@ -16,7 +16,7 @@
job.autoscaler.catch-up.duration
-15 min
+30 min
Duration
The target duration for fully processing any backlog after a
scaling operation. Set to 0 to disable backlog based scaling.
@@ -52,7 +52,7 @@
job.autoscaler.metrics.window
-10 min
+15 min
Duration
Scaling metrics aggregation window size.
@@ -76,7 +76,7 @@
job.autoscaler.restart.time
-3 min
+5 min
Duration
Expected restart time to be used until the operator can
determine it reliably from history.
@@ -136,7 +136,7 @@
job.autoscaler.target.utilization.boundary
-0.4
+0.3
Double
Target vertex utilization boundary. Scaling won't be performed
if the current processing rate is within [target_rate / (target_utilization -
boundary), (target_rate / (target_utilization + boundary)]
diff --git
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScaler.java
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScaler.java
index ff2b7331..f31fac10 100644
---
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScaler.java
+++
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScaler.java
@@ -20,16 +20,26 @@ package org.apache.flink.autoscaler;
import org.apache.flink.annotation.Internal;
/**
- * The generic Autoscaler.
+ * Flink Job AutoScaler.
*
* @param The job key.
+ * @param Instance of {@link JobAutoScalerContext}.
*/
@Internal
public interface JobAutoScaler>
{
-/** Called as part of the reconciliation loop. */
+/**
+ * Compute and apply new parallelism overrides for the provided job
context.
+ *
+ * @param context Job context.
+ * @throws Exception
+ */
void scale(Context context) throws Exception;
-/** Called when the job is deleted. */
-void cleanup(KEY key);
+/**
+ * Called when the job is deleted.
+ *
+ * @param jobKey Job key.
+ */
+void cleanup(KEY jobKey);
}
diff --git
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
index ca640371..69c646f5 100644
---
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
+++
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScalerImpl.java
@@ -91,6 +91,7 @@ public class JobAutoScalerImpl>
}
if (ctx.getJobStatus() != JobStatus.RUNNING) {
+LOG.debug("Autoscaler is waiting for stable, running state");
lastEvaluatedMetrics.remove(ctx.getJobKey());
return;
}
diff --git
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java
index e8872a35..4ac06fb1 100644
---
a/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java
+++
b/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java
@@ -60,7 +60,7 @@ public class AutoScalerOptions {
public static final ConfigOption
(flink-shaded) branch master updated: [FLINK-33340] Upgrade jackson to 2.15.3
This is an automated email from the ASF dual-hosted git repository.
snuyanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-shaded.git
The following commit(s) were added to refs/heads/master by this push:
new 9fef53a [FLINK-33340] Upgrade jackson to 2.15.3
9fef53a is described below
commit 9fef53a5532d394bc853ba92088d23c98005da98
Author: Sergey Nuyanzin
AuthorDate: Mon Nov 13 02:05:17 2023 +0100
[FLINK-33340] Upgrade jackson to 2.15.3
---
.../flink-shaded-jackson-2/pom.xml | 5 -
.../src/main/resources/META-INF/NOTICE | 16
.../flink-shaded-jackson-module-jsonSchema-2/pom.xml | 5 -
.../src/main/resources/META-INF/NOTICE | 8
flink-shaded-jackson-parent/pom.xml | 2 +-
flink-shaded-swagger/pom.xml | 2 +-
flink-shaded-swagger/src/main/resources/META-INF/NOTICE | 6 +++---
pom.xml | 2 +-
8 files changed, 26 insertions(+), 20 deletions(-)
diff --git a/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml
b/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml
index a4c35b5..cd90649 100644
--- a/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml
+++ b/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml
@@ -25,7 +25,7 @@ under the License.
org.apache.flink
flink-shaded-jackson-parent
-2.14.2-17.0
+2.15.3-17.0
flink-shaded-jackson${flink.ci.license.suffix}
@@ -78,6 +78,9 @@ under the License.
org.apache.maven.plugins
maven-shade-plugin
+
+3.4.1
shade-flink
diff --git
a/flink-shaded-jackson-parent/flink-shaded-jackson-2/src/main/resources/META-INF/NOTICE
b/flink-shaded-jackson-parent/flink-shaded-jackson-2/src/main/resources/META-INF/NOTICE
index 4e81f4e..aaade5d 100644
---
a/flink-shaded-jackson-parent/flink-shaded-jackson-2/src/main/resources/META-INF/NOTICE
+++
b/flink-shaded-jackson-parent/flink-shaded-jackson-2/src/main/resources/META-INF/NOTICE
@@ -6,11 +6,11 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software
License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
-- com.fasterxml.jackson.core:jackson-annotations:2.14.2
-- com.fasterxml.jackson.core:jackson-core:2.14.2
-- com.fasterxml.jackson.core:jackson-databind:2.14.2
-- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.14.2
-- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.14.2
-- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.14.2
-- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.2
-- org.yaml:snakeyaml:1.33
+- com.fasterxml.jackson.core:jackson-annotations:2.15.3
+- com.fasterxml.jackson.core:jackson-core:2.15.3
+- com.fasterxml.jackson.core:jackson-databind:2.15.3
+- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.15.3
+- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.15.3
+- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.15.3
+- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.15.3
+- org.yaml:snakeyaml:2.1
diff --git
a/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/pom.xml
b/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/pom.xml
index ff88a40..dca2751 100644
---
a/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/pom.xml
+++
b/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/pom.xml
@@ -25,7 +25,7 @@ under the License.
org.apache.flink
flink-shaded-jackson-parent
-2.14.2-17.0
+2.15.3-17.0
flink-shaded-jackson-module-jsonSchema${flink.ci.license.suffix}
@@ -64,6 +64,9 @@ under the License.
org.apache.maven.plugins
maven-shade-plugin
+
+3.4.1
shade-flink
diff --git
a/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/src/main/resources/META-INF/NOTICE
b/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/src/main/resources/META-INF/NOTICE
index 8e27db5..9b06129 100644
---
a/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/src/main/resources/META-INF/NOTICE
+++
b/flink-shaded-jackson-parent/flink-shaded-jackson-module-jsonSchema-2/src/main/resources/META-INF/NOTICE
@@ -6,8 +6,8 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software
License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
-- com.fasterxml.jackson.core:jackson-annotations:2.14.2
--
(flink) branch master updated: [FLINK-33213][table] Flink SQL calculate SqlMonotonicity for Calc
This is an automated email from the ASF dual-hosted git repository.
snuyanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 29e49482a82 [FLINK-33213][table] Flink SQL calculate SqlMonotonicity
for Calc
29e49482a82 is described below
commit 29e49482a82e8c1cd404e42c3aae0944188d956e
Author: ParyshevSergey <67409218+paryshevser...@users.noreply.github.com>
AuthorDate: Mon Nov 13 00:07:57 2023 +0700
[FLINK-33213][table] Flink SQL calculate SqlMonotonicity for Calc
Co-authored-by: Sergey
---
.../metadata/FlinkRelMdModifiedMonotonicity.scala | 76 ++-
.../FlinkRelMdModifiedMonotonicityTest.scala | 102 -
.../runtime/stream/sql/AggregateITCase.scala | 49 ++
.../runtime/stream/table/AggregateITCase.scala | 57
4 files changed, 281 insertions(+), 3 deletions(-)
diff --git
a/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdModifiedMonotonicity.scala
b/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdModifiedMonotonicity.scala
index 6c21d3a32c6..f25e2cebdd2 100644
---
a/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdModifiedMonotonicity.scala
+++
b/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdModifiedMonotonicity.scala
@@ -39,6 +39,7 @@ import org.apache.calcite.rel.metadata._
import org.apache.calcite.rex.{RexCall, RexCallBinding, RexInputRef, RexNode}
import org.apache.calcite.sql.{SqlKind, SqlOperatorBinding}
import org.apache.calcite.sql.fun.{SqlCountAggFunction, SqlMinMaxAggFunction,
SqlSumAggFunction, SqlSumEmptyIsZeroAggFunction}
+import org.apache.calcite.sql.fun.SqlStdOperatorTable.{AND, EQUALS,
GREATER_THAN, GREATER_THAN_OR_EQUAL, IN, IS_NOT_NULL, IS_NOT_TRUE, IS_NULL,
IS_TRUE, LESS_THAN, LESS_THAN_OR_EQUAL, NOT, NOT_EQUALS, NOT_IN, OR, SEARCH}
import org.apache.calcite.sql.validate.SqlMonotonicity
import org.apache.calcite.sql.validate.SqlMonotonicity._
import org.apache.calcite.util.Util
@@ -47,6 +48,7 @@ import java.math.{BigDecimal => JBigDecimal}
import java.sql.{Date, Time, Timestamp}
import java.util.Collections
+import scala.annotation.tailrec
import scala.collection.JavaConversions._
/**
@@ -88,8 +90,32 @@ class FlinkRelMdModifiedMonotonicity private extends
MetadataHandler[ModifiedMon
}
def getRelModifiedMonotonicity(rel: Calc, mq: RelMetadataQuery):
RelModifiedMonotonicity = {
-val projects =
rel.getProgram.getProjectList.map(rel.getProgram.expandLocalRef)
-getProjectMonotonicity(projects, rel.getInput, mq)
+val program = rel.getProgram
+val projects = program.getProjectList.map(rel.getProgram.expandLocalRef)
+val result = getProjectMonotonicity(projects, rel.getInput, mq)
+
+// check that `where` section exist
+if (program.getCondition != null && result != null) {
+ val inputMonotonicity = FlinkRelMetadataQuery
+.reuseOrCreate(mq)
+.getRelModifiedMonotonicity(rel.getInput)
+ val inputProjects = program.getExprList.filter(expr =>
expr.isInstanceOf[RexInputRef])
+ assert(inputMonotonicity.fieldMonotonicities.length ==
inputProjects.size)
+ val notConstantProjects = inputProjects.indices
+.map(
+ index =>
+(
+ inputProjects(index).asInstanceOf[RexInputRef],
+ inputMonotonicity.fieldMonotonicities(index)))
+.filter { case (_, monotonicity) => monotonicity != CONSTANT }
+.toArray
+ val condition = program.expandLocalRef(program.getCondition)
+ if (isNeedRetract(condition, notConstantProjects)) {
+program.getProjectList.indices
+ .foreach(index => result.fieldMonotonicities(index) = NOT_MONOTONIC)
+ }
+}
+result
}
private def getProjectMonotonicity(
@@ -638,6 +664,52 @@ class FlinkRelMdModifiedMonotonicity private extends
MetadataHandler[ModifiedMon
udf.getMonotonicity(binding)
}
+ private def isNeedRetract(
+ rexNode: RexNode,
+ projects: Array[(RexInputRef, SqlMonotonicity)]): Boolean = {
+rexNode match {
+ case inputRef: RexInputRef =>
+projects.exists { case (projectInput, _) => projectInput == inputRef }
+
+ case rexCall: RexCall =>
+val operands = rexCall.getOperands.map(operand =>
removeAsAndCast(operand))
+rexCall.getOperator match {
+ case AND | OR =>
+val left = isNeedRetract(operands(0), projects)
+val right = isNeedRetract(operands(1), projects)
+left || right
+
+ case GREATER_THAN | GREATER_THAN_OR_EQUAL =>
+projects
+ .find { case (inputRef, _) => operands.contains(inputRef) }
+ .exists {
(flink) branch master updated: [FLINK-33331] Update netty to 4.1.100 and arrow to 13.0.0
This is an automated email from the ASF dual-hosted git repository. snuyanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new c488adfbcdc [FLINK-1] Update netty to 4.1.100 and arrow to 13.0.0 c488adfbcdc is described below commit c488adfbcdca30320b8721c028f9289f1563ae05 Author: Sergey Nuyanzin AuthorDate: Sun Nov 12 15:26:43 2023 +0100 [FLINK-1] Update netty to 4.1.100 and arrow to 13.0.0 --- flink-python/pom.xml| 2 +- flink-python/src/main/resources/META-INF/NOTICE | 12 ++-- pom.xml | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/flink-python/pom.xml b/flink-python/pom.xml index d13c84e3a12..55b89dfb557 100644 --- a/flink-python/pom.xml +++ b/flink-python/pom.xml @@ -34,7 +34,7 @@ under the License. jar - 5.0.0 + 13.0.0 3.2.3
