[jira] [Commented] (FLINK-35192) Kubernetes operator oom
[ https://issues.apache.org/jira/browse/FLINK-35192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848066#comment-17848066 ] Gabor Somogyi commented on FLINK-35192: --- [~stupid_pig] I've read through the whole conversation here. Do I understand correctly that this jira can be resolved? > Kubernetes operator oom > --- > > Key: FLINK-35192 > URL: https://issues.apache.org/jira/browse/FLINK-35192 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.6.1 > Environment: jdk: openjdk11 > operator version: 1.6.1 >Reporter: chenyuzhi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.9.0 > > Attachments: image-2024-04-22-15-47-49-455.png, > image-2024-04-22-15-52-51-600.png, image-2024-04-22-15-58-23-269.png, > image-2024-04-22-15-58-42-850.png, image-2024-04-30-16-47-07-289.png, > image-2024-04-30-17-11-24-974.png, image-2024-04-30-20-38-25-195.png, > image-2024-04-30-20-39-05-109.png, image-2024-04-30-20-39-34-396.png, > image-2024-04-30-20-41-51-660.png, image-2024-04-30-20-43-20-125.png, > screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png > > > The kubernetest operator docker process was killed by kernel cause out of > memory(the time is 2024.04.03: 18:16) > !image-2024-04-22-15-47-49-455.png! > Metrics: > the pod memory (RSS) is increasing slowly in the past 7 days: > !screenshot-1.png! > However the jvm memory metrics of operator not shown obvious anomaly： > !image-2024-04-22-15-58-23-269.png! > !image-2024-04-22-15-58-42-850.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35192) Kubernetes operator oom
[ https://issues.apache.org/jira/browse/FLINK-35192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848060#comment-17848060 ] Gabor Somogyi commented on FLINK-35192: --- 8b789ee on main > Kubernetes operator oom > --- > > Key: FLINK-35192 > URL: https://issues.apache.org/jira/browse/FLINK-35192 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.6.1 > Environment: jdk: openjdk11 > operator version: 1.6.1 >Reporter: chenyuzhi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.9.0 > > Attachments: image-2024-04-22-15-47-49-455.png, > image-2024-04-22-15-52-51-600.png, image-2024-04-22-15-58-23-269.png, > image-2024-04-22-15-58-42-850.png, image-2024-04-30-16-47-07-289.png, > image-2024-04-30-17-11-24-974.png, image-2024-04-30-20-38-25-195.png, > image-2024-04-30-20-39-05-109.png, image-2024-04-30-20-39-34-396.png, > image-2024-04-30-20-41-51-660.png, image-2024-04-30-20-43-20-125.png, > screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png > > > The kubernetest operator docker process was killed by kernel cause out of > memory(the time is 2024.04.03: 18:16) > !image-2024-04-22-15-47-49-455.png! > Metrics: > the pod memory (RSS) is increasing slowly in the past 7 days: > !screenshot-1.png! > However the jvm memory metrics of operator not shown obvious anomaly： > !image-2024-04-22-15-58-23-269.png! > !image-2024-04-22-15-58-42-850.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-35371:
-
Assignee: Ammar Master (was: Gabor Somogyi)
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Ammar Master
>Assignee: Ammar Master
>Priority: Minor
> Labels: SSL
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi updated FLINK-35371:
--
Affects Version/s: 1.19.0
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Ammar Master
>Assignee: Gabor Somogyi
>Priority: Minor
> Labels: SSL
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-35371:
-
Assignee: Gabor Somogyi
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Ammar Master
>Assignee: Gabor Somogyi
>Priority: Minor
> Labels: SSL
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846935#comment-17846935
]
Gabor Somogyi commented on FLINK-35371:
---
That makes sense. Started to have a look...
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Ammar Master
>Priority: Minor
> Labels: SSL
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-35302) Flink REST server throws exception on unknown fields in RequestBody
[ https://issues.apache.org/jira/browse/FLINK-35302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-35302. --- Resolution: Fixed 36b1d2a on master > Flink REST server throws exception on unknown fields in RequestBody > --- > > Key: FLINK-35302 > URL: https://issues.apache.org/jira/browse/FLINK-35302 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.1 > > > As > [FLIP-401|https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance] > and FLINK-33268 mentioned, when an old version REST client receives response > from a new version REST server, with strict JSON mapper, the client will > throw exceptions on newly added fields, which is not convenient for > situations where a centralized client deals with REST servers of different > versions (e.g. k8s operator). > But this incompatibility can also happens at server side, when a new version > REST client sends requests to an old version REST server with additional > fields. Making server flexible with unknown fields can save clients from > backward compatibility code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35302) Flink REST server throws exception on unknown fields in RequestBody
[ https://issues.apache.org/jira/browse/FLINK-35302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-35302. - > Flink REST server throws exception on unknown fields in RequestBody > --- > > Key: FLINK-35302 > URL: https://issues.apache.org/jira/browse/FLINK-35302 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.1 > > > As > [FLIP-401|https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance] > and FLINK-33268 mentioned, when an old version REST client receives response > from a new version REST server, with strict JSON mapper, the client will > throw exceptions on newly added fields, which is not convenient for > situations where a centralized client deals with REST servers of different > versions (e.g. k8s operator). > But this incompatibility can also happens at server side, when a new version > REST client sends requests to an old version REST server with additional > fields. Making server flexible with unknown fields can save clients from > backward compatibility code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843657#comment-17843657 ] Gabor Somogyi commented on FLINK-33268: --- > I assume that there can still be a problem, when a newer version client sends > requests to an old version server with additional fields in RequestBody that > server does not recognize. That's correct. This change is not solving the complete set of combinations just making the client side more flexible. The main use-case what we wanted to fix is the client usage in the operator code. In short the operator uses a client and when received a new feature like slot sharing group information which was added lately then it was blowing up. To overcome this we needed to copy some things from Flink code which is ugly and in mid-long term must be removed, for example: https://github.com/apache/flink-kubernetes-operator/blob/e73363f3486ed9e1df5cc05c9d0baec7c8c3a37f/flink-autoscaler/src/main/java/org/apache/flink/runtime/rest/messages/job/JobDetailsInfo.java#L295 > Flink REST API response parsing throws exception on new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34574) Add CPU and memory size autoscaler quota
[ https://issues.apache.org/jira/browse/FLINK-34574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-34574: - Assignee: Gabor Somogyi > Add CPU and memory size autoscaler quota > > > Key: FLINK-34574 > URL: https://issues.apache.org/jira/browse/FLINK-34574 > Project: Flink > Issue Type: New Feature > Components: Autoscaler >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34574) Add CPU and memory size autoscaler quota
Gabor Somogyi created FLINK-34574: - Summary: Add CPU and memory size autoscaler quota Key: FLINK-34574 URL: https://issues.apache.org/jira/browse/FLINK-34574 Project: Flink Issue Type: New Feature Components: Autoscaler Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi closed FLINK-20090.
-
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Gabor Somogyi
>Priority: Not a Priority
> Labels: pull-request-available
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi updated FLINK-20090:
--
Fix Version/s: 1.20.0
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Gabor Somogyi
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-20090.
---
Resolution: Fixed
[{{34a7734}}|https://github.com/apache/flink/commit/34a7734c489b080d34ff2194a29d3c1d25d3ab45]
on master
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Gabor Somogyi
>Priority: Not a Priority
> Labels: pull-request-available
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820034#comment-17820034
]
Gabor Somogyi commented on FLINK-20090:
---
I'm working on this and planning to open a PR beginning of next week.
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Gabor Somogyi
>Priority: Not a Priority
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-20090:
-
Assignee: Gabor Somogyi
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Assignee: Gabor Somogyi
>Priority: Not a Priority
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-34267) Python connector test fails when running on MacBook with m1 processor
[
https://issues.apache.org/jira/browse/FLINK-34267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi closed FLINK-34267.
-
> Python connector test fails when running on MacBook with m1 processor
> -
>
> Key: FLINK-34267
> URL: https://issues.apache.org/jira/browse/FLINK-34267
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Build System / CI, Connectors / Common
> Environment: m1 MacBook Pro
> MacOS 14.2.1
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Attempt to execute lint_python.sh on m1 macbook fails while trying to install
> miniconda environment
> {code}
> =installing environment=
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> download miniconda... [SUCCESS]
> installing conda...
> tail: illegal offset -- +018838: Invalid argument
> tail: illegal offset -- +018838: Invalid argument
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/download/miniconda.sh:
> line 353:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/preconda.tar.bz2:
> No such file or directory
> upgrade pip...
> ./dev/lint-python.sh: line 215:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/python:
> No such file or directory
> upgrade pip... [SUCCESS]
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python3.7...
> ./dev/lint-python.sh: line 247:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 1/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 2/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 3/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 failed after retrying 3 times.You can retry to
> execute the script again.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34267) Python connector test fails when running on MacBook with m1 processor
[
https://issues.apache.org/jira/browse/FLINK-34267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-34267:
-
Assignee: Aleksandr Pilipenko
> Python connector test fails when running on MacBook with m1 processor
> -
>
> Key: FLINK-34267
> URL: https://issues.apache.org/jira/browse/FLINK-34267
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Build System / CI, Connectors / Common
> Environment: m1 MacBook Pro
> MacOS 14.2.1
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Attempt to execute lint_python.sh on m1 macbook fails while trying to install
> miniconda environment
> {code}
> =installing environment=
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> download miniconda... [SUCCESS]
> installing conda...
> tail: illegal offset -- +018838: Invalid argument
> tail: illegal offset -- +018838: Invalid argument
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/download/miniconda.sh:
> line 353:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/preconda.tar.bz2:
> No such file or directory
> upgrade pip...
> ./dev/lint-python.sh: line 215:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/python:
> No such file or directory
> upgrade pip... [SUCCESS]
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python3.7...
> ./dev/lint-python.sh: line 247:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 1/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 2/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 3/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 failed after retrying 3 times.You can retry to
> execute the script again.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34267) Python connector test fails when running on MacBook with m1 processor
[
https://issues.apache.org/jira/browse/FLINK-34267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-34267.
---
Resolution: Fixed
[{{e6e1426}}|https://github.com/apache/flink-connector-shared-utils/commit/e6e14268b8316352031b25f4b67ed64dc142b683]
on ci_utils
> Python connector test fails when running on MacBook with m1 processor
> -
>
> Key: FLINK-34267
> URL: https://issues.apache.org/jira/browse/FLINK-34267
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Build System / CI, Connectors / Common
> Environment: m1 MacBook Pro
> MacOS 14.2.1
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Attempt to execute lint_python.sh on m1 macbook fails while trying to install
> miniconda environment
> {code}
> =installing environment=
> installing wget...
> install wget... [SUCCESS]
> installing miniconda...
> download miniconda...
> download miniconda... [SUCCESS]
> installing conda...
> tail: illegal offset -- +018838: Invalid argument
> tail: illegal offset -- +018838: Invalid argument
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/download/miniconda.sh:
> line 353:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/preconda.tar.bz2:
> No such file or directory
> upgrade pip...
> ./dev/lint-python.sh: line 215:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/python:
> No such file or directory
> upgrade pip... [SUCCESS]
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python3.7...
> ./dev/lint-python.sh: line 247:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 1/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 2/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 retrying 3/3
> ./dev/lint-python.sh: line 254:
> /Users/apilipenko/Dev/flink-connector-aws/flink-python/dev/.conda/bin/conda:
> No such file or directory
> conda install 3.7 failed after retrying 3 times.You can retry to
> execute the script again.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-34198) Remove e2e test operator log error check
[ https://issues.apache.org/jira/browse/FLINK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-34198. - > Remove e2e test operator log error check > > > Key: FLINK-34198 > URL: https://issues.apache.org/jira/browse/FLINK-34198 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.8.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > There are too many false positives because of negative test cases and its not > realistic to be so strict that error typed messages can't appear in the > operator log. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34198) Remove e2e test operator log error check
[ https://issues.apache.org/jira/browse/FLINK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-34198: -- Affects Version/s: 1.8.0 (was: 1.8.4) > Remove e2e test operator log error check > > > Key: FLINK-34198 > URL: https://issues.apache.org/jira/browse/FLINK-34198 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.8.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > There are too many false positives because of negative test cases and its not > realistic to be so strict that error typed messages can't appear in the > operator log. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34198) Remove e2e test operator log error check
[
https://issues.apache.org/jira/browse/FLINK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-34198.
---
Fix Version/s: 1.8.0
Resolution: Fixed
[{{31d01f2}}|https://github.com/apache/flink-kubernetes-operator/commit/31d01f246d8a344b560aab1653b7aba561baea26]
on main
> Remove e2e test operator log error check
>
>
> Key: FLINK-34198
> URL: https://issues.apache.org/jira/browse/FLINK-34198
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.4
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> There are too many false positives because of negative test cases and its not
> realistic to be so strict that error typed messages can't appear in the
> operator log.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34198) Remove e2e test operator log error check
[ https://issues.apache.org/jira/browse/FLINK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-34198: - Assignee: Gabor Somogyi > Remove e2e test operator log error check > > > Key: FLINK-34198 > URL: https://issues.apache.org/jira/browse/FLINK-34198 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.8.4 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > > There are too many false positives because of negative test cases and its not > realistic to be so strict that error typed messages can't appear in the > operator log. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34198) Remove e2e test operator log error check
Gabor Somogyi created FLINK-34198: - Summary: Remove e2e test operator log error check Key: FLINK-34198 URL: https://issues.apache.org/jira/browse/FLINK-34198 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Affects Versions: 1.8.4 Reporter: Gabor Somogyi There are too many false positives because of negative test cases and its not realistic to be so strict that error typed messages can't appear in the operator log. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30117) Evaluate operator error log whitelist entries added in FLINK-29475
[ https://issues.apache.org/jira/browse/FLINK-30117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30117. --- Resolution: Won't Fix See comment in FLINK-30310. > Evaluate operator error log whitelist entries added in FLINK-29475 > -- > > Key: FLINK-30117 > URL: https://issues.apache.org/jira/browse/FLINK-30117 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.17.0 >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30311) CI error: Back-off pulling image "flink:1.14"
[
https://issues.apache.org/jira/browse/FLINK-30311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi closed FLINK-30311.
-
> CI error: Back-off pulling image "flink:1.14"
> -
>
> Key: FLINK-30311
> URL: https://issues.apache.org/jira/browse/FLINK-30311
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
>Reporter: Peter Vary
>Priority: Major
>
> CI failed with: {{Flink Deployment failed 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"}}
> We should find the root cause of this issue and try to mitigate it.
> [https://github.com/apache/flink-kubernetes-operator/actions/runs/3627824632/jobs/6118131271]
>
> {code:java}
> 2022-12-06T08:45:03.0243558Z [m[33m2022-12-06 08:41:44,716[m
> [36mo.a.f.k.o.c.FlinkDeploymentController[m
> [1;31m[ERROR][default/flink-example-statemachine] Flink Deployment failed
> 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"
> 2022-12-06T08:45:03.0245385Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerBackoff(AbstractFlinkDeploymentObserver.java:194)
> 2022-12-06T08:45:03.0246604Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:150)
> 2022-12-06T08:45:03.0247780Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:84)
> 2022-12-06T08:45:03.0248934Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:55)
> 2022-12-06T08:45:03.0249941Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:56)
> 2022-12-06T08:45:03.0250844Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:32)
> 2022-12-06T08:45:03.0252038Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:113)
> 2022-12-06T08:45:03.0252936Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
> 2022-12-06T08:45:03.0253850Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
> 2022-12-06T08:45:03.0254412Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
> 2022-12-06T08:45:03.0255322Z at
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
> 2022-12-06T08:45:03.0256081Z at
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
> 2022-12-06T08:45:03.0256872Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
> 2022-12-06T08:45:03.0257804Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
> 2022-12-06T08:45:03.0258720Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
> 2022-12-06T08:45:03.0259635Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
> 2022-12-06T08:45:03.0260448Z at
> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
> 2022-12-06T08:45:03.0261070Z at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> 2022-12-06T08:45:03.0261595Z at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 2022-12-06T08:45:03.0262005Z at java.base/java.lang.Thread.run(Unknown
> Source) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-30283) Evaluate operator error log entry: Error while patching status
[ https://issues.apache.org/jira/browse/FLINK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30283. --- Resolution: Won't Fix See comment in FLINK-30310. > Evaluate operator error log entry: Error while patching status > -- > > Key: FLINK-30283 > URL: https://issues.apache.org/jira/browse/FLINK-30283 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30283) Evaluate operator error log entry: Error while patching status
[ https://issues.apache.org/jira/browse/FLINK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30283. - > Evaluate operator error log entry: Error while patching status > -- > > Key: FLINK-30283 > URL: https://issues.apache.org/jira/browse/FLINK-30283 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30311) CI error: Back-off pulling image "flink:1.14"
[
https://issues.apache.org/jira/browse/FLINK-30311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-30311.
---
Resolution: Won't Fix
See comment in FLINK-30310.
> CI error: Back-off pulling image "flink:1.14"
> -
>
> Key: FLINK-30311
> URL: https://issues.apache.org/jira/browse/FLINK-30311
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
>Reporter: Peter Vary
>Priority: Major
>
> CI failed with: {{Flink Deployment failed 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"}}
> We should find the root cause of this issue and try to mitigate it.
> [https://github.com/apache/flink-kubernetes-operator/actions/runs/3627824632/jobs/6118131271]
>
> {code:java}
> 2022-12-06T08:45:03.0243558Z [m[33m2022-12-06 08:41:44,716[m
> [36mo.a.f.k.o.c.FlinkDeploymentController[m
> [1;31m[ERROR][default/flink-example-statemachine] Flink Deployment failed
> 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"
> 2022-12-06T08:45:03.0245385Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerBackoff(AbstractFlinkDeploymentObserver.java:194)
> 2022-12-06T08:45:03.0246604Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:150)
> 2022-12-06T08:45:03.0247780Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:84)
> 2022-12-06T08:45:03.0248934Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:55)
> 2022-12-06T08:45:03.0249941Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:56)
> 2022-12-06T08:45:03.0250844Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:32)
> 2022-12-06T08:45:03.0252038Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:113)
> 2022-12-06T08:45:03.0252936Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
> 2022-12-06T08:45:03.0253850Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
> 2022-12-06T08:45:03.0254412Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
> 2022-12-06T08:45:03.0255322Z at
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
> 2022-12-06T08:45:03.0256081Z at
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
> 2022-12-06T08:45:03.0256872Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
> 2022-12-06T08:45:03.0257804Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
> 2022-12-06T08:45:03.0258720Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
> 2022-12-06T08:45:03.0259635Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
> 2022-12-06T08:45:03.0260448Z at
> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
> 2022-12-06T08:45:03.0261070Z at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> 2022-12-06T08:45:03.0261595Z at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 2022-12-06T08:45:03.0262005Z at java.base/java.lang.Thread.run(Unknown
> Source) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30117) Evaluate operator error log whitelist entries added in FLINK-29475
[ https://issues.apache.org/jira/browse/FLINK-30117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30117. - > Evaluate operator error log whitelist entries added in FLINK-29475 > -- > > Key: FLINK-30117 > URL: https://issues.apache.org/jira/browse/FLINK-30117 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.17.0 >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30149) Evaluate operator error log whitelist entry: Error during event processing
[ https://issues.apache.org/jira/browse/FLINK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30149. --- Resolution: Won't Fix See comment in FLINK-30310. > Evaluate operator error log whitelist entry: Error during event processing > -- > > Key: FLINK-30149 > URL: https://issues.apache.org/jira/browse/FLINK-30149 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30149) Evaluate operator error log whitelist entry: Error during event processing
[ https://issues.apache.org/jira/browse/FLINK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30149. - > Evaluate operator error log whitelist entry: Error during event processing > -- > > Key: FLINK-30149 > URL: https://issues.apache.org/jira/browse/FLINK-30149 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30147) Evaluate operator error log whitelist entry: Failed to submit a listener notification task
[ https://issues.apache.org/jira/browse/FLINK-30147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30147. - > Evaluate operator error log whitelist entry: Failed to submit a listener > notification task > -- > > Key: FLINK-30147 > URL: https://issues.apache.org/jira/browse/FLINK-30147 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30148) Evaluate operator error log whitelist entry: Failed to submit job to session cluster
[ https://issues.apache.org/jira/browse/FLINK-30148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30148. --- Resolution: Won't Fix See comment in FLINK-30310. > Evaluate operator error log whitelist entry: Failed to submit job to session > cluster > > > Key: FLINK-30148 > URL: https://issues.apache.org/jira/browse/FLINK-30148 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30148) Evaluate operator error log whitelist entry: Failed to submit job to session cluster
[ https://issues.apache.org/jira/browse/FLINK-30148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30148. - > Evaluate operator error log whitelist entry: Failed to submit job to session > cluster > > > Key: FLINK-30148 > URL: https://issues.apache.org/jira/browse/FLINK-30148 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30147) Evaluate operator error log whitelist entry: Failed to submit a listener notification task
[ https://issues.apache.org/jira/browse/FLINK-30147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30147. --- Resolution: Won't Fix See comment in FLINK-30310. > Evaluate operator error log whitelist entry: Failed to submit a listener > notification task > -- > > Key: FLINK-30147 > URL: https://issues.apache.org/jira/browse/FLINK-30147 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30310) Re-enable e2e test error check
[ https://issues.apache.org/jira/browse/FLINK-30310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-30310. - > Re-enable e2e test error check > -- > > Key: FLINK-30310 > URL: https://issues.apache.org/jira/browse/FLINK-30310 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > > In FLINK-30307 e2e test error check has been turned off temporarily. We must > re-enable it after release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-30310) Re-enable e2e test error check
[ https://issues.apache.org/jira/browse/FLINK-30310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-30310. --- Resolution: Won't Fix > Re-enable e2e test error check > -- > > Key: FLINK-30310 > URL: https://issues.apache.org/jira/browse/FLINK-30310 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > > In FLINK-30307 e2e test error check has been turned off temporarily. We must > re-enable it after release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30310) Re-enable e2e test error check
[ https://issues.apache.org/jira/browse/FLINK-30310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809448#comment-17809448 ] Gabor Somogyi commented on FLINK-30310: --- There are too many false positives because of negative test cases and its not realistic to be so strict that error typed messages can't appear in the operator log so closing this with won't do and remove this dead code part. > Re-enable e2e test error check > -- > > Key: FLINK-30310 > URL: https://issues.apache.org/jira/browse/FLINK-30310 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > > In FLINK-30307 e2e test error check has been turned off temporarily. We must > re-enable it after release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33268: -- Fix Version/s: 1.19.0 > Flink REST API response parsing throws exception on new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33268. - > Flink REST API response parsing throws exception on new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[
https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-33268.
---
Resolution: Fixed
[{{19cb9de}}|https://github.com/apache/flink/commit/19cb9de5c54b9535be15ca850f5e1ebd2e21c244]
on master
> Flink REST API response parsing throws exception on new fields
> --
>
> Key: FLINK-33268
> URL: https://issues.apache.org/jira/browse/FLINK-33268
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.19.0
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
>
> At the moment Flink is not ignoring unknown fields when parsing REST
> responses. An example for such a class is JobDetailsInfo but this applies to
> all others. It would be good to add this support to increase compatibility.
> The real life use-case is when the Flink k8s operator wants to handle 2 jobs
> with 2 different Flink versions where the newer version has added a new field
> to any REST response. Such case the operator gets an exception when for
> example it tries to poll the job details with the additional field.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33268: - Assignee: Gabor Somogyi > Flink REST API response parsing throws exception on new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33559) Externalize Kafka Python connector code
[ https://issues.apache.org/jira/browse/FLINK-33559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33559. --- Resolution: Fixed > Externalize Kafka Python connector code > --- > > Key: FLINK-33559 > URL: https://issues.apache.org/jira/browse/FLINK-33559 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Kafka >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > See description of parent ticket for context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33559) Externalize Kafka Python connector code
[ https://issues.apache.org/jira/browse/FLINK-33559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795239#comment-17795239 ] Gabor Somogyi commented on FLINK-33559: --- c38a040 on main > Externalize Kafka Python connector code > --- > > Key: FLINK-33559 > URL: https://issues.apache.org/jira/browse/FLINK-33559 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Kafka >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > See description of parent ticket for context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33559) Externalize Kafka Python connector code
[ https://issues.apache.org/jira/browse/FLINK-33559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33559. - > Externalize Kafka Python connector code > --- > > Key: FLINK-33559 > URL: https://issues.apache.org/jira/browse/FLINK-33559 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Kafka >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > See description of parent ticket for context. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33556) Test infrastructure for externalized python code
[ https://issues.apache.org/jira/browse/FLINK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33556. - > Test infrastructure for externalized python code > > > Key: FLINK-33556 > URL: https://issues.apache.org/jira/browse/FLINK-33556 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Common >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > We need to establish the reusable parts of the python infrastructure as part > of the shared connector utils such that it can be easily reused. Ideally we > would create a github workflow similar to > https://github.com/apache/flink-connector-shared-utils/blob/ci_utils/.github/workflows/ci.yml. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33556) Test infrastructure for externalized python code
[ https://issues.apache.org/jira/browse/FLINK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794102#comment-17794102 ] Gabor Somogyi commented on FLINK-33556: --- 7691962 on ci_utils > Test infrastructure for externalized python code > > > Key: FLINK-33556 > URL: https://issues.apache.org/jira/browse/FLINK-33556 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Common >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > We need to establish the reusable parts of the python infrastructure as part > of the shared connector utils such that it can be easily reused. Ideally we > would create a github workflow similar to > https://github.com/apache/flink-connector-shared-utils/blob/ci_utils/.github/workflows/ci.yml. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33556) Test infrastructure for externalized python code
[ https://issues.apache.org/jira/browse/FLINK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33556. --- Resolution: Fixed > Test infrastructure for externalized python code > > > Key: FLINK-33556 > URL: https://issues.apache.org/jira/browse/FLINK-33556 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Common >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > We need to establish the reusable parts of the python infrastructure as part > of the shared connector utils such that it can be easily reused. Ideally we > would create a github workflow similar to > https://github.com/apache/flink-connector-shared-utils/blob/ci_utils/.github/workflows/ci.yml. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33556) Test infrastructure for externalized python code
[ https://issues.apache.org/jira/browse/FLINK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794087#comment-17794087 ] Gabor Somogyi commented on FLINK-33556: --- e4f3898 on master > Test infrastructure for externalized python code > > > Key: FLINK-33556 > URL: https://issues.apache.org/jira/browse/FLINK-33556 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Common >Affects Versions: 1.18.0 >Reporter: Márton Balassi >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > We need to establish the reusable parts of the python infrastructure as part > of the shared connector utils such that it can be easily reused. Ideally we > would create a github workflow similar to > https://github.com/apache/flink-connector-shared-utils/blob/ci_utils/.github/workflows/ci.yml. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33268) Flink REST API response parsing throws exception on new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33268: -- Summary: Flink REST API response parsing throws exception on new fields (was: Flink REST API response parsing should support backward compatible changes like new fields) > Flink REST API response parsing throws exception on new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Priority: Major > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33268) Flink REST API response parsing should support backward compatible changes like new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33268: -- Description: At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. The real life use-case is when the Flink k8s operator wants to handle 2 jobs with 2 different Flink versions where the newer version has added a new field to any REST response. Such case the operator gets an exception when for example it tries to poll the job details with the additional field. was: At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. The real life use-case is when the Flink k8s operator wants to handle 2 jobs with 2 different Flink versions where the newer version has added a new field to any REST response. Such case the operator gets an exception when tries to poll the job details with the additional field. > Flink REST API response parsing should support backward compatible changes > like new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Priority: Major > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when for > example it tries to poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33268) Flink REST API response parsing should support backward compatible changes like new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33268: -- Description: At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. The real life use-case is when the Flink k8s operator wants to handle 2 jobs with 2 different Flink versions where the newer version has added a new field to any REST response. Such case the operator gets an exception when tries to poll the job details with the additional field. was: At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. The real life use-case is when the operator wants to handle 2 jobs with 2 different Flink versions where the newer version has added a new field to any REST response. Such case the operator gets an exception when tries to poll the job details with the additional field. > Flink REST API response parsing should support backward compatible changes > like new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Priority: Major > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the Flink k8s operator wants to handle 2 jobs > with 2 different Flink versions where the newer version has added a new field > to any REST response. Such case the operator gets an exception when tries to > poll the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33268) Flink REST API response parsing should support backward compatible changes like new fields
[ https://issues.apache.org/jira/browse/FLINK-33268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33268: -- Description: At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. The real life use-case is when the operator wants to handle 2 jobs with 2 different Flink versions where the newer version has added a new field to any REST response. Such case the operator gets an exception when tries to poll the job details with the additional field. was:At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. > Flink REST API response parsing should support backward compatible changes > like new fields > -- > > Key: FLINK-33268 > URL: https://issues.apache.org/jira/browse/FLINK-33268 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Priority: Major > > At the moment Flink is not ignoring unknown fields when parsing REST > responses. An example for such a class is JobDetailsInfo but this applies to > all others. It would be good to add this support to increase compatibility. > The real life use-case is when the operator wants to handle 2 jobs with 2 > different Flink versions where the newer version has added a new field to any > REST response. Such case the operator gets an exception when tries to poll > the job details with the additional field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33531) Nightly Python fails with NPE at metadataHandlerProvider on AZP
[
https://issues.apache.org/jira/browse/FLINK-33531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785515#comment-17785515
]
Gabor Somogyi commented on FLINK-33531:
---
Since I've added python 3.11 lately I've double checked my part.
This has been added on 15th of Oct:
{code:java}
commit 2da9a9639216b8c48850ee714065f090a80dcd65
Author: Gabor Somogyi
Date: Sun Oct 15 09:31:08 2023 +0200
[FLINK-33030][python] Add python 3.11 support
Also bump grpcio-tools version
...
{code}
Seems like the latest green nightly happened 30th of Oct so that's not the
cause:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54166=logs=9cada3cb-c1d3-5621-16da-0f718fb86602]
I've gone back time starting from the mentioned 30th of Oct, double checked the
master nightlies and seems like it was stable. No idea what happened but after
that it became unstable.
> Nightly Python fails with NPE at metadataHandlerProvider on AZP
> ---
>
> Key: FLINK-33531
> URL: https://issues.apache.org/jira/browse/FLINK-33531
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Blocker
> Labels: test-stability
>
> It seems starting 02.11.2023 every master nightly fails with this (that's why
> it is a blocker)
> for instance
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54512=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901]
> {noformat}
> 2023-11-12T02:10:24.5082784Z Nov 12 02:10:24 if is_error(answer)[0]:
> 2023-11-12T02:10:24.5083620Z Nov 12 02:10:24 if len(answer) > 1:
> 2023-11-12T02:10:24.5084326Z Nov 12 02:10:24 type = answer[1]
> 2023-11-12T02:10:24.5085164Z Nov 12 02:10:24 value =
> OUTPUT_CONVERTER[type](answer[2:], gateway_client)
> 2023-11-12T02:10:24.5086061Z Nov 12 02:10:24 if answer[1] ==
> REFERENCE_TYPE:
> 2023-11-12T02:10:24.5086850Z Nov 12 02:10:24 > raise
> Py4JJavaError(
> 2023-11-12T02:10:24.5087677Z Nov 12 02:10:24 "An
> error occurred while calling {0}{1}{2}.\n".
> 2023-11-12T02:10:24.5088538Z Nov 12 02:10:24
> format(target_id, ".", name), value)
> 2023-11-12T02:10:24.5089551Z Nov 12 02:10:24 E
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o3371.executeInsert.
> 2023-11-12T02:10:24.5090832Z Nov 12 02:10:24 E :
> java.lang.NullPointerException: metadataHandlerProvider
> 2023-11-12T02:10:24.5091832Z Nov 12 02:10:24 Eat
> java.util.Objects.requireNonNull(Objects.java:228)
> 2023-11-12T02:10:24.5093399Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.getMetadataHandlerProvider(RelMetadataQueryBase.java:122)
> 2023-11-12T02:10:24.5094480Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
> 2023-11-12T02:10:24.5095365Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
> 2023-11-12T02:10:24.5096306Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
> 2023-11-12T02:10:24.5097238Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
> 2023-11-12T02:10:24.5098014Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> 2023-11-12T02:10:24.5098753Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
> 2023-11-12T02:10:24.5099517Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
> 2023-11-12T02:10:24.5100373Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
> 2023-11-12T02:10:24.5101313Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
> 2023-11-12T02:10:24.5102410Z Nov 12 02:10:24 Eat
> org.apache.flink.calcite.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
> 2023-11-12T02:10:24.5103343Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
> 2023-11-12T02:10:24.5104105Z Nov 12 02:10:24 E
[jira] [Closed] (FLINK-33515) PythonDriver need to stream python process output to log instead of collecting it in memory
[ https://issues.apache.org/jira/browse/FLINK-33515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33515. - > PythonDriver need to stream python process output to log instead of > collecting it in memory > --- > > Key: FLINK-33515 > URL: https://issues.apache.org/jira/browse/FLINK-33515 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > PythonDriver now collects the python process output in a Stringbuilder > instead of streaming it. It can cause OOM when the python process is > generating huge amount of output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33515) PythonDriver need to stream python process output to log instead of collecting it in memory
[ https://issues.apache.org/jira/browse/FLINK-33515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33515. --- Fix Version/s: 1.19.0 Resolution: Fixed caa324a on master > PythonDriver need to stream python process output to log instead of > collecting it in memory > --- > > Key: FLINK-33515 > URL: https://issues.apache.org/jira/browse/FLINK-33515 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > PythonDriver now collects the python process output in a Stringbuilder > instead of streaming it. It can cause OOM when the python process is > generating huge amount of output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33513) Metastore delegation-token can be cached?
[ https://issues.apache.org/jira/browse/FLINK-33513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784739#comment-17784739 ] Gabor Somogyi commented on FLINK-33513: --- If that hurts the solution is not caching but adding a token provider for metastore like HiveServer2DelegationTokenProvider. > Metastore delegation-token can be cached? > - > > Key: FLINK-33513 > URL: https://issues.apache.org/jira/browse/FLINK-33513 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: katty he >Priority: Major > > Now, every time, getDelegationToken wil be called when asking for metastore, > how about build a cache, we cache the token for the first time, then we can > just get token from cache? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33515) PythonDriver need to stream python process output to log instead of collecting it in memory
Gabor Somogyi created FLINK-33515: - Summary: PythonDriver need to stream python process output to log instead of collecting it in memory Key: FLINK-33515 URL: https://issues.apache.org/jira/browse/FLINK-33515 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.19.0 Reporter: Gabor Somogyi PythonDriver now collects the python process output in a Stringbuilder instead of streaming it. It can cause OOM when the python process is generating huge amount of output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33515) PythonDriver need to stream python process output to log instead of collecting it in memory
[ https://issues.apache.org/jira/browse/FLINK-33515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33515: - Assignee: Gabor Somogyi > PythonDriver need to stream python process output to log instead of > collecting it in memory > --- > > Key: FLINK-33515 > URL: https://issues.apache.org/jira/browse/FLINK-33515 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > > PythonDriver now collects the python process output in a Stringbuilder > instead of streaming it. It can cause OOM when the python process is > generating huge amount of output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33268) Flink REST API response parsing should support backward compatible changes like new fields
Gabor Somogyi created FLINK-33268: - Summary: Flink REST API response parsing should support backward compatible changes like new fields Key: FLINK-33268 URL: https://issues.apache.org/jira/browse/FLINK-33268 Project: Flink Issue Type: Improvement Components: Runtime / REST Affects Versions: 1.19.0 Reporter: Gabor Somogyi At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all others. It would be good to add this support to increase compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33172) Bump numpy version
[ https://issues.apache.org/jira/browse/FLINK-33172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33172. --- Fix Version/s: 1.19.0 Resolution: Fixed ab26175 on master > Bump numpy version > -- > > Key: FLINK-33172 > URL: https://issues.apache.org/jira/browse/FLINK-33172 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33172) Bump numpy version
[ https://issues.apache.org/jira/browse/FLINK-33172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33172. - > Bump numpy version > -- > > Key: FLINK-33172 > URL: https://issues.apache.org/jira/browse/FLINK-33172 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770818#comment-17770818
]
Gabor Somogyi commented on FLINK-33155:
---
The fact that YARN is not able to renew the token after 7 days (with default
config) is a known for long time. If this is the issue what one wants to solve
then I would say that must be solved on YARN side and not hack around in Flink.
We've had the same issue when we've formed the Spark authentication framework
back in the days and we've had the same conclusion there...
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
> at
>
[jira] [Commented] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770703#comment-17770703 ] Gabor Somogyi commented on FLINK-32223: --- I've just double checked and before this commit HybridShuffleITCase was flaky here: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=8993 After the merge one can see the same issue: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53525=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=8701 The previously problematic hadoop_313 stage now passed. I consider that this looks good. If somebody sees an issue plz ping. > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33172) Bump numpy version
[ https://issues.apache.org/jira/browse/FLINK-33172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33172: - Assignee: Gabor Somogyi > Bump numpy version > -- > > Key: FLINK-33172 > URL: https://issues.apache.org/jira/browse/FLINK-33172 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33172) Bump numpy version
Gabor Somogyi created FLINK-33172: - Summary: Bump numpy version Key: FLINK-33172 URL: https://issues.apache.org/jira/browse/FLINK-33172 Project: Flink Issue Type: Improvement Components: API / Python Affects Versions: 1.19.0 Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770398#comment-17770398 ] Gabor Somogyi commented on FLINK-32223: --- The normal build passed against master: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53521=results Waiting on nightly... > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-32223. - > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-32223. --- Fix Version/s: 1.19.0 Resolution: Fixed e4c15aa on master > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769498#comment-17769498
]
Gabor Somogyi commented on FLINK-33155:
---
I would say green light on this :) Just saying that getting tokens from
different source where the content is different is relatively a big change. W/o
the mentioned change this is not going to work properly...
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:236)
> at
>
[jira] [Comment Edited] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768665#comment-17768665
]
Gabor Somogyi edited comment on FLINK-33155 at 9/25/23 12:20 PM:
-
Not updating UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION is a known
limitation of YARN.
If the mentioned code runs on the JM side and delegation tokens are enabled
then it makes sense since the JM keeps it's tokens up-to-date all the time.
Couple of notes:
* Changing the default behavior from file to UGI can be a breaking change to
users which are depending on that some way
* DT handling is a single threaded operation but as I see TM creation uses
multiple threads which may end-up in undefined behavior
was (Author: gaborgsomogyi):
Not updating UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION is a known
limitation of YARN.
If the mentioned code runs on the JM side and delegation tokens are enabled
then it makes sense since the JM keeps it's tokens up-to-date all the time.
Couple of notes:
* Changing the default behavior from file to UGI can be a breaking change to
users which are depending on that some way...
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at
[jira] [Commented] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768665#comment-17768665
]
Gabor Somogyi commented on FLINK-33155:
---
Not updating UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION is a known
limitation of YARN.
If the mentioned code runs on the JM side and delegation tokens are enabled
then it makes sense since the JM keeps it's tokens up-to-date all the time.
Couple of notes:
* Changing the default behavior from file to UGI can be a breaking change to
users which are depending on that some way...
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
> at
>
[jira] [Closed] (FLINK-32976) NullPointException when starting flink cluster in standalone mode
[
https://issues.apache.org/jira/browse/FLINK-32976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi closed FLINK-32976.
-
> NullPointException when starting flink cluster in standalone mode
> -
>
> Key: FLINK-32976
> URL: https://issues.apache.org/jira/browse/FLINK-32976
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.17.1
>Reporter: Feng Jin
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> It can be reproduced when starting flink cluster with hadoop configuration.
>
> {code:java}
> //代码占位符
> // Set up hadoop conf , hadoop classpath
> // start jobManager
> ./jobmanager.sh start-foreground {code}
>
> The error message as follows:
>
> {code:java}
> //代码占位符
> Caused by: java.ang.NullPointerException
> at org.apache.flink. runtime.
> security.token.hadoop.HadoopFSDelegationTokenProvider.getFileSystemsToAccess(HadoopFSDelegationTokenProvider.java:173)~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvidertionTokens$1(HadoopFSDelegationTokenProvider.java:113)
> ~[flink-dist-1.17.1.jar:1.17.1
> at java.security.AccessController.doprivileged(Native Method)~[?:1.8.0 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~[?:1.8.0 281]
> at org. apache.hadoop .
> security.UserGroupInformation.doAs(UserGroupInformation. java:1876)
> ~flink-shacd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org. apache.flink. runtime.security.token .hadoop
> .HadoopFSDelegationTokenProvider.obtainDelegationTcens(HadoopFSDelegationTokenProvider.java:108)~flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink. runtime. security.token.DefaultDelegationTokenManager .
> lambda$obtainDel
> SAndGetNextRenewal$1(DefaultDelegationTokenManager .java:264)~
> flink-dist-1.17.1.jar:1.17.1]
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> ~?:1.8.0 281
> at
> java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)~[?:1.8.0
> 281]at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> ~?:1.8.0 281at
> java,util.stream.Reduce0ps$Reduce0p.evaluateSequential(Reduce0ps.java:708)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)~[?:1.8.0
> 281]at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:479)
> ~?:1.8.0 281
> at java.util.stream.ReferencePipeline.min(ReferencePipeline.java:520)~?:1.8.0
> 281at org. apache. flink. runtime.
> security.token.DefaultDelegationTokenManager
> .obtainDelegationTokensAndGeNextRenewal(DefaultDelegationTokenManager
> .java:286)~[flink-dist-1.17.1.jar:1.17.1
> at org.apache. flink.runtime. security.token.DefaultDelegationTokenManager.
> obtainDelegationTokens(DefaltDelegationTokenManager.java:242)~[flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializes@)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.entrypoint.clusterEntrypoint.nk-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint:232)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8. 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~?:1.8.0 281]
> at org. apache.hadoop . security.UserGroupInformation. doAs
> (UserGroupInformation.
> java:1876)~[flink-shadd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org.apache.flink.runtime.security. contexts
> .HadoopSecurityContext.runSecured(HadoopSecurijava:41)
> ~[flink-dist-1.17.1.jar:1.17.1
> at org. apache.flink. runtime. entrypoint. ClusterEntrypoint .
> startCluster(clusterEntrypoint. java:229)link-dist-1.17.1.jar:1.17.1]...2
> more{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-32976) NullPointException when starting flink cluster in standalone mode
[
https://issues.apache.org/jira/browse/FLINK-32976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-32976.
---
Fix Version/s: 1.19.0
Resolution: Fixed
1fb95c3 on master
> NullPointException when starting flink cluster in standalone mode
> -
>
> Key: FLINK-32976
> URL: https://issues.apache.org/jira/browse/FLINK-32976
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.17.1
>Reporter: Feng Jin
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> It can be reproduced when starting flink cluster with hadoop configuration.
>
> {code:java}
> //代码占位符
> // Set up hadoop conf , hadoop classpath
> // start jobManager
> ./jobmanager.sh start-foreground {code}
>
> The error message as follows:
>
> {code:java}
> //代码占位符
> Caused by: java.ang.NullPointerException
> at org.apache.flink. runtime.
> security.token.hadoop.HadoopFSDelegationTokenProvider.getFileSystemsToAccess(HadoopFSDelegationTokenProvider.java:173)~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvidertionTokens$1(HadoopFSDelegationTokenProvider.java:113)
> ~[flink-dist-1.17.1.jar:1.17.1
> at java.security.AccessController.doprivileged(Native Method)~[?:1.8.0 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~[?:1.8.0 281]
> at org. apache.hadoop .
> security.UserGroupInformation.doAs(UserGroupInformation. java:1876)
> ~flink-shacd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org. apache.flink. runtime.security.token .hadoop
> .HadoopFSDelegationTokenProvider.obtainDelegationTcens(HadoopFSDelegationTokenProvider.java:108)~flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink. runtime. security.token.DefaultDelegationTokenManager .
> lambda$obtainDel
> SAndGetNextRenewal$1(DefaultDelegationTokenManager .java:264)~
> flink-dist-1.17.1.jar:1.17.1]
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> ~?:1.8.0 281
> at
> java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)~[?:1.8.0
> 281]at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> ~?:1.8.0 281at
> java,util.stream.Reduce0ps$Reduce0p.evaluateSequential(Reduce0ps.java:708)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)~[?:1.8.0
> 281]at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:479)
> ~?:1.8.0 281
> at java.util.stream.ReferencePipeline.min(ReferencePipeline.java:520)~?:1.8.0
> 281at org. apache. flink. runtime.
> security.token.DefaultDelegationTokenManager
> .obtainDelegationTokensAndGeNextRenewal(DefaultDelegationTokenManager
> .java:286)~[flink-dist-1.17.1.jar:1.17.1
> at org.apache. flink.runtime. security.token.DefaultDelegationTokenManager.
> obtainDelegationTokens(DefaltDelegationTokenManager.java:242)~[flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializes@)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.entrypoint.clusterEntrypoint.nk-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint:232)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8. 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~?:1.8.0 281]
> at org. apache.hadoop . security.UserGroupInformation. doAs
> (UserGroupInformation.
> java:1876)~[flink-shadd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org.apache.flink.runtime.security. contexts
> .HadoopSecurityContext.runSecured(HadoopSecurijava:41)
> ~[flink-dist-1.17.1.jar:1.17.1
> at org. apache.flink. runtime. entrypoint. ClusterEntrypoint .
> startCluster(clusterEntrypoint. java:229)link-dist-1.17.1.jar:1.17.1]...2
> more{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33105: -- Fix Version/s: kubernetes-operator-1.7.0 > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33105. --- Resolution: Fixed 3211549 on main > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33105. - > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33108) Error during error status handling
[
https://issues.apache.org/jira/browse/FLINK-33108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766693#comment-17766693
]
Gabor Somogyi commented on FLINK-33108:
---
After quite some investment I still don't see what is actually doing the race.
I would say it would be good to keep this feature at the individual job level
for now.
> Error during error status handling
> --
>
> Key: FLINK-33108
> URL: https://issues.apache.org/jira/browse/FLINK-33108
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gabor Somogyi
>Priority: Major
>
> e2e_ci (v1_13, flink, native, test_multi_sessionjob.sh) failed with the
> following issue:
> {code:java}
> Error: m2023-09-18 08:26:41,813 i.j.o.p.e.ReconciliationDispatcher
> [ERROR][flink/session-cluster-1] Error during error status handling.
> org.apache.flink.kubernetes.operator.exception.StatusConflictException:
> Status have been modified externally in version 1374 Previous:
> {"jobStatus":{"jobName":null,"jobId":null,"state":null,"startTime":null,"updateTime":null,"savepointInfo":{"lastSavepoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0},"checkpointInfo":{"lastCheckpoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"lastPeriodicCheckpointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"total-cpu":"0.25","flink-version":"1.13.6","flink-revision":"b2ca390
> @
>
[jira] [Commented] (FLINK-33108) Error during error status handling
[
https://issues.apache.org/jira/browse/FLINK-33108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766334#comment-17766334
]
Gabor Somogyi commented on FLINK-33108:
---
I'm not telling that this causes the issue but the problematic fields has been
added here: https://github.com/apache/flink-kubernetes-operator/pull/558
> Error during error status handling
> --
>
> Key: FLINK-33108
> URL: https://issues.apache.org/jira/browse/FLINK-33108
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gabor Somogyi
>Priority: Major
>
> e2e_ci (v1_13, flink, native, test_multi_sessionjob.sh) failed with the
> following issue:
> {code:java}
> Error: m2023-09-18 08:26:41,813 i.j.o.p.e.ReconciliationDispatcher
> [ERROR][flink/session-cluster-1] Error during error status handling.
> org.apache.flink.kubernetes.operator.exception.StatusConflictException:
> Status have been modified externally in version 1374 Previous:
> {"jobStatus":{"jobName":null,"jobId":null,"state":null,"startTime":null,"updateTime":null,"savepointInfo":{"lastSavepoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0},"checkpointInfo":{"lastCheckpoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"lastPeriodicCheckpointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"total-cpu":"0.25","flink-version":"1.13.6","flink-revision":"b2ca390
> @
>
[jira] [Commented] (FLINK-33108) Error during error status handling
[
https://issues.apache.org/jira/browse/FLINK-33108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766311#comment-17766311
]
Gabor Somogyi commented on FLINK-33108:
---
The issue comes only occasionally and not consistent so seems like a race.
> Error during error status handling
> --
>
> Key: FLINK-33108
> URL: https://issues.apache.org/jira/browse/FLINK-33108
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gabor Somogyi
>Priority: Major
>
> e2e_ci (v1_13, flink, native, test_multi_sessionjob.sh) failed with the
> following issue:
> {code:java}
> Error: m2023-09-18 08:26:41,813 i.j.o.p.e.ReconciliationDispatcher
> [ERROR][flink/session-cluster-1] Error during error status handling.
> org.apache.flink.kubernetes.operator.exception.StatusConflictException:
> Status have been modified externally in version 1374 Previous:
> {"jobStatus":{"jobName":null,"jobId":null,"state":null,"startTime":null,"updateTime":null,"savepointInfo":{"lastSavepoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0},"checkpointInfo":{"lastCheckpoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"lastPeriodicCheckpointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"total-cpu":"0.25","flink-version":"1.13.6","flink-revision":"b2ca390
> @
> 2022-02-03T14:54:22+01:00","total-memory":"1073741824"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1695025410957,"lastReconciledSpec":"{\"spec\":{\"job\":null,\"restartNonce\":null,\"flinkConfiguration\":{\"high-availability\":\"org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory\",\"high-availability.storageDir\":\"file:///opt/flink/volume/flink-ha\",\"state.checkpoints.dir\":\"file:///opt/flink/volume/flink-cp\",\"state.savepoints.dir\":\"file:///opt/flink/volume/flink-sp\",\"taskmanager.numberOfTaskSlots\":\"2\"},\"image\":\"flink:1.13\",\"imagePullPolicy\":null,\"serviceAccount\":\"flink\",\"flinkVersion\":\"v1_13\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"pod-template\"},\"spec\":{\"containers\":[{\"name\":\"flink-main-container\",\"resources\":{\"limits\":{\"ephemeral-storage\":\"2048Mi\"},\"requests\":{\"ephemeral-storage\":\"2048Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/opt/flink/volume\",\"name\":\"flink-volume\"}]}],\"volumes\":[{\"name\":\"flink-volume\",\"persistentVolumeClaim\":{\"claimName\":\"session-cluster-1-pvc\"}}]}},\"jobManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":\"native\"},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":null,\"restartNonce\":null,\"flinkConfiguration\":{\"high-availability\":\"org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory\",\"high-availability.storageDir\":\"file:///opt/flink/volume/flink-ha\",\"state.checkpoints.dir\":\"file:///opt/flink/volume/flink-cp\",\"state.savepoints.dir\":\"file:///opt/flink/volume/flink-sp\",\"taskmanager.numberOfTaskSlots\":\"2\"},\"image\":\"flink:1.13\",\"imagePullPolicy\":null,\"serviceAccount\":\"flink\",\"flinkVersion\":\"v1_13\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"pod-template\"},\"spec\":{\"containers\":[{\"name\":\"flink-main-container\",\"resources\":{\"limits\":{\"ephemeral-storage\":\"2048Mi\"},\"requests\":{\"ephemeral-storage\":\"2048Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/opt/flink/volume\",\"name\":\"flink-volume\"}]}],\"volumes\":[{\"name\":\"flink-volume\",\"persistentVolumeClaim\":{\"claimName\":\"session-cluster-1-pvc\"}}]}},\"jobManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":\"native\"},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":null}
>
[jira] [Commented] (FLINK-33108) Error during error status handling
[
https://issues.apache.org/jira/browse/FLINK-33108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766309#comment-17766309
]
Gabor Somogyi commented on FLINK-33108:
---
After some digging the following value changed from:
{code:java}
"clusterInfo": {
"flink-revision": "b2ca390 @ 2022-02-03T14:54:22+01:00",
"flink-version": "1.13.6",
"total-cpu": "0.25",
"total-memory": "1073741824"
},
{code}
to:
{code:java}
"clusterInfo": {
"flink-revision": "b2ca390 @ 2022-02-03T14:54:22+01:00",
"flink-version": "1.13.6",
"total-cpu": "0.5",
"total-memory": "2147483648"
},
{code}
> Error during error status handling
> --
>
> Key: FLINK-33108
> URL: https://issues.apache.org/jira/browse/FLINK-33108
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gabor Somogyi
>Priority: Major
>
> e2e_ci (v1_13, flink, native, test_multi_sessionjob.sh) failed with the
> following issue:
> {code:java}
> Error: m2023-09-18 08:26:41,813 i.j.o.p.e.ReconciliationDispatcher
> [ERROR][flink/session-cluster-1] Error during error status handling.
> org.apache.flink.kubernetes.operator.exception.StatusConflictException:
> Status have been modified externally in version 1374 Previous:
> {"jobStatus":{"jobName":null,"jobId":null,"state":null,"startTime":null,"updateTime":null,"savepointInfo":{"lastSavepoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0},"checkpointInfo":{"lastCheckpoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"lastPeriodicCheckpointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"total-cpu":"0.25","flink-version":"1.13.6","flink-revision":"b2ca390
> @
>
[jira] [Created] (FLINK-33108) Error during error status handling
Gabor Somogyi created FLINK-33108:
-
Summary: Error during error status handling
Key: FLINK-33108
URL: https://issues.apache.org/jira/browse/FLINK-33108
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Gabor Somogyi
e2e_ci (v1_13, flink, native, test_multi_sessionjob.sh) failed with the
following issue:
{code:java}
Error: m2023-09-18 08:26:41,813 i.j.o.p.e.ReconciliationDispatcher
[ERROR][flink/session-cluster-1] Error during error status handling.
org.apache.flink.kubernetes.operator.exception.StatusConflictException: Status
have been modified externally in version 1374 Previous:
{"jobStatus":{"jobName":null,"jobId":null,"state":null,"startTime":null,"updateTime":null,"savepointInfo":{"lastSavepoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0},"checkpointInfo":{"lastCheckpoint":null,"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"lastPeriodicCheckpointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"total-cpu":"0.25","flink-version":"1.13.6","flink-revision":"b2ca390
@
2022-02-03T14:54:22+01:00","total-memory":"1073741824"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1695025410957,"lastReconciledSpec":"{\"spec\":{\"job\":null,\"restartNonce\":null,\"flinkConfiguration\":{\"high-availability\":\"org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory\",\"high-availability.storageDir\":\"file:///opt/flink/volume/flink-ha\",\"state.checkpoints.dir\":\"file:///opt/flink/volume/flink-cp\",\"state.savepoints.dir\":\"file:///opt/flink/volume/flink-sp\",\"taskmanager.numberOfTaskSlots\":\"2\"},\"image\":\"flink:1.13\",\"imagePullPolicy\":null,\"serviceAccount\":\"flink\",\"flinkVersion\":\"v1_13\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"pod-template\"},\"spec\":{\"containers\":[{\"name\":\"flink-main-container\",\"resources\":{\"limits\":{\"ephemeral-storage\":\"2048Mi\"},\"requests\":{\"ephemeral-storage\":\"2048Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/opt/flink/volume\",\"name\":\"flink-volume\"}]}],\"volumes\":[{\"name\":\"flink-volume\",\"persistentVolumeClaim\":{\"claimName\":\"session-cluster-1-pvc\"}}]}},\"jobManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":\"native\"},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":null,\"restartNonce\":null,\"flinkConfiguration\":{\"high-availability\":\"org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory\",\"high-availability.storageDir\":\"file:///opt/flink/volume/flink-ha\",\"state.checkpoints.dir\":\"file:///opt/flink/volume/flink-cp\",\"state.savepoints.dir\":\"file:///opt/flink/volume/flink-sp\",\"taskmanager.numberOfTaskSlots\":\"2\"},\"image\":\"flink:1.13\",\"imagePullPolicy\":null,\"serviceAccount\":\"flink\",\"flinkVersion\":\"v1_13\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"pod-template\"},\"spec\":{\"containers\":[{\"name\":\"flink-main-container\",\"resources\":{\"limits\":{\"ephemeral-storage\":\"2048Mi\"},\"requests\":{\"ephemeral-storage\":\"2048Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/opt/flink/volume\",\"name\":\"flink-volume\"}]}],\"volumes\":[{\"name\":\"flink-volume\",\"persistentVolumeClaim\":{\"claimName\":\"session-cluster-1-pvc\"}}]}},\"jobManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.25,\"memory\":\"1024m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":\"native\"},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":null}
Latest:
[jira] [Updated] (FLINK-30311) CI error: Back-off pulling image "flink:1.14"
[
https://issues.apache.org/jira/browse/FLINK-30311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi updated FLINK-30311:
--
Component/s: Kubernetes Operator
> CI error: Back-off pulling image "flink:1.14"
> -
>
> Key: FLINK-30311
> URL: https://issues.apache.org/jira/browse/FLINK-30311
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
>Reporter: Peter Vary
>Priority: Major
>
> CI failed with: {{Flink Deployment failed 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"}}
> We should find the root cause of this issue and try to mitigate it.
> [https://github.com/apache/flink-kubernetes-operator/actions/runs/3627824632/jobs/6118131271]
>
> {code:java}
> 2022-12-06T08:45:03.0243558Z [m[33m2022-12-06 08:41:44,716[m
> [36mo.a.f.k.o.c.FlinkDeploymentController[m
> [1;31m[ERROR][default/flink-example-statemachine] Flink Deployment failed
> 2022-12-06T08:45:03.0244383Z
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> Back-off pulling image "flink:1.14"
> 2022-12-06T08:45:03.0245385Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerBackoff(AbstractFlinkDeploymentObserver.java:194)
> 2022-12-06T08:45:03.0246604Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:150)
> 2022-12-06T08:45:03.0247780Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:84)
> 2022-12-06T08:45:03.0248934Z at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:55)
> 2022-12-06T08:45:03.0249941Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:56)
> 2022-12-06T08:45:03.0250844Z at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:32)
> 2022-12-06T08:45:03.0252038Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:113)
> 2022-12-06T08:45:03.0252936Z at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
> 2022-12-06T08:45:03.0253850Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
> 2022-12-06T08:45:03.0254412Z at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
> 2022-12-06T08:45:03.0255322Z at
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
> 2022-12-06T08:45:03.0256081Z at
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
> 2022-12-06T08:45:03.0256872Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
> 2022-12-06T08:45:03.0257804Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
> 2022-12-06T08:45:03.0258720Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
> 2022-12-06T08:45:03.0259635Z at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
> 2022-12-06T08:45:03.0260448Z at
> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
> 2022-12-06T08:45:03.0261070Z at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> 2022-12-06T08:45:03.0261595Z at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 2022-12-06T08:45:03.0262005Z at java.base/java.lang.Thread.run(Unknown
> Source) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33105: -- Component/s: Kubernetes Operator > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33105: - Assignee: Gabor Somogyi > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33105) Evaluate operator error log entry: Exception while fetching cluster info
[ https://issues.apache.org/jira/browse/FLINK-33105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33105: -- Summary: Evaluate operator error log entry: Exception while fetching cluster info (was: Evaluate operator log entry: Exception while fetching cluster info) > Evaluate operator error log entry: Exception while fetching cluster info > > > Key: FLINK-33105 > URL: https://issues.apache.org/jira/browse/FLINK-33105 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33105) Evaluate operator log entry: Exception while fetching cluster info
Gabor Somogyi created FLINK-33105: - Summary: Evaluate operator log entry: Exception while fetching cluster info Key: FLINK-33105 URL: https://issues.apache.org/jira/browse/FLINK-33105 Project: Flink Issue Type: Sub-task Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765457#comment-17765457 ] Gabor Somogyi commented on FLINK-33029: --- I've just double checked and wheel building has been fixed in the next nightly: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53251=results > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reopened FLINK-33029: --- > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765205#comment-17765205 ] Gabor Somogyi commented on FLINK-33029: --- Nightly failed: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53196=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb Creating a PR to fix it. > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-33029. --- Resolution: Fixed 50cb4ee on master > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-33029. - > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated FLINK-33029: -- Fix Version/s: 1.19.0 > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33030) Add python 3.11 support
[ https://issues.apache.org/jira/browse/FLINK-33030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764154#comment-17764154 ] Gabor Somogyi commented on FLINK-33030: --- Pemja version 0.4.0 is going to be released soon which will unblock this. Thanks! > Add python 3.11 support > --- > > Key: FLINK-33030 > URL: https://issues.apache.org/jira/browse/FLINK-33030 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33030) Add python 3.11 support
[
https://issues.apache.org/jira/browse/FLINK-33030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764098#comment-17764098
]
Gabor Somogyi edited comment on FLINK-33030 at 9/12/23 12:21 PM:
-
Pemja is not supporting python 3.11:
{code:java}
pip install -r dev/dev-requirements.txt
...
Collecting pemja==0.3.1 (from -r dev/dev-requirements.txt (line 31))
Downloading pemja-0.3.1.tar.gz (48 kB)
48.4/48.4 kB 1.2 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [9 lines of output]
Collecting packaging==20.5
Using cached packaging-20.5-py2.py3-none-any.whl (35 kB)
Collecting setuptools==59.2.0
Using cached setuptools-59.2.0-py3-none-any.whl (952 kB)
Collecting wheel==0.37.0
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
ERROR: Ignored the following versions that require a different python
version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python
>=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python
>=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
{code}
Asked [~hxbks2ks] for help: https://github.com/alibaba/pemja/issues/36
was (Author: gaborgsomogyi):
Pemja is not supporting python 3.11:
{code:java}
pip install -r dev/dev-requirements.txt
...
Collecting pemja==0.3.1 (from -r dev/dev-requirements.txt (line 31))
Downloading pemja-0.3.1.tar.gz (48 kB)
48.4/48.4 kB 1.2 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [9 lines of output]
Collecting packaging==20.5
Using cached packaging-20.5-py2.py3-none-any.whl (35 kB)
Collecting setuptools==59.2.0
Using cached setuptools-59.2.0-py3-none-any.whl (952 kB)
Collecting wheel==0.37.0
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
ERROR: Ignored the following versions that require a different python
version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python
>=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python
>=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
{code}
Asked [~huangxing] for help: https://github.com/alibaba/pemja/issues/36
> Add python 3.11 support
> ---
>
> Key: FLINK-33030
> URL: https://issues.apache.org/jira/browse/FLINK-33030
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
>Affects Versions: 1.19.0
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33030) Add python 3.11 support
[
https://issues.apache.org/jira/browse/FLINK-33030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764098#comment-17764098
]
Gabor Somogyi commented on FLINK-33030:
---
Pemja is not supporting python 3.11:
{code:java}
pip install -r dev/dev-requirements.txt
...
Collecting pemja==0.3.1 (from -r dev/dev-requirements.txt (line 31))
Downloading pemja-0.3.1.tar.gz (48 kB)
48.4/48.4 kB 1.2 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [9 lines of output]
Collecting packaging==20.5
Using cached packaging-20.5-py2.py3-none-any.whl (35 kB)
Collecting setuptools==59.2.0
Using cached setuptools-59.2.0-py3-none-any.whl (952 kB)
Collecting wheel==0.37.0
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
ERROR: Ignored the following versions that require a different python
version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python
>=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python
>=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
{code}
Asked [~huangxing] for help: https://github.com/alibaba/pemja/issues/36
> Add python 3.11 support
> ---
>
> Key: FLINK-33030
> URL: https://issues.apache.org/jira/browse/FLINK-33030
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
>Affects Versions: 1.19.0
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33030) Add python 3.11 support
[ https://issues.apache.org/jira/browse/FLINK-33030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33030: - Assignee: Gabor Somogyi > Add python 3.11 support > --- > > Key: FLINK-33030 > URL: https://issues.apache.org/jira/browse/FLINK-33030 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33029) Drop python 3.7 support
[ https://issues.apache.org/jira/browse/FLINK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned FLINK-33029: - Assignee: Gabor Somogyi > Drop python 3.7 support > --- > > Key: FLINK-33029 > URL: https://issues.apache.org/jira/browse/FLINK-33029 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.19.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33030) Add python 3.11 support
Gabor Somogyi created FLINK-33030: - Summary: Add python 3.11 support Key: FLINK-33030 URL: https://issues.apache.org/jira/browse/FLINK-33030 Project: Flink Issue Type: New Feature Components: API / Python Affects Versions: 1.19.0 Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
