Avi Sanwal created FLINK-34561:
----------------------------------
Summary: Downgrading flink-kubernetes-operator causes failure
Key: FLINK-34561
URL: https://issues.apache.org/jira/browse/FLINK-34561
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Avi Sanwal
Hi,
We are currently using flink-kubernetes-operator 1.4 version (with v1beta1 CRD)
to manage flink deployments.
We tried an upgrade to version 1.6 version, but when we try to rollback to 1.4,
noticed that flink-operator fails to come up with this execption:
{noformat}
Exception in thread "main" io.javaoperatorsdk.operator.OperatorException: Error
starting operator
at io.javaoperatorsdk.operator.Operator.start(Operator.java:125)
at
org.apache.flink.kubernetes.operator.FlinkOperator.run(FlinkOperator.java:215)
at
org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:229)
Caused by: io.javaoperatorsdk.operator.OperatorException:
java.lang.IllegalStateException:
com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize
spec, this indicates a bug... (through reference chain:
org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
at
io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:107)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at
io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:102)
at
io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:82)
at
io.javaoperatorsdk.operator.ControllerManager.start(ControllerManager.java:36)
at io.javaoperatorsdk.operator.Operator.start(Operator.java:120)
... 2 more
Caused by: java.lang.IllegalStateException:
com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize
spec, this indicates a bug... (through reference chain:
org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
at
io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:112)
at java.base/java.util.Optional.map(Unknown Source)
at
io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.get(InformerManager.java:175)
at
io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.get(ManagedInformerEventSource.java:110)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor.submitReconciliationExecution(EventProcessor.java:128)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor.handleMarkedEventForResource(EventProcessor.java:120)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor.handleAlreadyMarkedEvents(EventProcessor.java:376)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor.start(EventProcessor.java:371)
at
io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:336)
at
io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:37)
at
io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:96)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not
deserialize spec, this indicates a bug... (through reference chain:
org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
at
com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:782)
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
at
io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at
com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4624)
at
com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3869)
at
io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:110)
... 14 more
Caused by: java.lang.RuntimeException: Could not deserialize spec, this
indicates a bug...
at
org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:64)
at
org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpecWithMeta(ReconciliationStatus.java:71)
at
org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpec(ReconciliationStatus.java:59)
at
org.apache.flink.kubernetes.operator.api.status.CommonStatus.getLifecycleState(CommonStatus.java:75)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:689)
at
io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
... 24 more
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
Unrecognized field "ephemeralStorage" (class
org.apache.flink.kubernetes.operator.api.spec.Resource), not marked as
ignorable (2 known properties: "cpu", "memory"])
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain:
org.apache.flink.kubernetes.operator.api.spec.FlinkDeploymentSpec["jobManager"]->org.apache.flink.kubernetes.operator.api.spec.JobManagerSpec["resource"]->org.apache.flink.kubernetes.operator.api.spec.Resource["ephemeralStorage"])
at
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
at
com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
at
com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at
com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at
com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at
com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at
com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4706)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2879)
at
com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3343)
at
org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:60)
... 34 more
{noformat}
Looks like some FlinkDeployments were created by 1.6 version and later 1.4 was
able to handle those.
But as per
[docs|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/compatibility/],
the FlinkDeployment created by 1.6 should have been backward compatible with
1.4.
{quote}Starting from v1beta1 (operator version 1.0.0) we aim to provide
backward compatibility for the already deployed Flink custom resources
(FlinkDeployment, FlinkSessionJob).
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
