[
https://issues.apache.org/jira/browse/FLINK-34561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822889#comment-17822889
]
Avi Sanwal commented on FLINK-34561:
------------------------------------
[~gyfora], thanks for checking! Apologies, I misunderstood the intent behind
backward compatibility. Now I am clear that the downgrade is not forward
compatible currently.
Still, I'd like to request if this is possible to keep the forward
compatibility in mind for future releases, considering that due to unexpected
issues, if we need to rollback the operator to older version, currently it
would cause significant downtime.
Glad to hear that deserialization is already made more liberal in 1.7.0, we
would consider upgrading to it.
> Downgrading flink-kubernetes-operator causes failure
> ----------------------------------------------------
>
> Key: FLINK-34561
> URL: https://issues.apache.org/jira/browse/FLINK-34561
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Avi Sanwal
> Priority: Major
>
> Hi,
> We are currently using flink-kubernetes-operator 1.4 version (with v1beta1
> CRD) to manage flink deployments.
> We tried an upgrade to version 1.6 version, but when we try to rollback to
> 1.4, noticed that flink-operator fails to come up with this execption:
> {noformat}
> Exception in thread "main" io.javaoperatorsdk.operator.OperatorException:
> Error starting operator
> at io.javaoperatorsdk.operator.Operator.start(Operator.java:125)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.run(FlinkOperator.java:215)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:229)
> Caused by: io.javaoperatorsdk.operator.OperatorException:
> java.lang.IllegalStateException:
> com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize
> spec, this indicates a bug... (through reference chain:
> org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
> at
> io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:107)
> at java.base/java.util.ArrayList.forEach(Unknown Source)
> at
> io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:102)
> at
> io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:82)
> at
> io.javaoperatorsdk.operator.ControllerManager.start(ControllerManager.java:36)
> at io.javaoperatorsdk.operator.Operator.start(Operator.java:120)
> ... 2 more
> Caused by: java.lang.IllegalStateException:
> com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize
> spec, this indicates a bug... (through reference chain:
> org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
> at
> io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:112)
> at java.base/java.util.Optional.map(Unknown Source)
> at
> io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.get(InformerManager.java:175)
> at
> io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.get(ManagedInformerEventSource.java:110)
> at
> io.javaoperatorsdk.operator.processing.event.EventProcessor.submitReconciliationExecution(EventProcessor.java:128)
> at
> io.javaoperatorsdk.operator.processing.event.EventProcessor.handleMarkedEventForResource(EventProcessor.java:120)
> at
> io.javaoperatorsdk.operator.processing.event.EventProcessor.handleAlreadyMarkedEvents(EventProcessor.java:376)
> at
> io.javaoperatorsdk.operator.processing.event.EventProcessor.start(EventProcessor.java:371)
> at
> io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:336)
> at
> io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:37)
> at
> io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:96)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not
> deserialize spec, this indicates a bug... (through reference chain:
> org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
> at
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:782)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
> at
> io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
> at
> com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4624)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3869)
> at
> io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:110)
> ... 14 more
> Caused by: java.lang.RuntimeException: Could not deserialize spec, this
> indicates a bug...
> at
> org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:64)
> at
> org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpecWithMeta(ReconciliationStatus.java:71)
> at
> org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpec(ReconciliationStatus.java:59)
> at
> org.apache.flink.kubernetes.operator.api.status.CommonStatus.getLifecycleState(CommonStatus.java:75)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:689)
> at
> io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
> ... 24 more
> Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> Unrecognized field "ephemeralStorage" (class
> org.apache.flink.kubernetes.operator.api.spec.Resource), not marked as
> ignorable (2 known properties: "cpu", "memory"])
> at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain:
> org.apache.flink.kubernetes.operator.api.spec.FlinkDeploymentSpec["jobManager"]->org.apache.flink.kubernetes.operator.api.spec.JobManagerSpec["resource"]->org.apache.flink.kubernetes.operator.api.spec.Resource["ephemeralStorage"])
> at
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
> at
> com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
> at
> com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4706)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2879)
> at
> com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3343)
> at
> org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:60)
> ... 34 more
> {noformat}
> Looks like some FlinkDeployments were created by 1.6 version and later 1.4
> was able to handle those.
> But as per
> [docs|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/compatibility/],
> the FlinkDeployment created by 1.6 should have been backward compatible with
> 1.4.
> {quote}Starting from v1beta1 (operator version 1.0.0) we aim to provide
> backward compatibility for the already deployed Flink custom resources
> (FlinkDeployment, FlinkSessionJob).
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
