[jira] [Updated] (FLINK-26446) Update Feature Radar in Apache Flink Roadmap
[ https://issues.apache.org/jira/browse/FLINK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-26446: - Description: Deployment/Coordination: * Java 8 -> Deprecation * Add Deployment Modes ** Application Mode -> Stable ** Session Mode -> Stable ** Per-Job Mode -> Deprecated * Adaptive Scheduler -> Ready & Evolving Connectors (removing all connectors from this list due to connectors being externalized: let's only focus on the interfaces) * Remove NiFi Source * Remove Kafka, File [via Unified Sink API] * HybridSource -> Ready & Evolving * Remove Hive SQL Source & Sink * Remove JDBC Sink * Remove Kinesis Source & Sink * Remove Kafka, File, Pulsar [via Unified Source API] * Remove Change-Data-Capture API and Connectors * Remove Rabbit MQ Source * Remove PubSub Source & Sink * Remove HBase SQL Source & Sink * Remove Elastic Search Sink * Remove Cassandra Sink * Legacy File Source & Sink -> Deprecated * Legacy Kafka Source & Sink -> Deprecated Resource Managers: * Remove Scala Shell APIs: * SourceFunction & SinkFunction -> Deprecated * Add Unified Sink API -> Ready & Evolving * Add Unified Source API -> Stable * Add ASync API -> Beta * Add Topology Sink -> Beta * Add SQL Upgrade Support -> MVP Languages: * Remove Scala 2.11 Libraries: * Gelly -> Deprecated * State Processor API -> Ready & Evolving was: Deployment/Coordination: * Java 8 -> Deprecation * Add Deployment Modes ** Application Mode -> Stable ** Session Mode -> Stable ** Per-Job Mode -> Deprecated * Adaptive Scheduler -> Ready & Evolving Connectors (removing all connectors from this list due to connectors being externalized: let's only focus on the interfaces) * Remove NiFi Source * Remove Kafka, File [via Unified Sink API] * HybridSource -> Ready & Evolving * Remove Hive SQL Source & Sink * Remove JDBC Sink * Remove Kinesis Source & Sink * Remove Kafka, File, Pulsar [via Unified Source API] * Remove Change-Data-Capture API and Connectors * Remove Rabbit MQ Source * Remove PubSub Source & Sink * Remove HBase SQL Source & Sink * Remove Elastic Search Sink * Remove Cassandra Sink * Legacy File Source & Sink -> Deprecated * Legacy Kafka Source & Sink -> Deprecated Resource Managers: * Remove Scala Shell APIs: * SourceFunction & SinkFunction -> Deprecated * Add Unified Sink API -> Ready & Evolving * Add Unified Source API -> Stable * Add ASync API -> Beta * Add Topology Sink -> Beta * Add SQL Upgrade Support -> MVP Languages: * Remove Scala 2.11 Libraries: * Gelly -> Deprecated > Update Feature Radar in Apache Flink Roadmap > - > > Key: FLINK-26446 > URL: https://issues.apache.org/jira/browse/FLINK-26446 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Konstantin Knauf >Priority: Critical > Fix For: 1.15.0 > > > Deployment/Coordination: > * Java 8 -> Deprecation > * Add Deployment Modes > ** Application Mode -> Stable > ** Session Mode -> Stable > ** Per-Job Mode -> Deprecated > * Adaptive Scheduler -> Ready & Evolving > Connectors (removing all connectors from this list due to connectors being > externalized: let's only focus on the interfaces) > * Remove NiFi Source > * Remove Kafka, File [via Unified Sink API] > * HybridSource -> Ready & Evolving > * Remove Hive SQL Source & Sink > * Remove JDBC Sink > * Remove Kinesis Source & Sink > * Remove Kafka, File, Pulsar [via Unified Source API] > * Remove Change-Data-Capture API and Connectors > * Remove Rabbit MQ Source > * Remove PubSub Source & Sink > * Remove HBase SQL Source & Sink > * Remove Elastic Search Sink > * Remove Cassandra Sink > * Legacy File Source & Sink -> Deprecated > * Legacy Kafka Source & Sink -> Deprecated > Resource Managers: > * Remove Scala Shell > APIs: > * SourceFunction & SinkFunction -> Deprecated > * Add Unified Sink API -> Ready & Evolving > * Add Unified Source API -> Stable > * Add ASync API -> Beta > * Add Topology Sink -> Beta > * Add SQL Upgrade Support -> MVP > Languages: > * Remove Scala 2.11 > Libraries: > * Gelly -> Deprecated > * State Processor API -> Ready & Evolving -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26374) JSON_OBJECT may throw NullPointerException on nullable column
Seth Wiesman created FLINK-26374:
Summary: JSON_OBJECT may throw NullPointerException on nullable
column
Key: FLINK-26374
URL: https://issues.apache.org/jira/browse/FLINK-26374
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.15.0
Reporter: Seth Wiesman
>From ML:
Using the latest SNAPSHOT BUILD.
If I have a column definition as
.column(
"events",
DataTypes.ARRAY(
DataTypes.ROW(
DataTypes.FIELD("status", DataTypes.STRING().notNull()),
DataTypes.FIELD("timestamp",
DataTypes.STRING().notNull()),
DataTypes.FIELD("increment_identifier",
DataTypes.STRING().nullable()
And a query as
JSON_OBJECT('events' VALUE events) event_json
Will generate JSON correctly ONLY if increment_identifier is NOT NULL but will
throw a NullPointerException on the first record that has that column as null.
Exception is not helpful.
Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:259)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
at akka.dispatch.OnComplete.internal(Future.scala:300)
at akka.dispatch.OnComplete.internal(Future.scala:297)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at
[jira] [Closed] (FLINK-26340) Add ability in Golang SDK to create new statefun.Context from existing one, but with a new underlying context.Context
[ https://issues.apache.org/jira/browse/FLINK-26340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-26340. Resolution: Fixed > Add ability in Golang SDK to create new statefun.Context from existing one, > but with a new underlying context.Context > - > > Key: FLINK-26340 > URL: https://issues.apache.org/jira/browse/FLINK-26340 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.3.0 >Reporter: Galen Warren >Assignee: Galen Warren >Priority: Minor > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > In the Golang SDK, statefun.Context embeds the context.Context interface and > is implemented by the statefunContext struct, which embeds a context.Context. > To support common patterns in Golang related to adding values to context, it > would be useful to be able to create a derived statefun.Context that is > equivalent to the original in terms of statefun functionality but which wraps > a different context.Context. > The proposal is to add a: > WithContext(ctx context.Context) statefun.Context > ... method to the statefun.Context interface and implement it on > statefunContext. This method would return the derived statefun context. > This is a breaking change to statefun.Context, but, given its purpose, we do > not expect there to be implementations of this interface outside the Golang > SDK. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26340) Add ability in Golang SDK to create new statefun.Context from existing one, but with a new underlying context.Context
[ https://issues.apache.org/jira/browse/FLINK-26340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497743#comment-17497743 ] Seth Wiesman commented on FLINK-26340: -- resolved in: 725202cd69b78442d1287deec000ae2d52da4bda > Add ability in Golang SDK to create new statefun.Context from existing one, > but with a new underlying context.Context > - > > Key: FLINK-26340 > URL: https://issues.apache.org/jira/browse/FLINK-26340 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.3.0 >Reporter: Galen Warren >Assignee: Galen Warren >Priority: Minor > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > In the Golang SDK, statefun.Context embeds the context.Context interface and > is implemented by the statefunContext struct, which embeds a context.Context. > To support common patterns in Golang related to adding values to context, it > would be useful to be able to create a derived statefun.Context that is > equivalent to the original in terms of statefun functionality but which wraps > a different context.Context. > The proposal is to add a: > WithContext(ctx context.Context) statefun.Context > ... method to the statefun.Context interface and implement it on > statefunContext. This method would return the derived statefun context. > This is a breaking change to statefun.Context, but, given its purpose, we do > not expect there to be implementations of this interface outside the Golang > SDK. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26200) Redirect statefun directly to docs
[ https://issues.apache.org/jira/browse/FLINK-26200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493547#comment-17493547 ] Seth Wiesman commented on FLINK-26200: -- resolved in 5a8b85bd6c449ce4edebacf332568cd4b6845837 > Redirect statefun directly to docs > -- > > Key: FLINK-26200 > URL: https://issues.apache.org/jira/browse/FLINK-26200 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-26200) Redirect statefun directly to docs
[ https://issues.apache.org/jira/browse/FLINK-26200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-26200. Resolution: Fixed > Redirect statefun directly to docs > -- > > Key: FLINK-26200 > URL: https://issues.apache.org/jira/browse/FLINK-26200 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26200) Redirect statefun directly to docs
Seth Wiesman created FLINK-26200: Summary: Redirect statefun directly to docs Key: FLINK-26200 URL: https://issues.apache.org/jira/browse/FLINK-26200 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Seth Wiesman Assignee: Seth Wiesman -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman closed FLINK-25423.
Resolution: Fixed
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485909#comment-17485909
]
Seth Wiesman commented on FLINK-25423:
--
fixed in master: 8069a0f0b8c2172a90aa509de08df865884ff815
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25528) state processor api do not support increment checkpoint
[ https://issues.apache.org/jira/browse/FLINK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469379#comment-17469379 ] Seth Wiesman commented on FLINK-25528: -- The state processor api only takes a single snapshot at end of input. Even if we used incremental snapshots it would still be the entire state backend. > state processor api do not support increment checkpoint > --- > > Key: FLINK-25528 > URL: https://issues.apache.org/jira/browse/FLINK-25528 > Project: Flink > Issue Type: Improvement > Components: API / State Processor, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > > As the title, in the state-processor-api, we use the savepoint opition to > snapshot state defaultly in org.apache.flink.state.api.output.SnapshotUtils. > But in many cases, we maybe need to snapshot state incremently which have > better performance than savepoint. > Shall we add the config to chose the checkpoint way just like flink stream > backend? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466227#comment-17466227
]
Seth Wiesman commented on FLINK-25423:
--
[~ym] I've opened a PR for this change. It would be great if you could help
review.
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-24912) Migrate state processor API to DataStream API
[ https://issues.apache.org/jira/browse/FLINK-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24912. Fix Version/s: 1.15.0 Release Note: The State Processor API has been migrated from Flinks legacy DataSet API to now run over DataStreams run under BATCH execution. Resolution: Fixed > Migrate state processor API to DataStream API > - > > Key: FLINK-24912 > URL: https://issues.apache.org/jira/browse/FLINK-24912 > Project: Flink > Issue Type: Improvement > Components: API / State Processor >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Jun Qin >Priority: Major > Fix For: 1.15.0 > > > Now that DataStream supports bounded execution, we should investigate > migrating the State Processor API off DataSet. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24912) Migrate state processor API to DataStream API
[ https://issues.apache.org/jira/browse/FLINK-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466158#comment-17466158 ] Seth Wiesman commented on FLINK-24912: -- Resolved in 3b54d27b5174c588b80525e04a9af37613730681 > Migrate state processor API to DataStream API > - > > Key: FLINK-24912 > URL: https://issues.apache.org/jira/browse/FLINK-24912 > Project: Flink > Issue Type: Improvement > Components: API / State Processor >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Jun Qin >Priority: Major > > Now that DataStream supports bounded execution, we should investigate > migrating the State Processor API off DataSet. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17465074#comment-17465074
]
Seth Wiesman edited comment on FLINK-25423 at 12/24/21, 7:52 PM:
-
Hi Yuan,
we can get this into 1.15 without any problem. Once FLINK-24912 is merged it
should be straightforward to implement
was (Author: sjwiesman):
Hi Yuan,
we can get this into 1.15 without any problem. Once FLINK-24921 is merged it
should be straightforward to implement
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17465074#comment-17465074
]
Seth Wiesman commented on FLINK-25423:
--
Hi Yuan,
we can get this into 1.15 without any problem. Once FLINK-24921 is merged it
should be straightforward to implement
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25026) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464800#comment-17464800
]
Seth Wiesman commented on FLINK-25026:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28549=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails on AZP
> --
>
> Key: FLINK-25026
> URL: https://issues.apache.org/jira/browse/FLINK-25026
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.3
>
>
> {{UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint}} fails
> on AZP with
> {code}
> 2021-11-23T00:58:03.8286352Z Nov 23 00:58:03 [ERROR] Tests run: 72, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 716.362 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase
> 2021-11-23T00:58:03.8288790Z Nov 23 00:58:03 [ERROR]
> shouldRescaleUnalignedCheckpoint[downscale union from 3 to 2,
> buffersPerChannel = 1] Time elapsed: 4.051 s <<< ERROR!
> 2021-11-23T00:58:03.8289953Z Nov 23 00:58:03
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-11-23T00:58:03.8291473Z Nov 23 00:58:03 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-11-23T00:58:03.8292776Z Nov 23 00:58:03 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:168)
> 2021-11-23T00:58:03.8294520Z Nov 23 00:58:03 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:534)
> 2021-11-23T00:58:03.8295909Z Nov 23 00:58:03 at
> jdk.internal.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
> 2021-11-23T00:58:03.8297310Z Nov 23 00:58:03 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-23T00:58:03.8298922Z Nov 23 00:58:03 at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2021-11-23T00:58:03.8300298Z Nov 23 00:58:03 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-23T00:58:03.8301741Z Nov 23 00:58:03 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-23T00:58:03.8303233Z Nov 23 00:58:03 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-23T00:58:03.8304514Z Nov 23 00:58:03 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-23T00:58:03.8305736Z Nov 23 00:58:03 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-11-23T00:58:03.8306856Z Nov 23 00:58:03 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-23T00:58:03.8308218Z Nov 23 00:58:03 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-23T00:58:03.8309532Z Nov 23 00:58:03 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-23T00:58:03.8310780Z Nov 23 00:58:03 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-23T00:58:03.8312026Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-23T00:58:03.8313515Z Nov 23 00:58:03 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-23T00:58:03.8314842Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-23T00:58:03.8316116Z Nov 23 00:58:03 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-23T00:58:03.8317538Z Nov 23 00:58:03 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-23T00:58:03.8320044Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-23T00:58:03.8321044Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-23T00:58:03.8321978Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-23T00:58:03.8322915Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-11-23T00:58:03.8323848Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-11-23T00:58:03.8325330Z Nov 23 00:58:03 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-11-23T00:58:03.8337747Z Nov 23
[jira] [Commented] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464577#comment-17464577
]
Seth Wiesman commented on FLINK-25423:
--
This sounds reasonable. I’m actively working on porting the api from dataset to
datastream. I’ll pick this up as follow up to that work
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-25423) Enable loading state backend via configuration in state processor api
[
https://issues.apache.org/jira/browse/FLINK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman reassigned FLINK-25423:
Assignee: Seth Wiesman
> Enable loading state backend via configuration in state processor api
> -
>
> Key: FLINK-25423
> URL: https://issues.apache.org/jira/browse/FLINK-25423
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor, Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Seth Wiesman
>Priority: Major
> Fix For: 1.15.0
>
>
> Currently, state processor API would load savepoint via explictly
> initalizated state backend on client side, which is like
> {{StreamExecutionEnvironment#setStateBackend(stateBackend)}}:
> {code:java}
> Savepoint.load(bEnv, "hdfs://path/", new HashMapStateBackend());
> {code}
> As we all konw, stream env also support to load state backend via
> configuration to provide flexibility to load state backends especially some
> customized state backend. This could also benefit state processor API with
> similiar ability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25360) Add State Desc to CheckpointMetadata
[ https://issues.apache.org/jira/browse/FLINK-25360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464028#comment-17464028 ] Seth Wiesman commented on FLINK-25360: -- [~yunta] there isn't any way to do this with the state processor api beyond what's already been discussed. The checkpoint metadata does not contain enough information to rewrite the snapshot automatically. > Add State Desc to CheckpointMetadata > > > Key: FLINK-25360 > URL: https://issues.apache.org/jira/browse/FLINK-25360 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > Attachments: image-2021-12-17-20-01-42-423.png > > > Now we can't get the State Descriptor info in the checkpoint meta. Like the > case if we use state-processor-api to load state then rewrite state, we can't > flexible use the state. > Maybe there are other cases we need the State Descriptor, so can we add this > info? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25197) Using Statefun RequestReplyFunctionBuilder fails with Java 8 date/time type `java.time.Duration` not supported by default: add Module "org.apache.flink.shaded.jackson2.
[ https://issues.apache.org/jira/browse/FLINK-25197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-25197: Assignee: Galen Warren > Using Statefun RequestReplyFunctionBuilder fails with Java 8 date/time type > `java.time.Duration` not supported by default: add Module > "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jackson-datatype-jsr310" > to enable handling > --- > > Key: FLINK-25197 > URL: https://issues.apache.org/jira/browse/FLINK-25197 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Galen Warren >Assignee: Galen Warren >Priority: Major > Fix For: statefun-3.1.0 > > > When using RequestReplyFunctionBuilder to build a stateful functions job, the > job fails at runtime with: > Java 8 date/time type `java.time.Duration` not supported by default: add > Module > "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jackson-datatype-jsr310" > to enable handling > It appears this is because, in > [RequestReplyFunctionBuilder::transportClientPropertiesAsObjectNode|https://github.com/apache/flink-statefun/blob/b4ba9547b8f0105a28544fd28a5e0433666e9023/statefun-flink/statefun-flink-datastream/src/main/java/org/apache/flink/statefun/flink/datastream/RequestReplyFunctionBuilder.java#L127], > a default instance of ObjectMapper is used to serialize the client > properties, which now include a java.time.Duration. There is a > [StateFunObjectMapper|https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-common/src/main/java/org/apache/flink/statefun/flink/common/json/StateFunObjectMapper.java] > class in the project that has customized serde support, but it is not used > here. > The fix seems to be to: > * Use an instance of StateFunObjectMapper to serialize the client properties > in RequestReplyFunctionBuilder > * Modify StateFunObjectMapper to both serialize and deserialize instances of > java.time.Duration (currently, only deserialization is supported) > I've made these changes locally and it seems to fix the problem. Would you be > interested in a PR? Thanks. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25333) Harden UID management when constructing statefun universe
[ https://issues.apache.org/jira/browse/FLINK-25333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-25333. Resolution: Fixed > Harden UID management when constructing statefun universe > - > > Key: FLINK-25333 > URL: https://issues.apache.org/jira/browse/FLINK-25333 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.2.0 > > > As described in this stack overflow, we do not set UIDs on routers. Because > of how the stream graph is generated, the uids may be non-deterministic. We > should manually set all UIDs and enforce this via configuration. > > https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25333) Harden UID management when constructing statefun universe
[ https://issues.apache.org/jira/browse/FLINK-25333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460257#comment-17460257 ] Seth Wiesman commented on FLINK-25333: -- resolved in flink-statefun-docker: 7f8a9b29aee841a252308436e2fc2524702687bb flink-statefun: 99892751a60d31991161a097923fbd5ee08cb2eb > Harden UID management when constructing statefun universe > - > > Key: FLINK-25333 > URL: https://issues.apache.org/jira/browse/FLINK-25333 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.2.0 > > > As described in this stack overflow, we do not set UIDs on routers. Because > of how the stream graph is generated, the uids may be non-deterministic. We > should manually set all UIDs and enforce this via configuration. > > https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25333) Harden UID management when constructing statefun universe
[ https://issues.apache.org/jira/browse/FLINK-25333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460256#comment-17460256 ] Seth Wiesman commented on FLINK-25333: -- [~keremulutas] yes, that log line was actually the clue to me of what was happening. You should never see that with a statefun application. > Harden UID management when constructing statefun universe > - > > Key: FLINK-25333 > URL: https://issues.apache.org/jira/browse/FLINK-25333 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.2.0 > > > As described in this stack overflow, we do not set UIDs on routers. Because > of how the stream graph is generated, the uids may be non-deterministic. We > should manually set all UIDs and enforce this via configuration. > > https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman closed FLINK-25267.
Resolution: Duplicate
> Unable to (always) recover using checkpoint in HA setup (both Zookeeper and
> Kubernetes)
> ---
>
> Key: FLINK-25267
> URL: https://issues.apache.org/jira/browse/FLINK-25267
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Checkpointing,
> Stateful Functions
>Affects Versions: 1.12.1, statefun-3.0.0, statefun-3.1.0, 1.13.2
> Environment: MacOS 11.6, minikube v1.23.2, tried with both Stateful
> Functions 3.0.0 and Stateful Functions 3.1.0
>Reporter: Kerem Ulutaş
>Priority: Major
>
> My Stateful Functions job is running on Kubernetes (minikube on my local env)
> and has these settings:
> * Using StateFun v3.1.0
> * Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
> * Checkpointing mode is EXACTLY_ONCE
> * State backend is rocksdb and incremental checkpointing is enabled
> When I kill the jobmanager (master) pod, minikube starts another pod and this
> new pod fails when it tries to load last checkpoint:
> {code:java}
> ...
> 2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Initializing job myStatefunApp
> ().
> 2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using restart back off time strategy
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
> backoffTimeMS=1000) for myStatefunApp ().
> 2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
> [] - Initialized DefaultCompletedCheckpointStore in
> 'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
> with /checkpoints/.
> 2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Running initialization on master for job myStatefunApp
> ().
> 2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Successfully ran initialization on master in 0 ms.
> 2021-12-11 14:25:26,617 INFO
> org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] -
> Built 1 pipelined regions in 1 ms
> 2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using job/cluster config to configure application-defined
> state backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
> enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
> writeBatchSize=2097152}
> 2021-12-11 14:25:26,627 INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Using predefined options: DEFAULT.
> 2021-12-11 14:25:26,627 INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Using application-defined options factory:
> DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
> 2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using application-defined state backend:
> EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
> enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
> writeBatchSize=2097152}
> 2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Checkpoint storage is set to 'filesystem': (checkpoints
> "hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
> 2021-12-11 14:25:26,712 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Recovering checkpoints from
> ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
> 2021-12-11 14:25:26,724 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Found 1 checkpoints in
> ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
> 2021-12-11 14:25:26,725 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Trying to fetch 1 checkpoints from storage.
> 2021-12-11 14:25:26,725 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Trying to retrieve checkpoint 2.
> 2021-12-11 14:25:26,931 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring
> job from Checkpoint 2 @ 1639232587220 for
> located
[jira] [Commented] (FLINK-25267) Unable to (always) recover using checkpoint in HA setup (both Zookeeper and Kubernetes)
[
https://issues.apache.org/jira/browse/FLINK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460213#comment-17460213
]
Seth Wiesman commented on FLINK-25267:
--
Apologies for not seeing this ticket earlier. For those watching, there was
some non-determinism in how statefun generate UIDs on a particular operator. I
will close this as a duplicate for now.
> Unable to (always) recover using checkpoint in HA setup (both Zookeeper and
> Kubernetes)
> ---
>
> Key: FLINK-25267
> URL: https://issues.apache.org/jira/browse/FLINK-25267
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Checkpointing,
> Stateful Functions
>Affects Versions: 1.12.1, statefun-3.0.0, statefun-3.1.0, 1.13.2
> Environment: MacOS 11.6, minikube v1.23.2, tried with both Stateful
> Functions 3.0.0 and Stateful Functions 3.1.0
>Reporter: Kerem Ulutaş
>Priority: Major
>
> My Stateful Functions job is running on Kubernetes (minikube on my local env)
> and has these settings:
> * Using StateFun v3.1.0
> * Checkpoints are stored on HDFS (state.checkpoint-storage: filesystem)
> * Checkpointing mode is EXACTLY_ONCE
> * State backend is rocksdb and incremental checkpointing is enabled
> When I kill the jobmanager (master) pod, minikube starts another pod and this
> new pod fails when it tries to load last checkpoint:
> {code:java}
> ...
> 2021-12-11 14:25:26,426 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Initializing job myStatefunApp
> ().
> 2021-12-11 14:25:26,443 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using restart back off time strategy
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647,
> backoffTimeMS=1000) for myStatefunApp ().
> 2021-12-11 14:25:26,516 INFO org.apache.flink.runtime.util.ZooKeeperUtils
> [] - Initialized DefaultCompletedCheckpointStore in
> 'ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}'
> with /checkpoints/.
> 2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Running initialization on master for job myStatefunApp
> ().
> 2021-12-11 14:25:26,599 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Successfully ran initialization on master in 0 ms.
> 2021-12-11 14:25:26,617 INFO
> org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] -
> Built 1 pipelined regions in 1 ms
> 2021-12-11 14:25:26,626 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using job/cluster config to configure application-defined
> state backend: EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
> enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
> writeBatchSize=2097152}
> 2021-12-11 14:25:26,627 INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Using predefined options: DEFAULT.
> 2021-12-11 14:25:26,627 INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Using application-defined options factory:
> DefaultConfigurableOptionsFactory{configuredOptions={state.backend.rocksdb.thread.num=1}}.
> 2021-12-11 14:25:26,627 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Using application-defined state backend:
> EmbeddedRocksDBStateBackend{, localRocksDbDirectories=null,
> enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1,
> writeBatchSize=2097152}
> 2021-12-11 14:25:26,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Checkpoint storage is set to 'filesystem': (checkpoints
> "hdfs://hdfs-namenode:8020/tmp/statefun_checkpoints/myStatefunApp")
> 2021-12-11 14:25:26,712 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Recovering checkpoints from
> ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
> 2021-12-11 14:25:26,724 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Found 1 checkpoints in
> ZooKeeperStateHandleStore{namespace='statefun_zk_recovery/my-statefun-app/checkpoints/'}.
> 2021-12-11 14:25:26,725 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Trying to fetch 1 checkpoints from storage.
> 2021-12-11 14:25:26,725 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] -
> Trying to retrieve checkpoint 2.
> 2021-12-11
[jira] [Updated] (FLINK-25333) Harden UID management when constructing statefun universe
[ https://issues.apache.org/jira/browse/FLINK-25333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-25333: - Issue Type: Bug (was: Improvement) > Harden UID management when constructing statefun universe > - > > Key: FLINK-25333 > URL: https://issues.apache.org/jira/browse/FLINK-25333 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Fix For: statefun-3.2.0 > > > As described in this stack overflow, we do not set UIDs on routers. Because > of how the stream graph is generated, the uids may be non-deterministic. We > should manually set all UIDs and enforce this via configuration. > > https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25333) Harden UID management when constructing statefun universe
Seth Wiesman created FLINK-25333: Summary: Harden UID management when constructing statefun universe Key: FLINK-25333 URL: https://issues.apache.org/jira/browse/FLINK-25333 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Seth Wiesman Assignee: Seth Wiesman Fix For: statefun-3.2.0 As described in this stack overflow, we do not set UIDs on routers. Because of how the stream graph is generated, the uids may be non-deterministic. We should manually set all UIDs and enforce this via configuration. https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-21341) Update state reader to return InputFormat
[
https://issues.apache.org/jira/browse/FLINK-21341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman closed FLINK-21341.
Resolution: Won't Do
> Update state reader to return InputFormat
> -
>
> Key: FLINK-21341
> URL: https://issues.apache.org/jira/browse/FLINK-21341
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor
>Affects Versions: 1.13.0
>Reporter: Seth Wiesman
>Assignee: Jun Qin
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned, stale-assigned
>
> The state processor api on read currently takes an ExecutionEnvironment and
> returns a DataSet. As Flink now supports Bounded DataStream we want to
> support that as well but not have to maintain parallel apis. To that end we
> propose the following.
> Introduce a new `load` method to Savepoint.
>
> {code:java}
> SavepointReader Savepoint#load(String path, StateBackend backend);
> {code}
> SavepointReader will contain the same read methods as ExistingSavepoint but
> they will instead return InputFormat. This way the input format can be
> used with either DataSet, DataStream, or Table API.
>
> The reader methods in ExistingSavepoint should be deprecated.
>
> Additionally, OperatorStateInputFormat and KeyedStateInputFormat should both
> now extend ResultTypeQueryable so users get efficient serializer for their
> reads.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-25238) flink iceberg source reading array types fail with Cast Exception
[ https://issues.apache.org/jira/browse/FLINK-25238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-25238. Resolution: Won't Fix > flink iceberg source reading array types fail with Cast Exception > - > > Key: FLINK-25238 > URL: https://issues.apache.org/jira/browse/FLINK-25238 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.13.2 >Reporter: Praneeth Ramesh >Priority: Major > Attachments: Screen Shot 2021-12-09 at 6.58.56 PM.png, Screen Shot > 2021-12-09 at 7.04.10 PM.png > > > I have a stream with iceberg table as a source. I have few columns of array > types in the table. > I try to read using iceberg connector. > Flink Version : 1.13.2 > Iceberg Flink Version: 0.12.1 > > I see the error as below. > java.lang.ClassCastException: class > org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableArrayData cannot be > cast to class org.apache.flink.table.data.ColumnarArrayData > (org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableArrayData and > org.apache.flink.table.data.ColumnarArrayData are in unnamed module of loader > 'app') > at > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:90) > at > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:47) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:170) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:131) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:48) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411) > at > org.apache.iceberg.flink.source.StreamingReaderOperator.processSplits(StreamingReaderOperator.java:155) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) > at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) > at java.base/java.lang.Thread.run(Thread.java:834) > > Could be same issue as https://issues.apache.org/jira/browse/FLINK-21247 > except it happening for another type. > I see that Iceberg use custom types other than the types from > org.apache.flink.table.data like > org.apache.iceberg.flink.data.FlinkParquetReaders.ReusableArrayData and these > types are not handled in > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer > !Screen Shot 2021-12-09 at 6.58.56 PM.png! > Just to try I changed the above code to handle the iceberg type as a binary > Array and built it locally and used in my application and that worked. > > !Screen Shot 2021-12-09 at 7.04.10 PM.png! > Not sure if this is already handled in some newer versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25238) flink iceberg source reading array types fail with Cast Exception
[ https://issues.apache.org/jira/browse/FLINK-25238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457235#comment-17457235 ] Seth Wiesman commented on FLINK-25238: -- The Iceberg Sink is maintained by the Apache Iceberg community. I am going to close this ticket and recommend you reach out to them for assistance. > flink iceberg source reading array types fail with Cast Exception > - > > Key: FLINK-25238 > URL: https://issues.apache.org/jira/browse/FLINK-25238 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.13.2 >Reporter: Praneeth Ramesh >Priority: Major > Attachments: Screen Shot 2021-12-09 at 6.58.56 PM.png, Screen Shot > 2021-12-09 at 7.04.10 PM.png > > > I have a stream with iceberg table as a source. I have few columns of array > types in the table. > I try to read using iceberg connector. > Flink Version : 1.13.2 > Iceberg Flink Version: 0.12.1 > > I see the error as below. > java.lang.ClassCastException: class > org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableArrayData cannot be > cast to class org.apache.flink.table.data.ColumnarArrayData > (org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableArrayData and > org.apache.flink.table.data.ColumnarArrayData are in unnamed module of loader > 'app') > at > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:90) > at > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer.copy(ArrayDataSerializer.java:47) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:170) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:131) > at > org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:48) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411) > at > org.apache.iceberg.flink.source.StreamingReaderOperator.processSplits(StreamingReaderOperator.java:155) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) > at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) > at java.base/java.lang.Thread.run(Thread.java:834) > > Could be same issue as https://issues.apache.org/jira/browse/FLINK-21247 > except it happening for another type. > I see that Iceberg use custom types other than the types from > org.apache.flink.table.data like > org.apache.iceberg.flink.data.FlinkParquetReaders.ReusableArrayData and these > types are not handled in > org.apache.flink.table.runtime.typeutils.ArrayDataSerializer > !Screen Shot 2021-12-09 at 6.58.56 PM.png! > Just to try I changed the above code to handle the iceberg type as a binary > Array and built it locally and used in my application and that worked. > > !Screen Shot 2021-12-09 at 7.04.10 PM.png! > Not sure if this is already handled in some newer versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25236) Add a mechanism to generate and validate a jobgraph with a checkpoint before submission
[ https://issues.apache.org/jira/browse/FLINK-25236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456700#comment-17456700 ] Seth Wiesman edited comment on FLINK-25236 at 12/9/21, 9:31 PM: Unfortunately, this is not so simple to do. The method you linked to only validates that max parallelism has not been changed, and each operator has a mapping within the checkpoint. State changes are validated lazily when the state descriptor is registered within the runtime context; because state descriptors themselves are registered lazily. This means the only way to validate a DataStream application can restore from a snapshot fully is to attempt the restore. I would typically recommend either QA testing or Blue / Green deployments for these kinds of low latency requirements. Both are readily achieved with Flinks snapshot-based fault-tolerance model. Take a savepoint of your production workload and use that to start your new application in a QA environment. This new application can read from production sources, have internal production state, and so long as sinks are configured dynamically, this new version can write to a non-production output. If this restore works, then deploying to production is guaranteed to succeed. This also allows you the opportunity to validate the output of your changes before deploying them to production. was (Author: sjwiesman): Unfortunately, this is not so simple to do. The method you linked to only validates that max parallelism has not been changed, and each operator has a mapping within the checkpoint. State changes themselves are validated lazily when the state descriptor is registered within the runtime context; because state descriptors themselves are registered lazily. This means the only way to validate a DataStream application can restore from a snapshot fully is to attempt the restore. I would typically recommend either QA testing or Blue / Green deployments for these kinds of low latency requirements. Both are readily achieved with Flinks snapshot-based fault-tolerance model. Take a savepoint of your production workload and use that to start your new application in a QA environment. This new application can read from production sources, have internal production state, and so long as sinks are configured dynamically, this new version can write to a non-production output. If this restore works, then deploying to production is guaranteed to succeed. This also allows you the opportunity to validate the output of your changes before deploying them to production. > Add a mechanism to generate and validate a jobgraph with a checkpoint before > submission > --- > > Key: FLINK-25236 > URL: https://issues.apache.org/jira/browse/FLINK-25236 > Project: Flink > Issue Type: Improvement >Reporter: Ben Augarten >Priority: Major > > I've mostly worked on flink 1.9-1.12, but I believe this is still an issue > today. > > I've worked on a few flink applications now that have struggled to reliably > activate new versions of a currently running job. Sometimes, users make > changes to a job graph that make it so state cannot be restored. Sometimes > users make changes to a job graph that make it unable to be scheduled on a > given cluster (increased parallelism with insufficient task slots on the > cluster). These validations are [performed > here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L120] > > It's not flink's problem that these issues arise, but these issues are only > detected when the JM tries to run the given jobgraph. For exactly once > applications (and other applications where running two job graphs for the > same application is undesirable) there is unneeded downtime when users submit > jobgraphs with breaking changes because users must cancel the old job, submit > the new job to see if it is valid and will activate, and then resubmit the > old job when activation fails. As a user with low-latency requirements, this > change management solution is unfortunate, and there doesn't seem to be > anything technical preventing these validations from happening earlier. > > Suggestion: provide a mechanism for users to (1) create and (2) validate the > new job graph+checkpoint without running it so that they do not need to > cancel a currently running version of the job until they're more sure it will > activate -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25236) Add a mechanism to generate and validate a jobgraph with a checkpoint before submission
[ https://issues.apache.org/jira/browse/FLINK-25236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456700#comment-17456700 ] Seth Wiesman commented on FLINK-25236: -- Unfortunately, this is not so simple to do. The method you linked to only validates that max parallelism has not been changed, and each operator has a mapping within the checkpoint. State changes themselves are validated lazily when the state descriptor is registered within the runtime context; because state descriptors themselves are registered lazily. This means the only way to validate a DataStream application can restore from a snapshot fully is to attempt the restore. I would typically recommend either QA testing or Blue / Green deployments for these kinds of low latency requirements. Both are readily achieved with Flinks snapshot-based fault-tolerance model. Take a savepoint of your production workload and use that to start your new application in a QA environment. This new application can read from production sources, have internal production state, and so long as sinks are configured dynamically, this new version can write to a non-production output. If this restore works, then deploying to production is guaranteed to succeed. This also allows you the opportunity to validate the output of your changes before deploying them to production. > Add a mechanism to generate and validate a jobgraph with a checkpoint before > submission > --- > > Key: FLINK-25236 > URL: https://issues.apache.org/jira/browse/FLINK-25236 > Project: Flink > Issue Type: Improvement >Reporter: Ben Augarten >Priority: Major > > I've mostly worked on flink 1.9-1.12, but I believe this is still an issue > today. > > I've worked on a few flink applications now that have struggled to reliably > activate new versions of a currently running job. Sometimes, users make > changes to a job graph that make it so state cannot be restored. Sometimes > users make changes to a job graph that make it unable to be scheduled on a > given cluster (increased parallelism with insufficient task slots on the > cluster). These validations are [performed > here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L120] > > It's not flink's problem that these issues arise, but these issues are only > detected when the JM tries to run the given jobgraph. For exactly once > applications (and other applications where running two job graphs for the > same application is undesirable) there is unneeded downtime when users submit > jobgraphs with breaking changes because users must cancel the old job, submit > the new job to see if it is valid and will activate, and then resubmit the > old job when activation fails. As a user with low-latency requirements, this > change management solution is unfortunate, and there doesn't seem to be > anything technical preventing these validations from happening earlier. > > Suggestion: provide a mechanism for users to (1) create and (2) validate the > new job graph+checkpoint without running it so that they do not need to > cancel a currently running version of the job until they're more sure it will > activate -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24635) Clean up flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24635. Resolution: Fixed > Clean up flink-examples > --- > > Key: FLINK-24635 > URL: https://issues.apache.org/jira/browse/FLINK-24635 > Project: Flink > Issue Type: Improvement > Components: Examples >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The Flink DataStream examples have a number of deprecation warnings. These > are some of the first things new users look at and we should be showing best > practices. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24829) Rename example directories based on API
[ https://issues.apache.org/jira/browse/FLINK-24829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24829. Resolution: Won't Fix > Rename example directories based on API > --- > > Key: FLINK-24829 > URL: https://issues.apache.org/jira/browse/FLINK-24829 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Rename example directories to better reflect the current state of the > project; de-emphasis of dataset and stream / batch unification of datastream. > > flink-examples-batch --> flink-examples-dataset > flink-examples-streaming --> flink-examples-datastream > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24635) Clean up flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444836#comment-17444836 ] Seth Wiesman commented on FLINK-24635: -- fixed in b00bfaa66db90df7797be6c2003ceaf257e0309f > Clean up flink-examples > --- > > Key: FLINK-24635 > URL: https://issues.apache.org/jira/browse/FLINK-24635 > Project: Flink > Issue Type: Improvement > Components: Examples >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The Flink DataStream examples have a number of deprecation warnings. These > are some of the first things new users look at and we should be showing best > practices. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24831) Upgrade DataStream Window examples
[ https://issues.apache.org/jira/browse/FLINK-24831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24831. Fix Version/s: 1.15.0 Resolution: Fixed > Upgrade DataStream Window examples > -- > > Key: FLINK-24831 > URL: https://issues.apache.org/jira/browse/FLINK-24831 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Fix For: 1.15.0 > > > Upgrade DataStream window examples to not rely on any deprecated APIs and > work for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24833) Prevent use of deprecated APIs in flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444834#comment-17444834 ] Seth Wiesman commented on FLINK-24833: -- fixed in ef5fb7a15cb3964d0d59434798549bfa26adabe9 > Prevent use of deprecated APIs in flink-examples > > > Key: FLINK-24833 > URL: https://issues.apache.org/jira/browse/FLINK-24833 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > We should be able to setup java compiler for examples > to fail on any usage of deprecated APIs. > Something along the lines of: > > maven-compiler-plugin > ... > > > compile > process-sources > > compile > > > -Xlint:deprecation > true > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24833) Prevent use of deprecated APIs in flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24833. Fix Version/s: 1.15.0 Resolution: Fixed > Prevent use of deprecated APIs in flink-examples > > > Key: FLINK-24833 > URL: https://issues.apache.org/jira/browse/FLINK-24833 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Fix For: 1.15.0 > > > We should be able to setup java compiler for examples > to fail on any usage of deprecated APIs. > Something along the lines of: > > maven-compiler-plugin > ... > > > compile > process-sources > > compile > > > -Xlint:deprecation > true > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24831) Upgrade DataStream Window examples
[ https://issues.apache.org/jira/browse/FLINK-24831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444835#comment-17444835 ] Seth Wiesman commented on FLINK-24831: -- fixed in 443b7121373a7f7c4b7a25660a50b06cbee8cb01 > Upgrade DataStream Window examples > -- > > Key: FLINK-24831 > URL: https://issues.apache.org/jira/browse/FLINK-24831 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > Upgrade DataStream window examples to not rely on any deprecated APIs and > work for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24912) Migrate state processor API to DataStream API
Seth Wiesman created FLINK-24912: Summary: Migrate state processor API to DataStream API Key: FLINK-24912 URL: https://issues.apache.org/jira/browse/FLINK-24912 Project: Flink Issue Type: Improvement Components: API / State Processor Affects Versions: 1.15.0 Reporter: Seth Wiesman Assignee: Jun Qin Now that DataStream supports bounded execution, we should investigate migrating the State Processor API off DataSet. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24830) Update DataStream WordCount example
[ https://issues.apache.org/jira/browse/FLINK-24830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24830. Fix Version/s: 1.15.0 Resolution: Fixed > Update DataStream WordCount example > --- > > Key: FLINK-24830 > URL: https://issues.apache.org/jira/browse/FLINK-24830 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Upgrade DataStream wordcount example to not use any deprecated API's and work > for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24831) Upgrade DataStream Window examples
[ https://issues.apache.org/jira/browse/FLINK-24831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24831: Assignee: Seth Wiesman > Upgrade DataStream Window examples > -- > > Key: FLINK-24831 > URL: https://issues.apache.org/jira/browse/FLINK-24831 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > Upgrade DataStream window examples to not rely on any deprecated APIs and > work for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24830) Update DataStream WordCount example
[ https://issues.apache.org/jira/browse/FLINK-24830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444114#comment-17444114 ] Seth Wiesman commented on FLINK-24830: -- fixed in 83a2541475228a4ff9e9a9def4049fb742353549 > Update DataStream WordCount example > --- > > Key: FLINK-24830 > URL: https://issues.apache.org/jira/browse/FLINK-24830 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Upgrade DataStream wordcount example to not use any deprecated API's and work > for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24833) Prevent use of deprecated APIs in flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444113#comment-17444113 ] Seth Wiesman commented on FLINK-24833: -- [~RocMarshal] Thank you for the initiative, but I'm going to take this. I want to enable this check along with fixing the deprecations so no changes can sneak in. I am certainly happy to include you as a reviewer though. [~guoyangze] Will do! thanks for the heads up! > Prevent use of deprecated APIs in flink-examples > > > Key: FLINK-24833 > URL: https://issues.apache.org/jira/browse/FLINK-24833 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > We should be able to setup java compiler for examples > to fail on any usage of deprecated APIs. > Something along the lines of: > > maven-compiler-plugin > ... > > > compile > process-sources > > compile > > > -Xlint:deprecation > true > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24833) Prevent use of deprecated APIs in flink-examples
[ https://issues.apache.org/jira/browse/FLINK-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24833: Assignee: Seth Wiesman > Prevent use of deprecated APIs in flink-examples > > > Key: FLINK-24833 > URL: https://issues.apache.org/jira/browse/FLINK-24833 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > We should be able to setup java compiler for examples > to fail on any usage of deprecated APIs. > Something along the lines of: > > maven-compiler-plugin > ... > > > compile > process-sources > > compile > > > -Xlint:deprecation > true > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24782) Create 1.14 release branch for training repo
[ https://issues.apache.org/jira/browse/FLINK-24782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24782. Resolution: Fixed > Create 1.14 release branch for training repo > > > Key: FLINK-24782 > URL: https://issues.apache.org/jira/browse/FLINK-24782 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training / Exercises >Affects Versions: 1.14.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Create release-1.14 branch for apache/flink-training -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24782) Create 1.14 release branch for training repo
[ https://issues.apache.org/jira/browse/FLINK-24782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441946#comment-17441946 ] Seth Wiesman commented on FLINK-24782: -- fixed in: 0b1c83b16065484200564402bef2ca10ef19cb30 > Create 1.14 release branch for training repo > > > Key: FLINK-24782 > URL: https://issues.apache.org/jira/browse/FLINK-24782 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training / Exercises >Affects Versions: 1.14.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Create release-1.14 branch for apache/flink-training -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-24834) Add typed builders to DefaultRollingPolicy
[ https://issues.apache.org/jira/browse/FLINK-24834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24834. Resolution: Fixed > Add typed builders to DefaultRollingPolicy > -- > > Key: FLINK-24834 > URL: https://issues.apache.org/jira/browse/FLINK-24834 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The default RollingPolicy is configured based on durations and memory sizes, > but these parameters are specified via longs (milliseconds and bytes). We can > provide a better user experience by adding methods that accept > `java.time.Duration` and `org.apache.flink.configuration.MemorySize` instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24834) Add typed builders to DefaultRollingPolicy
[ https://issues.apache.org/jira/browse/FLINK-24834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441880#comment-17441880 ] Seth Wiesman commented on FLINK-24834: -- fixed in 059c00f4f6bbf0b868d4d8e0971e29d0168894f9 > Add typed builders to DefaultRollingPolicy > -- > > Key: FLINK-24834 > URL: https://issues.apache.org/jira/browse/FLINK-24834 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The default RollingPolicy is configured based on durations and memory sizes, > but these parameters are specified via longs (milliseconds and bytes). We can > provide a better user experience by adding methods that accept > `java.time.Duration` and `org.apache.flink.configuration.MemorySize` instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24834) Add typed builders to DefaultRollingPolicy
Seth Wiesman created FLINK-24834: Summary: Add typed builders to DefaultRollingPolicy Key: FLINK-24834 URL: https://issues.apache.org/jira/browse/FLINK-24834 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Affects Versions: 1.15.0 Reporter: Seth Wiesman Assignee: Seth Wiesman The default RollingPolicy is configured based on durations and memory sizes, but these parameters are specified via longs (milliseconds and bytes). We can provide a better user experience by adding methods that accept `java.time.Duration` and `org.apache.flink.configuration.MemorySize` instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24830) Update DataStream WordCount example
[ https://issues.apache.org/jira/browse/FLINK-24830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24830: Assignee: Seth Wiesman > Update DataStream WordCount example > --- > > Key: FLINK-24830 > URL: https://issues.apache.org/jira/browse/FLINK-24830 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > Upgrade DataStream wordcount example to not use any deprecated API's and work > for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24833) Prevent use of deprecated APIs in flink-examples
Seth Wiesman created FLINK-24833: Summary: Prevent use of deprecated APIs in flink-examples Key: FLINK-24833 URL: https://issues.apache.org/jira/browse/FLINK-24833 Project: Flink Issue Type: Sub-task Components: Examples Affects Versions: 1.15.0 Reporter: Seth Wiesman We should be able to setup java compiler for examples to fail on any usage of deprecated APIs. Something along the lines of: maven-compiler-plugin ... compile process-sources compile -Xlint:deprecation true -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24829) Rename example directories based on API
[ https://issues.apache.org/jira/browse/FLINK-24829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24829: Assignee: Seth Wiesman > Rename example directories based on API > --- > > Key: FLINK-24829 > URL: https://issues.apache.org/jira/browse/FLINK-24829 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > Rename example directories to better reflect the current state of the > project; de-emphasis of dataset and stream / batch unification of datastream. > > flink-examples-batch --> flink-examples-dataset > flink-examples-streaming --> flink-examples-datastream > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24831) Upgrade DataStream Window examples
Seth Wiesman created FLINK-24831: Summary: Upgrade DataStream Window examples Key: FLINK-24831 URL: https://issues.apache.org/jira/browse/FLINK-24831 Project: Flink Issue Type: Sub-task Components: Examples Affects Versions: 1.15.0 Reporter: Seth Wiesman Upgrade DataStream window examples to not rely on any deprecated APIs and work for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24830) Update DataStream WordCount example
Seth Wiesman created FLINK-24830: Summary: Update DataStream WordCount example Key: FLINK-24830 URL: https://issues.apache.org/jira/browse/FLINK-24830 Project: Flink Issue Type: Sub-task Components: Examples Affects Versions: 1.15.0 Reporter: Seth Wiesman Upgrade DataStream wordcount example to not use any deprecated API's and work for both batch and streaming workloads. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24829) Rename example directories based on API
[ https://issues.apache.org/jira/browse/FLINK-24829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440563#comment-17440563 ] Seth Wiesman commented on FLINK-24829: -- We might want to look at just dropping dataset examples in the future, but there's no maintenance overhead to keeping them so we'll keep them for now. > Rename example directories based on API > --- > > Key: FLINK-24829 > URL: https://issues.apache.org/jira/browse/FLINK-24829 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Priority: Major > > Rename example directories to better reflect the current state of the > project; de-emphasis of dataset and stream / batch unification of datastream. > > flink-examples-batch --> flink-examples-dataset > flink-examples-streaming --> flink-examples-datastream > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24829) Rename example directories based on API
Seth Wiesman created FLINK-24829: Summary: Rename example directories based on API Key: FLINK-24829 URL: https://issues.apache.org/jira/browse/FLINK-24829 Project: Flink Issue Type: Sub-task Components: Examples Affects Versions: 1.15.0 Reporter: Seth Wiesman Rename example directories to better reflect the current state of the project; de-emphasis of dataset and stream / batch unification of datastream. flink-examples-batch --> flink-examples-dataset flink-examples-streaming --> flink-examples-datastream -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24783) Improve monitoring experience and usability of state backend
[ https://issues.apache.org/jira/browse/FLINK-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17439346#comment-17439346 ] Seth Wiesman commented on FLINK-24783: -- [~yunta] These improvements all look really good! [~nkruber] and I recently gave a training course on RocksDB tuning and found you can forward RocksDB logs to the JVM logger[1]. The JavaDoc claims it can have some negative performance impact, but we found it very useful to have all logs in one place. What do you think about adding this as a configurable option which is disabled by default? [1] https://github.com/ververica/frocksdb/blob/FRocksDB-6.20.3/java/src/main/java/org/rocksdb/Logger.java > Improve monitoring experience and usability of state backend > - > > Key: FLINK-24783 > URL: https://issues.apache.org/jira/browse/FLINK-24783 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > > This ticket targets for improving the monitoring experiences and usability > for HashMap and EmbededRocksDB state backends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-24767) A keyBy following countWindow does not preserve order within the same partition
[
https://issues.apache.org/jira/browse/FLINK-24767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438953#comment-17438953
]
Seth Wiesman edited comment on FLINK-24767 at 11/4/21, 10:19 PM:
-
-When executing a `keyBy` under batch execution, the DataStream API groups
elements by key using a sort[1]. I have not investigated, but my suspicion is
this is not a stable sort and you are seeing the result of that.-
[~dwysakowicz] -I'm guessing this is expected?-
The sort grouping does respect event time timestamps, so I was mistaken. Is it
possible to have a reproducible example that doesn't rely on random numbers?
I'd also be curious to see if the countWindow has anything to do with it. Under
the hood, count windows are based on evictors and that's a weird code path.
[1]https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/execution_mode/#state-backends--state
was (Author: sjwiesman):
When executing a `keyBy` under batch execution, the DataStream API groups
elements by key using a sort[1]. I have not investigated, but my suspicion is
this is not a stable sort and you are seeing the result of that. [~dwysakowicz]
I'm guessing this is expected?
[1]https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/execution_mode/#state-backends--state
> A keyBy following countWindow does not preserve order within the same
> partition
> ---
>
> Key: FLINK-24767
> URL: https://issues.apache.org/jira/browse/FLINK-24767
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.3
>Reporter: Lee Y S
>Priority: Major
>
> I wrote a simple test of the countWindow method (in Kotlin) as below
> {code:java}
> import org.apache.flink.api.common.RuntimeExecutionMode
> import org.apache.flink.api.common.eventtime.WatermarkStrategy
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
> import kotlin.random.Randomobject
> CountWindowTest {
> @JvmStatic
> fun main(args: Array) {
> val env = StreamExecutionEnvironment.getExecutionEnvironment()
> env.setRuntimeMode(RuntimeExecutionMode.BATCH)
> val rand = Random(0)
> val data = (0..1000).map { Pair(rand.nextInt(10), it) }
> env.fromCollection(data).assignTimestampsAndWatermarks(
> WatermarkStrategy.forMonotonousTimestamps>()
> .withTimestampAssigner { e, _ -> e.second.toLong() })
> .keyBy { it.first }
> .countWindow(3L, 1)
> .reduce { a, b -> b }
> .keyBy { it.first }
> .filter { it.first == 5 }
> .print()
> env.execute()
> }
> }
> {code}
> The beginning of the output is as below
> 12> (5, 184)
> 12> (5, 18)
> 12> (5, 29)
> 12> (5, 37)
> 12> (5, 38)
> 12> (5, 112)
> 12> (5, 131)
> The first line (5, 184) is not in order from the rest.
> The problem disappears if I remove the keyBy after the reduce or use stream
> mode instead of batch mode.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24767) A keyBy following countWindow does not preserve order within the same partition
[
https://issues.apache.org/jira/browse/FLINK-24767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438953#comment-17438953
]
Seth Wiesman commented on FLINK-24767:
--
When executing a `keyBy` under batch execution, the DataStream API groups
elements by key using a sort[1]. I have not investigated, but my suspicion is
this is not a stable sort and you are seeing the result of that. [~dwysakowicz]
I'm guessing this is expected?
[1]https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/execution_mode/#state-backends--state
> A keyBy following countWindow does not preserve order within the same
> partition
> ---
>
> Key: FLINK-24767
> URL: https://issues.apache.org/jira/browse/FLINK-24767
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.3
>Reporter: Lee Y S
>Priority: Major
>
> I wrote a simple test of the countWindow method (in Kotlin) as below
> {code:java}
> import org.apache.flink.api.common.RuntimeExecutionMode
> import org.apache.flink.api.common.eventtime.WatermarkStrategy
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
> import kotlin.random.Randomobject
> CountWindowTest {
> @JvmStatic
> fun main(args: Array) {
> val env = StreamExecutionEnvironment.getExecutionEnvironment()
> env.setRuntimeMode(RuntimeExecutionMode.BATCH)
> val rand = Random(0)
> val data = (0..1000).map { Pair(rand.nextInt(10), it) }
> env.fromCollection(data).assignTimestampsAndWatermarks(
> WatermarkStrategy.forMonotonousTimestamps>()
> .withTimestampAssigner { e, _ -> e.second.toLong() })
> .keyBy { it.first }
> .countWindow(3L, 1)
> .reduce { a, b -> b }
> .keyBy { it.first }
> .filter { it.first == 5 }
> .print()
> env.execute()
> }
> }
> {code}
> The beginning of the output is as below
> 12> (5, 184)
> 12> (5, 18)
> 12> (5, 29)
> 12> (5, 37)
> 12> (5, 38)
> 12> (5, 112)
> 12> (5, 131)
> The first line (5, 184) is not in order from the rest.
> The problem disappears if I remove the keyBy after the reduce or use stream
> mode instead of batch mode.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24782) Create 1.14 release branch for training repo
Seth Wiesman created FLINK-24782: Summary: Create 1.14 release branch for training repo Key: FLINK-24782 URL: https://issues.apache.org/jira/browse/FLINK-24782 Project: Flink Issue Type: Improvement Components: Documentation / Training / Exercises Affects Versions: 1.14.0 Reporter: Seth Wiesman Assignee: Seth Wiesman Create release-1.14 branch for apache/flink-training -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24723) Add ServicesResourceTransformer to the quickstart project
[ https://issues.apache.org/jira/browse/FLINK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24723. Resolution: Fixed > Add ServicesResourceTransformer to the quickstart project > - > > Key: FLINK-24723 > URL: https://issues.apache.org/jira/browse/FLINK-24723 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Ali Bahadir Zeybek >Assignee: Ali Bahadir Zeybek >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0 > > > When using quickstart project to write a flink application with Table API, it > requires ServicesResourceTransformer to be added to the maven shade plugin if > there are more than one connectors are used. > This plugin then merges all the META-INF/services files into one rather than > picking the first connector that is added in the pom file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24723) Add ServicesResourceTransformer to the quickstart project
[ https://issues.apache.org/jira/browse/FLINK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437400#comment-17437400 ] Seth Wiesman commented on FLINK-24723: -- fixed in master: ea484357b222d850f8df4d7b6ae8dacb110b19b7 > Add ServicesResourceTransformer to the quickstart project > - > > Key: FLINK-24723 > URL: https://issues.apache.org/jira/browse/FLINK-24723 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Ali Bahadir Zeybek >Assignee: Ali Bahadir Zeybek >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0 > > > When using quickstart project to write a flink application with Table API, it > requires ServicesResourceTransformer to be added to the maven shade plugin if > there are more than one connectors are used. > This plugin then merges all the META-INF/services files into one rather than > picking the first connector that is added in the pom file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24723) Add ServicesResourceTransformer to the quickstart project
[ https://issues.apache.org/jira/browse/FLINK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24723: - Fix Version/s: 1.15.0 > Add ServicesResourceTransformer to the quickstart project > - > > Key: FLINK-24723 > URL: https://issues.apache.org/jira/browse/FLINK-24723 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Ali Bahadir Zeybek >Assignee: Ali Bahadir Zeybek >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0 > > > When using quickstart project to write a flink application with Table API, it > requires ServicesResourceTransformer to be added to the maven shade plugin if > there are more than one connectors are used. > This plugin then merges all the META-INF/services files into one rather than > picking the first connector that is added in the pom file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-24723) Add ServicesResourceTransformer to the quickstart project
[ https://issues.apache.org/jira/browse/FLINK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24723: Assignee: Ali Bahadir Zeybek > Add ServicesResourceTransformer to the quickstart project > - > > Key: FLINK-24723 > URL: https://issues.apache.org/jira/browse/FLINK-24723 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Ali Bahadir Zeybek >Assignee: Ali Bahadir Zeybek >Priority: Minor > Labels: pull-request-available > > When using quickstart project to write a flink application with Table API, it > requires ServicesResourceTransformer to be added to the maven shade plugin if > there are more than one connectors are used. > This plugin then merges all the META-INF/services files into one rather than > picking the first connector that is added in the pom file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24635) Clean up flink-examples
Seth Wiesman created FLINK-24635: Summary: Clean up flink-examples Key: FLINK-24635 URL: https://issues.apache.org/jira/browse/FLINK-24635 Project: Flink Issue Type: Improvement Components: Examples Reporter: Seth Wiesman Assignee: Seth Wiesman Fix For: 1.15.0 The Flink DataStream examples have a number of deprecation warnings. These are some of the first things new users look at and we should be showing best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24516) Modernize Maven Archetype
[ https://issues.apache.org/jira/browse/FLINK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24516. Fix Version/s: 1.15.0 Resolution: Fixed > Modernize Maven Archetype > - > > Key: FLINK-24516 > URL: https://issues.apache.org/jira/browse/FLINK-24516 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The maven archetypes used by many to start their first Flink application do > not reflect the project's current state. > Issues: > * They still bundle the DataSet API and recommend it for batch processing > * The JavaDoc recommends deprecated APIs > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24516) Modernize Maven Archetype
[ https://issues.apache.org/jira/browse/FLINK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430545#comment-17430545 ] Seth Wiesman commented on FLINK-24516: -- fixed in: d8112a721329e7d276fff97ceaccf91fc90391a2 > Modernize Maven Archetype > - > > Key: FLINK-24516 > URL: https://issues.apache.org/jira/browse/FLINK-24516 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The maven archetypes used by many to start their first Flink application do > not reflect the project's current state. > Issues: > * They still bundle the DataSet API and recommend it for batch processing > * The JavaDoc recommends deprecated APIs > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-24593) Create Table API Quickstart
[ https://issues.apache.org/jira/browse/FLINK-24593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24593: Assignee: (was: Seth Wiesman) > Create Table API Quickstart > --- > > Key: FLINK-24593 > URL: https://issues.apache.org/jira/browse/FLINK-24593 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Priority: Major > > Following FLINK-24516 we should add a Table API quickstart. The dependencies > should be structured to easily run queries within the IDE but only package > what should actually be included in the fat jar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24516) Modernize Maven Archetype
[ https://issues.apache.org/jira/browse/FLINK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430541#comment-17430541 ] Seth Wiesman commented on FLINK-24516: -- [~MartijnVisser] I agree, I think that is follow-up work. I'm viewing this ticket as clean-up. I've opened FLINK-24593 to track that. > Modernize Maven Archetype > - > > Key: FLINK-24516 > URL: https://issues.apache.org/jira/browse/FLINK-24516 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The maven archetypes used by many to start their first Flink application do > not reflect the project's current state. > Issues: > * They still bundle the DataSet API and recommend it for batch processing > * The JavaDoc recommends deprecated APIs > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24593) Create Table API Quickstart
Seth Wiesman created FLINK-24593: Summary: Create Table API Quickstart Key: FLINK-24593 URL: https://issues.apache.org/jira/browse/FLINK-24593 Project: Flink Issue Type: Improvement Components: Quickstarts Reporter: Seth Wiesman Assignee: Seth Wiesman Following FLINK-24516 we should add a Table API quickstart. The dependencies should be structured to easily run queries within the IDE but only package what should actually be included in the fat jar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-24516) Modernize Maven Archetype
[ https://issues.apache.org/jira/browse/FLINK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24516: Assignee: Seth Wiesman > Modernize Maven Archetype > - > > Key: FLINK-24516 > URL: https://issues.apache.org/jira/browse/FLINK-24516 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > The maven archetypes used by many to start their first Flink application do > not reflect the project's current state. > Issues: > * They still bundle the DataSet API and recommend it for batch processing > * The JavaDoc recommends deprecated APIs > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24516) Modernize Maven Archetype
[ https://issues.apache.org/jira/browse/FLINK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427811#comment-17427811 ] Seth Wiesman commented on FLINK-24516: -- [~MartijnVisser] maybe you have some ideas about what this should look like? I think it should be a unified `DataStream` skeleton and potentially have a seperate Table api quickstart. > Modernize Maven Archetype > - > > Key: FLINK-24516 > URL: https://issues.apache.org/jira/browse/FLINK-24516 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Seth Wiesman >Priority: Major > > The maven archetypes used by many to start their first Flink application do > not reflect the project's current state. > Issues: > * They still bundle the DataSet API and recommend it for batch processing > * The JavaDoc recommends deprecated APIs > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24516) Modernize Maven Archetype
Seth Wiesman created FLINK-24516: Summary: Modernize Maven Archetype Key: FLINK-24516 URL: https://issues.apache.org/jira/browse/FLINK-24516 Project: Flink Issue Type: Improvement Components: Quickstarts Reporter: Seth Wiesman The maven archetypes used by many to start their first Flink application do not reflect the project's current state. Issues: * They still bundle the DataSet API and recommend it for batch processing * The JavaDoc recommends deprecated APIs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24464. Fix Version/s: statefun-3.2.0 Resolution: Fixed > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.2.0 > > > As reported > [here|https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions], > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425600#comment-17425600 ] Seth Wiesman commented on FLINK-24464: -- resolved in master: 461cdde1b92c8f285e425c461aebff162366fc64 > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > As reported > [here|https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions], > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman reassigned FLINK-24464: Assignee: Seth Wiesman > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > > As reported > [here|https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions], > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24464: - Affects Version/s: statefun-3.1.0 > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Seth Wiesman >Priority: Major > > As report > [here|[https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24464: - Description: As reported [here|https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions], the backlog metric sometimes reports a negative value. This is because the current count is stored in an in-memory variable, while the value used to decrement the count as the backlog clears is stored in Flink state. In the case of a job restart, the in-memory variable is reset to zero (it is a new instance) while the numbers in state are retained. A negative backlog is non-sensical, confusing, and should be prevented. was: As report [here|[https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] the backlog metric sometimes reports a negative value. This is because the current count is stored in an in-memory variable, while the value used to decrement the count as the backlog clears is stored in Flink state. In the case of a job restart, the in-memory variable is reset to zero (it is a new instance) while the numbers in state are retained. A negative backlog is non-sensical, confusing, and should be prevented. > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Seth Wiesman >Priority: Major > > As reported > [here|https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions], > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24464) Metrics sometimes report negative backlog
[ https://issues.apache.org/jira/browse/FLINK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24464: - Description: As report [here|[https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] the backlog metric sometimes reports a negative value. This is because the current count is stored in an in-memory variable, while the value used to decrement the count as the backlog clears is stored in Flink state. In the case of a job restart, the in-memory variable is reset to zero (it is a new instance) while the numbers in state are retained. A negative backlog is non-sensical, confusing, and should be prevented. was: As report [here| [https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] the backlog metric sometimes reports a negative value. This is because the current count is stored in an in-memory variable, while the value used to decrement the count as the backlog clears is stored in Flink state. In the case of a job restart, the in-memory variable is reset to zero (it is a new instance) while the numbers in state are retained. A negative backlog is non-sensical, confusing, and should be prevented. > Metrics sometimes report negative backlog > - > > Key: FLINK-24464 > URL: https://issues.apache.org/jira/browse/FLINK-24464 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Seth Wiesman >Priority: Major > > As report > [here|[https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] > the backlog metric sometimes reports a negative value. This is because the > current count is stored in an in-memory variable, while the value used to > decrement the count as the backlog clears is stored in Flink state. In the > case of a job restart, the in-memory variable is reset to zero (it is a new > instance) while the numbers in state are retained. > > A negative backlog is non-sensical, confusing, and should be prevented. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24464) Metrics sometimes report negative backlog
Seth Wiesman created FLINK-24464: Summary: Metrics sometimes report negative backlog Key: FLINK-24464 URL: https://issues.apache.org/jira/browse/FLINK-24464 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Seth Wiesman As report [here| [https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]] the backlog metric sometimes reports a negative value. This is because the current count is stored in an in-memory variable, while the value used to decrement the count as the backlog clears is stored in Flink state. In the case of a job restart, the in-memory variable is reset to zero (it is a new instance) while the numbers in state are retained. A negative backlog is non-sensical, confusing, and should be prevented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423039#comment-17423039 ] Seth Wiesman edited comment on FLINK-24427 at 9/30/21, 11:01 PM: - Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in dedicated classloaders. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 was (Author: sjwiesman): Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in a dedicated classloader. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423039#comment-17423039 ] Seth Wiesman commented on FLINK-24427: -- Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in a dedicated classloader. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23313) Reintroduce temporal table function documentation
[
https://issues.apache.org/jira/browse/FLINK-23313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman updated FLINK-23313:
-
Fix Version/s: 1.14.0
1.15.0
> Reintroduce temporal table function documentation
> -
>
> Key: FLINK-23313
> URL: https://issues.apache.org/jira/browse/FLINK-23313
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Reporter: Timo Walther
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
>
> FLIP-132 introduced the new {{FOR SYSTEM_TIME AS OF}} and dropped the main
> documentation for temporal table function. This causes a lot of confusion for
> users.
> First, because processing time joins are not supported yet.
> Second, because a primary key might not always be present in the current
> pipeline when using Table API.
> We have not deprecated `createTemporalTableFunction` and the documentation in
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/tableapi/#join-with-temporal-table
> is not enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-23313) Reintroduce temporal table function documentation
[
https://issues.apache.org/jira/browse/FLINK-23313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman closed FLINK-23313.
Resolution: Fixed
> Reintroduce temporal table function documentation
> -
>
> Key: FLINK-23313
> URL: https://issues.apache.org/jira/browse/FLINK-23313
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Reporter: Timo Walther
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.0
>
>
> FLIP-132 introduced the new {{FOR SYSTEM_TIME AS OF}} and dropped the main
> documentation for temporal table function. This causes a lot of confusion for
> users.
> First, because processing time joins are not supported yet.
> Second, because a primary key might not always be present in the current
> pipeline when using Table API.
> We have not deprecated `createTemporalTableFunction` and the documentation in
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/tableapi/#join-with-temporal-table
> is not enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23313) Reintroduce temporal table function documentation
[
https://issues.apache.org/jira/browse/FLINK-23313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422407#comment-17422407
]
Seth Wiesman commented on FLINK-23313:
--
resolved in release-1.14: fc8e269e9415a75fe9d1c5634eef2caaf059e1bc
master: a328e3b2a264a914d9fef4a94f91eecc6623a08c
> Reintroduce temporal table function documentation
> -
>
> Key: FLINK-23313
> URL: https://issues.apache.org/jira/browse/FLINK-23313
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Reporter: Timo Walther
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available
>
> FLIP-132 introduced the new {{FOR SYSTEM_TIME AS OF}} and dropped the main
> documentation for temporal table function. This causes a lot of confusion for
> users.
> First, because processing time joins are not supported yet.
> Second, because a primary key might not always be present in the current
> pipeline when using Table API.
> We have not deprecated `createTemporalTableFunction` and the documentation in
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/tableapi/#join-with-temporal-table
> is not enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24392) Upgrade presto s3 fs implementation to Trino >= 348
[ https://issues.apache.org/jira/browse/FLINK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421630#comment-17421630 ] Seth Wiesman commented on FLINK-24392: -- Potentially as follow up, I think its worth investigating after this upgrade if streaming uploads should be the default. I have a hard time thinking of a case when you wouldn't want this behavior but perhaps someone can come up with a counter example. > Upgrade presto s3 fs implementation to Trino >= 348 > --- > > Key: FLINK-24392 > URL: https://issues.apache.org/jira/browse/FLINK-24392 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.14.0 >Reporter: Robert Metzger >Priority: Major > Fix For: 1.15.0 > > > The Presto s3 filesystem implementation currently shipped with Flink doesn't > support streaming uploads. All data needs to be materialized to a single file > on disk, before it can be uploaded. > This can lead to situations where TaskManagers are running out of disk when > creating a savepoint. > The Hadoop filesystem implementation supports streaming uploads (by using > multipart uploads of smaller (say 100mb) files locally), but it does more API > calls, leading to other issues. > Trino version >= 348 supports streaming uploads. > During experiments, I also noticed that the current presto s3 fs > implementation seems to allocate a lot of memory outside the heap (when > shipping large data, for example when creating a savepoint). On a K8s pod > with a memory limit of 4000Mi, I was not able to run Flink with a > "taskmanager.memory.flink.size" above 3000m. This means that an additional > 1gb of memory needs to be allocated just for the peaks in memory allocation > when presto s3 is taking a savepoint. > As part of this upgrade, we also need to make sure that the new presto / > Trino version is not doing substantially more S3 API calls than the current > version. After switching away from the presto s3 to hadoop s3, I noticed that > disposing an old checkpoint (~100gb) can take up to 15 minutes. The upgraded > presto s3 fs should still be able to quickly dispose state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
[ https://issues.apache.org/jira/browse/FLINK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24373. Resolution: Fixed > Remove BETA Tag from FLIP-27 Source Docs > > > Key: FLINK-24373 > URL: https://issues.apache.org/jira/browse/FLINK-24373 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
[ https://issues.apache.org/jira/browse/FLINK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17420780#comment-17420780 ] Seth Wiesman commented on FLINK-24373: -- resolved in 1.14: 81d676ff72a553d546a2fad25d71d47467e76f95 master: master: 66d3e5cffe8a685b167d3d6807bdc12276e12771 > Remove BETA Tag from FLIP-27 Source Docs > > > Key: FLINK-24373 > URL: https://issues.apache.org/jira/browse/FLINK-24373 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
[ https://issues.apache.org/jira/browse/FLINK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24373: - Fix Version/s: 1.15.0 1.14.0 > Remove BETA Tag from FLIP-27 Source Docs > > > Key: FLINK-24373 > URL: https://issues.apache.org/jira/browse/FLINK-24373 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
Seth Wiesman created FLINK-24373: Summary: Remove BETA Tag from FLIP-27 Source Docs Key: FLINK-24373 URL: https://issues.apache.org/jira/browse/FLINK-24373 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Seth Wiesman Assignee: Seth Wiesman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23976) Add additional availability timing metrics to Job lifecycle events
[ https://issues.apache.org/jira/browse/FLINK-23976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418765#comment-17418765 ] Seth Wiesman commented on FLINK-23976: -- [~thw] interesting. Similar to the linked ticket we'll need to think about how this fits into application mode, in particular jobs that multiple job graphs and which are potentially chained. > Add additional availability timing metrics to Job lifecycle events > -- > > Key: FLINK-23976 > URL: https://issues.apache.org/jira/browse/FLINK-23976 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Flink currently contains a number of availability lifecycle metrics[1] > showing how long it takes to move through different job status'. We propose > adding two additional metrics; startingTime, and cancellingTime (open to > bikeshedding on the metric names). > > * startingTime is the time it takes a job to get to running. > * cancellingTime is the time spent in status CANCELLING > > > [1]https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/metrics/#availability -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24253) Load JdbcDialects via Service Loaders
[ https://issues.apache.org/jira/browse/FLINK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24253: - Affects Version/s: 1.15.0 > Load JdbcDialects via Service Loaders > - > > Key: FLINK-24253 > URL: https://issues.apache.org/jira/browse/FLINK-24253 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The JDBC connector currently supports a hardcoded set of JDBC dialects. To > support other JDBC datastores without adding additional feature complexity > into Flink, we should allow plugging in additional dialects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24253) Load JdbcDialects via Service Loaders
[ https://issues.apache.org/jira/browse/FLINK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-24253: - Fix Version/s: 1.15.0 > Load JdbcDialects via Service Loaders > - > > Key: FLINK-24253 > URL: https://issues.apache.org/jira/browse/FLINK-24253 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The JDBC connector currently supports a hardcoded set of JDBC dialects. To > support other JDBC datastores without adding additional feature complexity > into Flink, we should allow plugging in additional dialects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24253) Load JdbcDialects via Service Loaders
[ https://issues.apache.org/jira/browse/FLINK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24253. Resolution: Fixed > Load JdbcDialects via Service Loaders > - > > Key: FLINK-24253 > URL: https://issues.apache.org/jira/browse/FLINK-24253 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.15.0 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The JDBC connector currently supports a hardcoded set of JDBC dialects. To > support other JDBC datastores without adding additional feature complexity > into Flink, we should allow plugging in additional dialects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24253) Load JdbcDialects via Service Loaders
[ https://issues.apache.org/jira/browse/FLINK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418112#comment-17418112 ] Seth Wiesman commented on FLINK-24253: -- resolved in master: 412b49a0fa4756bc6a663782eec3252411bcbd5e > Load JdbcDialects via Service Loaders > - > > Key: FLINK-24253 > URL: https://issues.apache.org/jira/browse/FLINK-24253 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > > The JDBC connector currently supports a hardcoded set of JDBC dialects. To > support other JDBC datastores without adding additional feature complexity > into Flink, we should allow plugging in additional dialects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24301) use Async transport on all statefun playground examples
[ https://issues.apache.org/jira/browse/FLINK-24301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416329#comment-17416329 ] Seth Wiesman commented on FLINK-24301: -- fixed in dev: dffc59428a91853332a038e9d13970108d2302e9 release-3.1: 33bbe6a6f1f437afda24770958844c74d5d8f81e > use Async transport on all statefun playground examples > --- > > Key: FLINK-24301 > URL: https://issues.apache.org/jira/browse/FLINK-24301 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24301) use Async transport on all statefun playground examples
[ https://issues.apache.org/jira/browse/FLINK-24301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24301. Resolution: Fixed > use Async transport on all statefun playground examples > --- > > Key: FLINK-24301 > URL: https://issues.apache.org/jira/browse/FLINK-24301 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24284) Add a greeter and a showcase for the JavaScript SDK
[ https://issues.apache.org/jira/browse/FLINK-24284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-24284. Resolution: Done > Add a greeter and a showcase for the JavaScript SDK > --- > > Key: FLINK-24284 > URL: https://issues.apache.org/jira/browse/FLINK-24284 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > > We need to add a greeter and a showcase for the Javascript SDK to the > playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24284) Add a greeter and a showcase for the JavaScript SDK
[ https://issues.apache.org/jira/browse/FLINK-24284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416327#comment-17416327 ] Seth Wiesman commented on FLINK-24284: -- fixed in dev: 3b52dfc554540c03dbd4b2acfa4440ceb8914e38 > Add a greeter and a showcase for the JavaScript SDK > --- > > Key: FLINK-24284 > URL: https://issues.apache.org/jira/browse/FLINK-24284 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > > We need to add a greeter and a showcase for the Javascript SDK to the > playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
