[jira] [Updated] (FLINK-22977) jobmanager.rpc.addresss does not support hostname with an underscore
[
https://issues.apache.org/jira/browse/FLINK-22977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikesh Hajari updated FLINK-22977:
--
Component/s: Runtime / Network
Affects Version/s: 1.13.0
> jobmanager.rpc.addresss does not support hostname with an underscore
>
>
> Key: FLINK-22977
> URL: https://issues.apache.org/jira/browse/FLINK-22977
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.0
>Reporter: Nikesh Hajari
>Priority: Minor
>
> Using Docker to setup a Flink job manager - when the jobmanager.rpc.address
> has an underscore in the hostname - the error message below is received. I
> have included a working and non-working sample below by only changing one
> "dash" to an "underscore".
> 2021-06-14 02:25:48,485 WARN
> org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever [] -
> Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@lsp-dev-apache_flink-job-manager:6123/user/rpc/dispatcher_*.
> From Web UI: \{"errors":["Service temporarily unavailable due to an ongoing
> leader election. Please refresh."]}
>
> *This works - RPC without underscore*
> lsp-dev-apache-flink-job-manager:
> container_name: lsp-dev-apache-flink-job-manager
> networks:
> - flink-network
> ports:
> - '8081:8081'
> command: jobmanager
> environment:
> - |
> FLINK_PROPERTIES=
> jobmanager.rpc.address: lsp-dev-apache-flink-job-manager
>
> *This does not work - RPC with underscore*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22977) jobmanager.rpc.addresss does not support hostname with an underscore
Nikesh Hajari created FLINK-22977:
-
Summary: jobmanager.rpc.addresss does not support hostname with an
underscore
Key: FLINK-22977
URL: https://issues.apache.org/jira/browse/FLINK-22977
Project: Flink
Issue Type: Bug
Reporter: Nikesh Hajari
Using Docker to setup a Flink job manager - when the jobmanager.rpc.address has
an underscore in the hostname - the error message below is received. I have
included a working and non-working sample below by only changing one "dash" to
an "underscore".
2021-06-14 02:25:48,485 WARN
org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever [] -
Error while retrieving the leader gateway. Retrying to connect to
akka.tcp://flink@lsp-dev-apache_flink-job-manager:6123/user/rpc/dispatcher_*.
>From Web UI: \{"errors":["Service temporarily unavailable due to an ongoing
>leader election. Please refresh."]}
*This works - RPC without underscore*
lsp-dev-apache-flink-job-manager:
container_name: lsp-dev-apache-flink-job-manager
networks:
- flink-network
ports:
- '8081:8081'
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: lsp-dev-apache-flink-job-manager
*This does not work - RPC with underscore*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22483) Recover checkpoints when JobMaster gains leadership
[
https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362667#comment-17362667
]
Eduardo Winpenny Tejedor commented on FLINK-22483:
--
Hi [~trohrmann] , sorry for the delay but it's taken me a while to get some
spare time and to crack this one out. I'll give you an update. I've got a
couple of ideas as to how this could be done. Either move the handling of the
{{DefaultCompletedCheckpointStore}} to the {{DefaultJobMasterServiceFactory}}
as you suggested or move the recovering of checkpoints to a function close to
where it already is but only call it from {{DefaultJobMasterServiceFactory}}.
I'd like to know a couple more details before committing to either solution.
The stack trace presented in the ticket is indeed a call to
{{CompletedCheckpointStore::recover}} but it seems to me to already happen
[asynchronously|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultScheduler.java#L259].
I think the call that needs lifting and shifting is [this
one|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionGraphFactory.java#L134]
which does get invoked from the {{JobMaster}} constructor. Am I on the right
track here?
The call to {{CompletedCheckpointStore::recover}} lies deep in many layers of
calls, how many wrapping layers need to be shifted to the
{{DefaultJobMasterServiceFactory}}? Moving only the call to
{{CompletedCheckpointStore::recover}} wouldn't make sense without moving [this
whole block
too|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionGraphFactory.java#L129].
{code:java}
final CheckpointCoordinator checkpointCoordinator =
newExecutionGraph.getCheckpointCoordinator();
if (checkpointCoordinator != null) {
// check whether we find a valid checkpoint
if (!checkpointCoordinator.restoreInitialCheckpointIfPresent(
new HashSet<>(newExecutionGraph.getAllVertices().values( {
// check whether we can restore from a savepoint
tryRestoreExecutionGraphFromSavepoint(
newExecutionGraph, jobGraph.getSavepointRestoreSettings());
}
}
{code}
Again, let me know if I'm not on the right track.
Finally, where would the code need to be shifted exactly for it to not bother
the intended execution of the program? Before the instantiation of the
{{JobMaster}}? After {{JobMaster::start}}? Anywhere in between would still be
blocking wouldn't it? Possibly use a separate {{Executor}} for that?
Hope to get cracking on this one soon!
> Recover checkpoints when JobMaster gains leadership
> ---
>
> Key: FLINK-22483
> URL: https://issues.apache.org/jira/browse/FLINK-22483
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: stale-critical
> Fix For: 1.14.0
>
>
> Recovering checkpoints (from the CompletedCheckpointStore) is a potentially
> long-lasting/blocking operation, for example if the file system
> implementation is retrying to connect to a unavailable storage backend.
> Currently, we are calling the CompletedCheckpointStore.recover() method from
> the main thread of the JobManager, making it unresponsive to any RPC call
> while the recover method is blocked:
> {code}
> 2021-04-02 20:33:31,384 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job XXX
> switched from state RUNNING to RESTARTING.
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to
> minio.minio.svc:9000 [minio.minio.svc/] failed: Connection refused
> (Connection refused)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[?:?]
> at
>
[jira] [Updated] (FLINK-8606) Clean up Table API for Java users
[
https://issues.apache.org/jira/browse/FLINK-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8606:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Clean up Table API for Java users
> -
>
> Key: FLINK-8606
> URL: https://issues.apache.org/jira/browse/FLINK-8606
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Java users are able to see many variables and methods that are declared
> {{private[flink]}} or even {{protected}} in Scala. Classes such as
> {{TableEnvironment}} look very messy from the outside in Java. We should
> clean up the API and remove {{private[flink]}} or {{protected}} where ever
> possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15077) Support Semi/Anti LookupJoin in Blink planner
[
https://issues.apache.org/jira/browse/FLINK-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15077:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Support Semi/Anti LookupJoin in Blink planner
> -
>
> Key: FLINK-15077
> URL: https://issues.apache.org/jira/browse/FLINK-15077
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Reporter: Jing Zhang
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> Support the following sql in Blink planner:
> {code:sql}
> SELECT T.id, T.len, T.content FROM T WHERE T.id IN (
> SELECT id FROM csvDim FOR SYSTEM_TIME AS OF PROCTIME() AS D)
> {code}
> {code:sql}
> SELECT T.id, T.len, T.content FROM T WHERE EXISTS (
> SELECT * FROM csvDim FOR SYSTEM_TIME AS OF PROCTIME() AS D WHERE T.id =
> D.id)
> {code}
> {code:sql}
> SELECT T.id, T.len, T.content FROM T WHERE NOT EXISTS (
> SELECT * FROM csvDim FOR SYSTEM_TIME AS OF PROCTIME() AS D WHERE T.id =
> D.id)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8518) Support DOW, EPOCH, DECADE for EXTRACT
[ https://issues.apache.org/jira/browse/FLINK-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8518: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support DOW, EPOCH, DECADE for EXTRACT > -- > > Key: FLINK-8518 > URL: https://issues.apache.org/jira/browse/FLINK-8518 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > We upgraded Calcite to version 1.15 in FLINK-7934. The EXTRACT method > supports more conversion targets. The targets DOW, EPOCH, DECADE should be > implemented and tested for different datatypes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22949) java.io.InvalidClassException With Flink Kafka Beam
[
https://issues.apache.org/jira/browse/FLINK-22949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22949:
---
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as a
Blocker but is unassigned and neither itself nor its Sub-Tasks have been
updated for 1 days. I have gone ahead and marked it "stale-blocker". If this
ticket is a Blocker, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> java.io.InvalidClassException With Flink Kafka Beam
> ---
>
> Key: FLINK-22949
> URL: https://issues.apache.org/jira/browse/FLINK-22949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Ravikiran Borse
>Priority: Blocker
> Labels: stale-blocker
>
> Beam: 2.30.0
> Flink: 1.12.0
> Kafka: 2.6.0
> ERROR:root:java.io.InvalidClassException:
> org.apache.flink.streaming.api.graph.StreamConfig$NetworkInputConfig; local
> class incompatible: stream classdesc serialVersionUID = 3698633776553163849,
> local class serialVersionUID = -3137689219135046939
>
> In Flink Logs
> KafkaIO.Read.ReadFromKafkaViaSDF/{ParDo(GenerateKafkaSourceDescriptor),
> KafkaIO.ReadSourceDescriptors} (1/1)#0 (b0c31371874208adb0ccaff85b971883)
> switched from RUNNING to FAILED.
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not
> deserialize inputs
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getInputs(StreamConfig.java:265)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getTypeSerializerIn(StreamConfig.java:280)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getTypeSerializerIn1(StreamConfig.java:271)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.wrapOperatorIntoOutput(OperatorChain.java:639)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:591)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:526)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:164)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:485)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.graph.StreamConfig$NetworkInputConfig; local
> class incompatible: stream classdesc serialVersionUID = 3698633776553163849,
> local class serialVersionUID = -3137689219135046939
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8438) Ensure that implicit Scala TypeInformation works in Eclipse IDE
[ https://issues.apache.org/jira/browse/FLINK-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8438: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Ensure that implicit Scala TypeInformation works in Eclipse IDE > --- > > Key: FLINK-8438 > URL: https://issues.apache.org/jira/browse/FLINK-8438 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > It seems that Scala macros do not work properly in the Eclipse IDE. We need > to investigate this further and maybe need to update the Scala macros. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8712) Cannot execute job with multiple slot sharing groups on LocalExecutor
[
https://issues.apache.org/jira/browse/FLINK-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8712:
--
Labels: auto-deprioritized-critical stale-major (was:
auto-deprioritized-critical)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Cannot execute job with multiple slot sharing groups on LocalExecutor
> -
>
> Key: FLINK-8712
> URL: https://issues.apache.org/jira/browse/FLINK-8712
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major
>
> Currently, it is not possible to run a job with multiple slot sharing groups
> on the LocalExecutor. The problem is that we determine the number of required
> slots simply by looking for the max parallelism of the job but do not
> consider slot sharing groups.
>
> {code:java}
> // set up the streaming execution environment
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> final DataStreamSource input = env.addSource(new InfinitySource());
> final SingleOutputStreamOperator different = input.map(new
> MapFunction() {
>@Override
>public Integer map(Integer integer) throws Exception {
> return integer;
>}
> }).slotSharingGroup("Different");
> different.print();
> // execute program
> env.execute("Flink Streaming Java API Skeleton");{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8474) Add documentation for HBaseTableSource
[
https://issues.apache.org/jira/browse/FLINK-8474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8474:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add documentation for HBaseTableSource
> --
>
> Key: FLINK-8474
> URL: https://issues.apache.org/jira/browse/FLINK-8474
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Table SQL / Ecosystem
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Fabian Hueske
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> The {{HBaseTableSource}} is not documented in the [Table Source and Sinks
> documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/sourceSinks.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8869) Kafka restore from checkpoint without react to the new add partition to kafka server
[ https://issues.apache.org/jira/browse/FLINK-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8869: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Kafka restore from checkpoint without react to the new add partition to kafka > server > > > Key: FLINK-8869 > URL: https://issues.apache.org/jira/browse/FLINK-8869 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.3.2 >Reporter: Aitozi >Priority: Major > Labels: auto-unassigned, stale-major > > When job restore from a savepoint and the kafka server has added serval > partition , it doesnt consume data or produce data from/to the new partition -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-5601) Window operator does not checkpoint watermarks
[
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-5601:
--
Labels: auto-unassigned pull-request-available stale-critical (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.11.0
>Reporter: Ufuk Celebi
>Priority: Critical
> Labels: auto-unassigned, pull-request-available, stale-critical
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and
> rescaling and ran into failures, because the data generator required
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not
> dropped, because the watermarks needed to be re-established after restore
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not
> checkpointing watermarks?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8435) Adding BloomFilter/HyperLogLog state as Managed once
[ https://issues.apache.org/jira/browse/FLINK-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8435: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Adding BloomFilter/HyperLogLog state as Managed once > > > Key: FLINK-8435 > URL: https://issues.apache.org/jira/browse/FLINK-8435 > Project: Flink > Issue Type: New Feature > Components: Runtime / Task >Reporter: Moein Hosseini >Priority: Major > Labels: auto-unassigned, stale-major > > Many times doing something approximately is enough for users (Such as > counting). I think we should implement bloom filter and HyperLogLog as state > for keyed state. By this users are able to filter or process many things > approximately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8566) Replace retract/insert of same record for state retention timer resets
[
https://issues.apache.org/jira/browse/FLINK-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8566:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Replace retract/insert of same record for state retention timer resets
> --
>
> Key: FLINK-8566
> URL: https://issues.apache.org/jira/browse/FLINK-8566
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.5.0
>Reporter: Fabian Hueske
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Currently a simple query like {{SELECT DISTINCT a, b, c FROM tableX}} is
> translated into a plan that generates a retraction stream. However, one would
> assume that an append stream should be possible as well. In fact, the plan
> doesn't produce actual updates.
> Internally, the {{DISTINCT}} is translated into a {{GROUP BY}} with all
> distinct fields being keys and no aggregation functions. The corresponding
> operator produces updates, because aggregation function might update their
> results as new records are received. So we could just implement a dedicated
> operator for {{DISTINCT}}. However, this would not work if a user configures
> a state retention time. In this case, we emit retraction/insert messages for
> the same (distinct) record whenever a new row is received to reset the state
> clean-up timers of the downstream operators.
> One way to solve this issue to implement a dedicated mechanism to update
> state clean-up timers for unchanged records instead of sending out
> retraction/insert messages with identical records. This mechanism would just
> be used to reset the timers and could also be used for append streams. For
> example, we could replace the boolean flag in CRow with a byte that can take
> more than two values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8387) Do not allow filter(false)
[
https://issues.apache.org/jira/browse/FLINK-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8387:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Do not allow filter(false)
> --
>
> Key: FLINK-8387
> URL: https://issues.apache.org/jira/browse/FLINK-8387
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Code like the following looks correct but actually produces unintended side
> effects:
> {code}
> left.join(right)
> .where('a == 'd)
> .select('a)
> {code}
> {{==}} is not converted into an expression but directly evaluated into a
> boolean expression and implicitly converted into an {{false}} literal.
> Instead of an exception this leads to an always rejecting table program. We
> should not allow filters with {{false}} literals.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9171) Flink HCatolog integration
[ https://issues.apache.org/jira/browse/FLINK-9171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9171: -- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Flink HCatolog integration > --- > > Key: FLINK-9171 > URL: https://issues.apache.org/jira/browse/FLINK-9171 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Shuyi Chen >Priority: Major > Labels: stale-major > > This is a parent task for all HCatalog related integration in Flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9116) Introduce getAll and removeAll for MapState
[
https://issues.apache.org/jira/browse/FLINK-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9116:
--
Labels: auto-unassigned pull-request-available stale-major (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Introduce getAll and removeAll for MapState
> ---
>
> Key: FLINK-9116
> URL: https://issues.apache.org/jira/browse/FLINK-9116
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have supported {{putAll(List)}} in {{MapState}}, I think we should also
> support {{getAll(Iterable)}} and {{removeAll(Iterable)}} in {{MapState}}, it
> can be convenient in some scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9036) Add default value via suppliers
[
https://issues.apache.org/jira/browse/FLINK-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9036:
--
Labels: auto-unassigned pull-request-available stale-major (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add default value via suppliers
> ---
>
> Key: FLINK-9036
> URL: https://issues.apache.org/jira/browse/FLINK-9036
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Stephan Ewen
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Earlier versions had a default value in {{ValueState}}. We dropped that,
> because the value would have to be duplicated on each access, to be safe
> against side effects when using mutable types.
> For convenience, we should re-add the feature, but using a supplier/factory
> function to create the default value on access.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-8920) Split code generated input unboxing code
[ https://issues.apache.org/jira/browse/FLINK-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8920: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Split code generated input unboxing code > > > Key: FLINK-8920 > URL: https://issues.apache.org/jira/browse/FLINK-8920 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > In FLINK-8274 we introduced the possibility of splitting the generated code > into multiple methods in order to exceed the JVMs maximum method size (see > also https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.9). > At the moment we only split methods by fields, however, this is not enough in > all case. We should also split input unboxing code if necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9193) Deprecate non-well-defined output methods on DataStream
[
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9193:
--
Labels: auto-unassigned pull-request-available stale-major (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-major
>
> Some output methods on {{DataStream}} that write text to files are not safe
> to use in a streaming program as they have no consistency guarantees. They
> are:
> - {{writeAsText()}}
> - {{writeAsCsv()}}
> - {{writeToSocket()}}
> - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9175) Flink CEP with Checkpointing alway failed
[ https://issues.apache.org/jira/browse/FLINK-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9175: -- Labels: auto-deprioritized-critical stale-major (was: auto-deprioritized-critical) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Flink CEP with Checkpointing alway failed > - > > Key: FLINK-9175 > URL: https://issues.apache.org/jira/browse/FLINK-9175 > Project: Flink > Issue Type: Improvement > Components: Library / CEP, Runtime / Checkpointing >Affects Versions: 1.4.1 > Environment: * Checkpoint Interval: 1min > * Checkpoint Timeout: 2min > * Checkpoint Pause: 5s > * Checkpoint Concurrent: 1 > * Checkpoint Mode: EXACTLY_ONCE > * AllowedLateness: 100s > * CEP within time: 30s > * Kafka QPS：10,000 > * Source Parallelism: 16 > >Reporter: Godfrey He >Priority: Major > Labels: auto-deprioritized-critical, stale-major > Attachments: CEPCal.java, checkpoint failed.png, dataStream.png > > > I used RocksDBStateBackend to checkpoint for my job, and it always failed for > timeout. But when I closed CEP, only keeped the source operator, which was > working fine. And FsStateBackend was also finished checkpoint quickly without > timeout. > !dataStream.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
[ https://issues.apache.org/jira/browse/FLINK-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9114: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments > - > > Key: FLINK-9114 > URL: https://issues.apache.org/jira/browse/FLINK-9114 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / State Backends >Reporter: Jacob Park >Priority: Major > Labels: auto-unassigned, stale-major > > When you operate a Flink application that uses externalized checkpoints to > S3, it becomes difficult to determine which checkpoint is the latest to > recover from. Because S3 provides read-after-write consistency only for PUTS, > listing a S3 path is not guaranteed to be consistent, so we do not know what > checkpoint to recover from. > The goal of this improvement is to allow users to provide a custom > CheckpointRecoveryFactory for non-HA deployments such that we can use this > feature to fail checkpoints if we cannot guarantee we will know where a > checkpoint will be in S3, and co-publish checkpoint metadata to a strongly > consistent data store. > I propose the following changes: > # Modify AbstractNonHaServices and StandaloneHaServices to accept an > Executor for the custom CheckpointRecoveryFactory. > # Create a CheckpointRecoveryFactoryLoader to provide the custom > CheckpointRecoveryFactory from configurations. > # Add new configurations for this feature. > We considered the pluggable StateBackend and the potentially pluggable > HighAvailabilityServices. These were too convoluted to solve our problem, so > we would like to implement a custom CheckpointRecoveryFactory mechanism. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14875) Blink planner should pass the right ExecutionConfig to the creation of serializer
[
https://issues.apache.org/jira/browse/FLINK-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14875:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Blink planner should pass the right ExecutionConfig to the creation of
> serializer
> -
>
> Key: FLINK-14875
> URL: https://issues.apache.org/jira/browse/FLINK-14875
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Jing Zhang
>Priority: Minor
> Labels: auto-deprioritized-major
> Attachments: ImmutableCollectionKryoDeserializerITCase.java
>
>
> If source contains data which has immutable collection, the exception will be
> thrown out:
> {code:java}
> Caused by: com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> logTags_ (com.aliyun.openservices.log.common.Logs$LogGroup)
> mLogGroup (com.aliyun.openservices.log.common.LogGroupData)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.deserialize(ListSerializer.java:138)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.deserialize(ListSerializer.java:47)
> at
> org.apache.flink.util.InstantiationUtil.deserializeFromByteArray(InstantiationUtil.java:463)
> at
> org.apache.flink.table.dataformat.BinaryRow.getGeneric(BinaryRow.java:440)
> at BaseRowSerializerProjection$52.apply(Unknown Source)
> at BaseRowSerializerProjection$52.apply(Unknown Source)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.baseRowToBinary(BaseRowSerializer.java:250)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.serializeToPages(BaseRowSerializer.java:285)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.serializeToPages(BaseRowSerializer.java:55)
> at
> org.apache.flink.table.runtime.sort.BinaryInMemorySortBuffer.write(BinaryInMemorySortBuffer.java:190)
> at
> org.apache.flink.table.runtime.sort.BinaryExternalSorter.write(BinaryExternalSorter.java:540)
> ... 10 more
> Caused by: java.lang.UnsupportedOperationException
> at
> java.util.Collections$UnmodifiableCollection.add(Collections.java:1055)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> ... 27 more
> {code}
> the exception could also appears in a simple ITCase in attachments.
> I find similar problems in [How to set Unmodifiable collection serializer of
> Kryo in Spark
> code|https://stackoverflow.com/questions/46818293/how-to-set-unmodifiable-collection-serializer-of-kryo-in-spark-code],
> is there any way to set unmodifiable collection serializer of Kryo in at
> present?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14986) Support to get detailed Kubernetes cluster description
[ https://issues.apache.org/jira/browse/FLINK-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14986: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support to get detailed Kubernetes cluster description > -- > > Key: FLINK-14986 > URL: https://issues.apache.org/jira/browse/FLINK-14986 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > Currently Flink supports get yarn cluster description by > `YarnClusterDescriptor#getClusterDescription`. We should support the same > behavior in Kubernetes cluster. > Usually the cluster description contains the "total resources, available > resources, etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14763) flink sql cep parallelism error
[
https://issues.apache.org/jira/browse/FLINK-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14763:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> flink sql cep parallelism error
> -
>
> Key: FLINK-14763
> URL: https://issues.apache.org/jira/browse/FLINK-14763
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0
> Environment: flink on yarn
> flink 1.10
> hadoop 3.0
> kafka 2.2.0
>Reporter: richt richt
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> when i commit a cep sql with sql-client use parallelism large than 1 , it
> print error as blow
> {code:java}
> //代码占位符
> java.lang.IllegalArgumentException: KeyGroupRange{startKeyGroup=0,
> endKeyGroup=15} does not contain key group
> 16java.lang.IllegalArgumentException: KeyGroupRange{startKeyGroup=0,
> endKeyGroup=15} does not contain key group 16 at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:161) at
> org.apache.flink.runtime.state.heap.HeapPriorityQueueSet.globalKeyGroupToLocalIndex(HeapPriorityQueueSet.java:158)
> at
> org.apache.flink.runtime.state.heap.HeapPriorityQueueSet.getDedupMapForKeyGroup(HeapPriorityQueueSet.java:147)
> at
> org.apache.flink.runtime.state.heap.HeapPriorityQueueSet.getDedupMapForElement(HeapPriorityQueueSet.java:154)
> at
> org.apache.flink.runtime.state.heap.HeapPriorityQueueSet.add(HeapPriorityQueueSet.java:121)
> at
> org.apache.flink.runtime.state.heap.HeapPriorityQueueSet.add(HeapPriorityQueueSet.java:49)
> at
> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.registerEventTimeTimer(InternalTimerServiceImpl.java:216)
> at
> org.apache.flink.cep.operator.CepOperator.saveRegisterWatermarkTimer(CepOperator.java:285)
> at
> org.apache.flink.cep.operator.CepOperator.processElement(CepOperator.java:266)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:163)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:149)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:282)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.execution.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:151)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:430)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:696) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:521) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> it seems allocate some key to the wrong taskmanager
>
> the yaml is
> {code:java}
> //代码占位符
> execution:
> planner: blink
> type: streaming
> parallelism: 32
>
> - name: Ticker
> type: source-table
> update-mode: append
> connector:
> sink-partitioner: round-robin
> sink-partitioner-class:
> org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner
> property-version: 1
> type: kafka
> version: universal
> topic: test_part
> startup-mode: earliest-offset
> properties:
> - key: bootstrap.servers
> value: localhost:9092
> - key: group.id
> value: testGroup
> format:
> property-version: 1
> type: json
> derive-schema: true
> schema:
> - name: symbol
> type: VARCHAR
> - name: rtime
> type: TIMESTAMP
> rowtime:
> timestamps:
> type: from-field
> from: rowtime
> watermarks:
> type: periodic-bounded
> delay: 2000
> - name: price
> type: BIGINT
> - name: tax
> type: BIGINT
> {code}
> and the query is from the demo:
> {code:java}
> SELECT *
> FROM Ticker
> MATCH_RECOGNIZE(
> PARTITION BY symbol
> ORDER BY rtime
> MEASURES
> C.price AS lastPrice
> ONE ROW PER MATCH
> AFTER MATCH SKIP PAST LAST ROW
> PATTERN (A B* C)
> DEFINE
> A AS A.price > 10,
> B AS B.price < 15,
> C AS C.price > 12
> )
> {code}
> the data :
> {code:java}
>symbol rtime price
> tax
> ACME 2011-11-11T10:00
[jira] [Updated] (FLINK-14996) Missing java doc in org.apache.flink.runtime.operators.TempBarrier
[
https://issues.apache.org/jira/browse/FLINK-14996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14996:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Missing java doc in org.apache.flink.runtime.operators.TempBarrier
> --
>
> Key: FLINK-14996
> URL: https://issues.apache.org/jira/browse/FLINK-14996
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.9.1
>Reporter: Jiayi Liao
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> Empty java doc in {{org.apache.flink.runtime.operators.TempBarrier}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14879) Support 'DESCRIBE DATABSE databaseName' in TableEnvironment and SQL Client
[ https://issues.apache.org/jira/browse/FLINK-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14879: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support 'DESCRIBE DATABSE databaseName' in TableEnvironment and SQL Client > -- > > Key: FLINK-14879 > URL: https://issues.apache.org/jira/browse/FLINK-14879 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Client >Reporter: Terry Wang >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > 1. showDatabasesStatement: (has been supported) > SHOW DATABASES > 2. descDatabaseStatement: > DESCRIBE DATABASE [ EXTENDED] [ catalogName.] dataBasesName > See > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14867) Move TextInputFormat & TextOutputFormat to flink-core
[
https://issues.apache.org/jira/browse/FLINK-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14867:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Move TextInputFormat & TextOutputFormat to flink-core
> -
>
> Key: FLINK-14867
> URL: https://issues.apache.org/jira/browse/FLINK-14867
> Project: Flink
> Issue Type: Improvement
> Components: API / DataSet, API / DataStream
>Reporter: Zili Chen
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is one step to decouple the dependency from flink-streaming-java to
> flink-java. We already have a package {{o.a.f.core.io}} for these formats.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15012) Checkpoint directory not cleaned up
[
https://issues.apache.org/jira/browse/FLINK-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15012:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Checkpoint directory not cleaned up
> ---
>
> Key: FLINK-15012
> URL: https://issues.apache.org/jira/browse/FLINK-15012
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.1
>Reporter: Nico Kruber
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I started a Flink cluster with 2 TMs using {{start-cluster.sh}} and the
> following config (in addition to the default {{flink-conf.yaml}})
> {code:java}
> state.checkpoints.dir: file:///path/to/checkpoints/
> state.backend: rocksdb {code}
> After submitting a jobwith checkpoints enabled (every 5s), checkpoints show
> up, e.g.
> {code:java}
> bb969f842bbc0ecc3b41b7fbe23b047b/
> ├── chk-2
> │ ├── 238969e1-6949-4b12-98e7-1411c186527c
> │ ├── 2702b226-9cfc-4327-979d-e5508ab2e3d5
> │ ├── 4c51cb24-6f71-4d20-9d4c-65ed6e826949
> │ ├── e706d574-c5b2-467a-8640-1885ca252e80
> │ └── _metadata
> ├── shared
> └── taskowned {code}
> If I shut down the cluster via {{stop-cluster.sh}}, these files will remain
> on disk and not be cleaned up.
> In contrast, if I cancel the job, at least {{chk-2}} will be deleted, but
> still leaving the (empty) directories.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15017: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22561) Restore checkpoint from another cluster
[ https://issues.apache.org/jira/browse/FLINK-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22561: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Restore checkpoint from another cluster > --- > > Key: FLINK-22561 > URL: https://issues.apache.org/jira/browse/FLINK-22561 > Project: Flink > Issue Type: Improvement > Environment: flink-1.12.1 > hadoop 3.0.0-cdh6.3.2 >Reporter: zhang haoyan >Priority: Minor > Labels: auto-deprioritized-major > > Hi, we have two hadoop clusters and run some flink streaming job on it > When we migrate some jobs from one cluster to another, we start the job just > like: > flink - s hdfs://clusterA/xx/xxx/chk-35 xxx.jar, but we will get > an error like 'java.io.FileNotFoundException: File does not exist: > /xxx/flink-checkpoints/cb46fc9afab25ced935e751dab49bee2/shared/329f6b79-292a-4d5d-b47b-a1cb63d02fe5', > > it seems can't read shared path from another cluster > We don't want copy the share path manually every time and switch jobs between > this two cluster will be very often. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15018) Add event time page to Concepts
[ https://issues.apache.org/jira/browse/FLINK-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15018: --- Labels: auto-deprioritized-major auto-unassigned pull-request-available (was: auto-unassigned pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add event time page to Concepts > --- > > Key: FLINK-15018 > URL: https://issues.apache.org/jira/browse/FLINK-15018 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.10.4, 1.11.4, 1.14.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We should disentangle the conceptual overview of event time from the > DataStream documentation so that it can be used by DataStream and SQL users. > This has the added benefit of making the page more stable as it will not have > to be rewritten or touched every time an API is changed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22427) Support DEFAULT value syntax to simplify data clean process
[
https://issues.apache.org/jira/browse/FLINK-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22427:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Support DEFAULT value syntax to simplify data clean process
> ---
>
> Key: FLINK-22427
> URL: https://issues.apache.org/jira/browse/FLINK-22427
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Environment: Flink version 1.11.1
>Reporter: Moses
>Priority: Minor
> Labels: auto-deprioritized-major
>
> The SQL statement is designed as below:
> {code:sql}
> CREATE TABLE user_behavior (
> user_id BIGINT,
> item_id BIGINT,
> category_id BIGINT,
> behavior STRING default 'soo', Here I wanna use default value
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behavior',
> 'scan.startup.mode' = 'earliest-offset',
> 'properties.bootstrap.servers' = 'kafka:9094'
> 'format' = 'json'
> );
> {code}
> Throws Exception:
> {code:java}
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.sql.parser.impl.ParseException: Encountered "default"
> at line 5, column 21.
> Was expecting one of:
> "CONSTRAINT" ...
> {code}
> It's more friendly for us to support that allowing users to set default
> value during table schema definition. That will reduce UDXF usage and
> simplify data clean process to some extent.
> There are two points need to be solved:
> * Support SQL grammar.
> * Set field to specified value while field value is missing.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15000) WebUI Metrics is very slow in large parallelism
[
https://issues.apache.org/jira/browse/FLINK-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15000:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> WebUI Metrics is very slow in large parallelism
> ---
>
> Key: FLINK-15000
> URL: https://issues.apache.org/jira/browse/FLINK-15000
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0, 1.9.1
>Reporter: fa zheng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.14.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> metrics in web ui are very slow when parallelism is huge. It's hard to add
> metric and choose one metric. I run carTopSpeedWindowingExample with command
> {code:java}
> //代码占位符
> flink run -m yarn-cluster -p 1200 examples/streaming/TopSpeedWindowing.jar
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15004:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.1, 1.10.0
>Reporter: godfrey he
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22582) Scrollbar "jumps" when clicking on tabs for Scala and Python
[ https://issues.apache.org/jira/browse/FLINK-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22582: --- Labels: auto-deprioritized-major starter (was: stale-major starter) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Scrollbar "jumps" when clicking on tabs for Scala and Python > - > > Key: FLINK-22582 > URL: https://issues.apache.org/jira/browse/FLINK-22582 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Matthias >Priority: Minor > Labels: auto-deprioritized-major, starter > > For example, I am here: > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/operators/overview/#rescaling, > if you click between the Java/Scala/Python code example tab, the page will > scroll up and down to a totally different section. This makes it very hard to > read for users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16754) Consider refactoring of ProcessMemoryUtilsTestBase to avoid inheritance
[ https://issues.apache.org/jira/browse/FLINK-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16754: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Consider refactoring of ProcessMemoryUtilsTestBase to avoid inheritance > --- > > Key: FLINK-16754 > URL: https://issues.apache.org/jira/browse/FLINK-16754 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Configuration, Runtime / Coordination >Reporter: Andrey Zagrebin >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > After FLINK-16615 we have class structure of memory utils with isolation of > responsibilities, mostly through composition. We should consider to refactor > the tests as well to get more abstraction targeted tests with better > isolation and w/o implicit test inheritance contracts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9413) Tasks can fail with PartitionNotFoundException if consumer deployment takes too long
[
https://issues.apache.org/jira/browse/FLINK-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9413:
--
Labels: auto-deprioritized-critical pull-request-available stale-major
(was: auto-deprioritized-critical pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Tasks can fail with PartitionNotFoundException if consumer deployment takes
> too long
>
>
> Key: FLINK-9413
> URL: https://issues.apache.org/jira/browse/FLINK-9413
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, pull-request-available,
> stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{Tasks}} can fail with a {{PartitionNotFoundException}} if the deployment of
> the producer takes too long. More specifically, if it takes longer than the
> {{taskmanager.network.request-backoff.max}}, then the {{Task}} will give up
> and fail.
> The problem is that we calculate the {{InputGateDeploymentDescriptor}} for a
> consuming task once the producer has been assigned a slot but we do not wait
> until it is actually running. The problem should be fixed if we wait until
> the task is in state {{RUNNING}} before assigning the result partition to the
> consumer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9205) Delete non-well-defined SinkFunctions
[
https://issues.apache.org/jira/browse/FLINK-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9205:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Delete non-well-defined SinkFunctions
> -
>
> Key: FLINK-9205
> URL: https://issues.apache.org/jira/browse/FLINK-9205
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Specifically, these are:
> {code}
> OutputFormatSinkFunction.java
> WriteFormat.java
> WriteFormatAsCsv.java
> WriteFormatAsText.java
> WriteSinkFunction.java
> WriteSinkFunctionByMillis.java
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9247) Allow creating table sources/sinks from an Avro schema
[ https://issues.apache.org/jira/browse/FLINK-9247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9247: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Allow creating table sources/sinks from an Avro schema > -- > > Key: FLINK-9247 > URL: https://issues.apache.org/jira/browse/FLINK-9247 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > FLINK-8537 introduced table factories for reading Avro records from a Kafka > source. However, the configuration of the source requires an Avro generated > class to be present in the classpath. This might make it difficult for users > of the SQL Client (FLINK-7594) to quickly create a pipeline from a > string-based Avro schema. In order to improve this, we need table source > factories that can also deal with a schema instead of a Avro record class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9219) Add support for OpenGIS features in Table & SQL API
[
https://issues.apache.org/jira/browse/FLINK-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9219:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add support for OpenGIS features in Table & SQL API
> ---
>
> Key: FLINK-9219
> URL: https://issues.apache.org/jira/browse/FLINK-9219
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> CALCITE-1968 added core functionality for handling
> spatial/geographical/geometry data. It should not be too hard to expose these
> features also in Flink's Table & SQL API. We would need a new {{GEOMETRY}}
> data type and connect the function APIs.
> Right now the following functions are supported by Calcite:
> {code}
> ST_AsText, ST_AsWKT, ST_Boundary, ST_Buffer, ST_Contains,
> ST_ContainsProperly, ST_Crosses, ST_Disjoint, ST_Distance, ST_DWithin,
> ST_Envelope, ST_EnvelopesIntersect, ST_Equals, ST_GeometryType,
> ST_GeometryTypeCode, ST_GeomFromText, ST_Intersects, ST_Is3D,
> ST_LineFromText, ST_MakeLine, ST_MakePoint, ST_MLineFromText,
> ST_MPointFromText, ST_MPolyFromText, ST_Overlaps, ST_Point, ST_PointFromText,
> ST_PolyFromText, ST_SetSRID, ST_Touches, ST_Transform, ST_Union, ST_Within,
> ST_Z
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9507) Introduce ReduceFunction to CEP to minor the cost for IterativeCondition
[ https://issues.apache.org/jira/browse/FLINK-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9507: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Introduce ReduceFunction to CEP to minor the cost for IterativeCondition > > > Key: FLINK-9507 > URL: https://issues.apache.org/jira/browse/FLINK-9507 > Project: Flink > Issue Type: Improvement > Components: Library / CEP >Affects Versions: 1.5.0 >Reporter: Aitozi >Priority: Major > Labels: auto-unassigned, stale-major > > When we use the cep to describe a condition about the events that has been > matched, we have to call the method "getEventsForPattern" to extract the > partly matched elements from the sharedBuffer in RocksDB, which is very cost, > I think we can introduce a ReduceFunction on the pattern desc to allow us to > aggregate the value in advance. Is this practical ？ What's your guys > opinion? [~kkl0u] [~dawidwys] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9497) Job Configuration showing the password in plain text
[ https://issues.apache.org/jira/browse/FLINK-9497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9497: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Job Configuration showing the password in plain text > > > Key: FLINK-9497 > URL: https://issues.apache.org/jira/browse/FLINK-9497 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.3.2 >Reporter: Vinay >Priority: Major > Labels: auto-unassigned, stale-major > > I am submitting the job using Remote Execution environment, all the > configurations are shown in plain text when clicked on a particular job , > some of these configurations contain passwords (trustore,keystore, db etc > password) which should be masked as shown on the Job Manager Configuration > Screen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9239) Reintroduce example program in quickstarts
[ https://issues.apache.org/jira/browse/FLINK-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9239: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Reintroduce example program in quickstarts > -- > > Key: FLINK-9239 > URL: https://issues.apache.org/jira/browse/FLINK-9239 > Project: Flink > Issue Type: Improvement > Components: Quickstarts >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > FLINK-8761 removed all example programs from the quickstarts. I would propose > to reintroduce at least one tiny example that is runnable out of the box with > no parameters. Right now users are facing an exception that no sinks are > defined when running one of the program skeleton. Such example programs are > very useful in order to verify an IDE or local cluster setup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9453) Flink 1.5 incorrectly shows statistics in web UI
[
https://issues.apache.org/jira/browse/FLINK-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9453:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Flink 1.5 incorrectly shows statistics in web UI
>
>
> Key: FLINK-9453
> URL: https://issues.apache.org/jira/browse/FLINK-9453
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Affects Versions: 1.5.0
>Reporter: Andrei Shumanski
>Priority: Major
> Labels: auto-unassigned, stale-major
> Attachments: Flink_1.4.PNG, Flink_1.5.PNG
>
>
> Flink 1.5.0 incorrectly shows execution statistics and differently from Flink
> 1.4.1.
> I attach screenshots from both versions. It looks like 1.5.0 sums all steps
> in operators connected with "forward()". It makes it impossible to understand
> how much data was actually consumed.
> My test code:
>
> {code:java}
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> List input = new ArrayList<>();
> input.add(1);
> input.add(2);
> input.add(3);
> input.add(4);
> input.add(5);
> DataStream stream = env.fromCollection(input).rebalance();
> stream.map(i -> i+1).name("incr").forward().map(i ->
> i-1).name("decr").forward().print();
> env.execute();
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9391) Support UNNEST in Table API
[ https://issues.apache.org/jira/browse/FLINK-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-9391: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support UNNEST in Table API > --- > > Key: FLINK-9391 > URL: https://issues.apache.org/jira/browse/FLINK-9391 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > Labels: auto-unassigned, stale-major > > FLINK-6033 introduced the UNNEST function for SQL. We should also add this > function to the Table API to keep the APIs in sync. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9431) Introduce TimeEnd State to flink cep
[
https://issues.apache.org/jira/browse/FLINK-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9431:
--
Labels: auto-unassigned pull-request-available stale-major (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Introduce TimeEnd State to flink cep
>
>
> Key: FLINK-9431
> URL: https://issues.apache.org/jira/browse/FLINK-9431
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.4.2
>Reporter: Aitozi
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Now flink cep have no support to reach a Final State upon past some time. if
> i use a pattern like
> {code:java}Pattern.begin('A').notFollowedBy("B"){code}, if i want A element
> be emitted after 5minutes, i have no way.
> I want to introduce a timeEnd State to work with notFollowedBy to figure out
> with this scenior.
> It can be used like this
> {code:java}Pattern.begin('A').notFollowedBy("B").timeEnd("end").{code},
> [~dawidwys] [~kkl0u] Is this meaningful?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9334) Docs to have a code snippet of Kafka partition discovery
[
https://issues.apache.org/jira/browse/FLINK-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9334:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Docs to have a code snippet of Kafka partition discovery
>
>
> Key: FLINK-9334
> URL: https://issues.apache.org/jira/browse/FLINK-9334
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Reporter: Juho Autio
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Tzu-Li (Gordon) said:
> {quote}Yes, it might be helpful to have a code snippet to demonstrate the
> configuration for partition discovery.
> {quote}
>
>
> *Background*
>
> The docs correctly say:
>
> {quote}To enable it, set a non-negative value for
> +flink.partition-discovery.interval-millis+ in the _provided properties
> config_
> {quote}
>
> So it should be set in the Properties that are passed in the constructor of
> FlinkKafkaConsumer.
>
> I had somehow assumed that this should go to flink-conf.yaml (maybe because
> it starts with "flink."?), and obviously the FlinkKafkaConsumer doesn't read
> that.
>
> A piece of example code might've helped me avoid this mistake.
>
> This was discussed on the user mailing list:
>
> [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Consumers-Partition-Discovery-doesn-t-work-tp19129p19484.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9432) Support extract epoch, decade, millisecond, microsecond
[
https://issues.apache.org/jira/browse/FLINK-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9432:
--
Labels: auto-unassigned pull-request-available stale-major (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Support extract epoch, decade, millisecond, microsecond
> ---
>
> Key: FLINK-9432
> URL: https://issues.apache.org/jira/browse/FLINK-9432
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-major
>
> The task is to separate activity from depending on
> https://issues.apache.org/jira/browse/CALCITE-2303 from all others that could
> be done without upgrade avatica/calcite in
> https://issues.apache.org/jira/browse/FLINK-8518
> Now the implementations of next functions are blocked
> {code:sql}
> extract(decade from ...)
> extract(epoch from ...)
> extract(millisecond from ...)
> extract(microsecond from ...)
> extract(isodow from ...)
> extract(isoyear from ...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9195) Delete non-well-defined output methods on DataStream
[
https://issues.apache.org/jira/browse/FLINK-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9195:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Delete non-well-defined output methods on DataStream
>
>
> Key: FLINK-9195
> URL: https://issues.apache.org/jira/browse/FLINK-9195
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Some output methods on {{DataStream}} that write text to files are not safe
> to use in a streaming program as they have no consistency guarantees. They
> are:
> - {{writeAsText()}}
> - {{writeAsCsv()}}
> - {{writeToSocket()}}
> - {{writeUsingOutputFormat()}}
> Along with those we should also delete the {{SinkFunctions}} that they use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9399) TPCHQuery3 example fails due to Serialization issues
[
https://issues.apache.org/jira/browse/FLINK-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9399:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> TPCHQuery3 example fails due to Serialization issues
>
>
> Key: FLINK-9399
> URL: https://issues.apache.org/jira/browse/FLINK-9399
> Project: Flink
> Issue Type: Bug
> Components: Examples
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Running {{org.apache.flink.examples.java.relational.TPCHQuery3}} fails
> because the static class {{ShippingPriorityItem}} has private visibility, and
> hence cannot be serialized.
> *Stacktrace*
> {noformat}
> Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception:
> The data preparation for task 'CHAIN Join (Join at main(TPCHQuery3.java:155))
> -> Combine (SUM(1), at main(TPCHQuery3.java:165)' , caused an error: Cannot
> instantiate tuple.
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:625)
> at
> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:234)
> at
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
> at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
> at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
> at
> org.apache.flink.examples.java.relational.TPCHQuery3.main(TPCHQuery3.java:174)
> Caused by: java.lang.Exception: The data preparation for task 'CHAIN Join
> (Join at main(TPCHQuery3.java:155)) -> Combine (SUM(1), at
> main(TPCHQuery3.java:165)' , caused an error: Cannot instantiate tuple.
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:479)
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Cannot instantiate tuple.
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.createInstance(TupleSerializer.java:71)
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.createInstance(TupleSerializer.java:30)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.buildInitialTable(MutableHashTable.java:799)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.open(MutableHashTable.java:487)
> at
> org.apache.flink.runtime.operators.hash.NonReusingBuildFirstHashJoinIterator.open(NonReusingBuildFirstHashJoinIterator.java:98)
> at
> org.apache.flink.runtime.operators.JoinDriver.prepare(JoinDriver.java:207)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:473)
> ... 3 more
> Caused by: java.lang.IllegalAccessException: Class
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer can not access a
> member of class
> org.apache.flink.examples.java.relational.TPCHQuery3$ShippingPriorityItem
> with modifiers "public"
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
> at java.lang.Class.newInstance(Class.java:436)
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.createInstance(TupleSerializer.java:62)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9527) Paths initialized with URI are not normalized
[
https://issues.apache.org/jira/browse/FLINK-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9527:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Paths initialized with URI are not normalized
> -
>
> Key: FLINK-9527
> URL: https://issues.apache.org/jira/browse/FLINK-9527
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.0.3, 1.1.4, 1.2.1, 1.3.3, 1.4.2, 1.5.0, 1.6.0
>Reporter: Chesnay Schepler
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> The {{Path}} class has a lot of logic to normalize paths, like removing
> trailing slashes.
> However, all this normalization can be circumvented by passing in a {{URI}},
> where no normalization is done. I guess this was done on the assumption that
> the normalization was compatible with {{URI}}, but it turns out that this is
> not the case in the case of trailing slashes.
> An {{URI}} may have a trailing slash for directories, whereas a normalized
> {{Path}} does not.
> Unfortunately this breaks some assumptions made in the {{Path}} class, in
> particular {{Path#getParent()}} does not properly handle this case.
> Both assertions in the code below will fail.
> {code}
> public void testPathNormalization() {
> String path = "C:/Users/Zento/AppData/Local/Temp/test/";
> URI uri = URI.create(path);
> Path normalizedPath = new Path(path);
> Path uriPath = new Path(uri);
> Assert.assertEquals(normalizedPath.toString(),
> uriPath.toString());
> Assert.assertEquals("C:/Users/Zento/AppData/Local/Temp",
> uriPath.getParent().getPath());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9531) Improve logging during type extraction of the Row type
[
https://issues.apache.org/jira/browse/FLINK-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-9531:
--
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Improve logging during type extraction of the Row type
> --
>
> Key: FLINK-9531
> URL: https://issues.apache.org/jira/browse/FLINK-9531
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Reporter: Timo Walther
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Users of the Table API in Java will get several warnings about the usage of
> Rows even though the table program is specified correctly.
> {code}
> org.apache.flink.api.java.typeutils.TypeExtractor - class
> org.apache.flink.types.Row is not a valid POJO type
> {code}
> We should improve the user experience here. A correctly defined table program
> should not cause those messages. Users of the row type should rather be
> informed that they have to provide type information manually.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15206) support dynamic catalog table for truly unified SQL job
[
https://issues.apache.org/jira/browse/FLINK-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15206:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> support dynamic catalog table for truly unified SQL job
> ---
>
> Key: FLINK-15206
> URL: https://issues.apache.org/jira/browse/FLINK-15206
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Bowen Li
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> currently if users have both an online and an offline job with same business
> logic in Flink SQL, their codebase is still not unified. They would keep two
> SQL statements whose only difference is the source (or/and sink) table (with
> different params). E.g.
> {code:java}
> // online job
> insert into x select * from kafka_table (starting time) ...;
> // offline backfill job
> insert into x select * from hive_table (starting and ending time) ...;
> {code}
> We can introduce a "dynamic catalog table". The dynamic catalog table acts as
> a view, and is just an abstract table of multiple actual tables behind it
> that can be switched under some configuration flags. When execute a job,
> depending on the configuration, the dynamic catalog table can point to an
> actual source table.
> A use case for this is the example given above - when executed in streaming
> mode, {{my_source_dynamic_table}} should point to a kafka catalog table with
> a new starting position, and in batch mode, {{my_source_dynamic_table}}
> should point to a hive catalog table with starting/ending positions.
>
> One thing to note is that the starting position of kafka_table, and
> starting/ending position of hive_table are different every time. needs more
> thinking of how can we accommodate that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15208) client submits multiple sub-jobs for job with dynamic catalog table
[ https://issues.apache.org/jira/browse/FLINK-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15208: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > client submits multiple sub-jobs for job with dynamic catalog table > --- > > Key: FLINK-15208 > URL: https://issues.apache.org/jira/browse/FLINK-15208 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Client >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > with dynamic catalog table in FLINK-15206, users can maintain a single SQL > job for both their online and offline job. However, they still need to change > their configurations in order to submit different jobs over time. > E.g. when users update logic of their streaming job, they need to bootstrap > both a new online job and backfill offline job, let's call them sub-jobs of a > job with dynamic catalog table. They would have to > 1) manually change execution mode in yaml config to "streaming", execute the > sql and submit the streaming job > 2) manually change execution mode in yaml config to "batch", execute the sql > and submit the batch job > we should introduce a mechanism to allow users submit all or a subset of > sub-jobs all at once. In the backfill use case mentioned above, ideally users > should just execute the SQL once, and Flink should spin up two jobs for our > users. > Streaming platforms at some big companies like Uber and Netflix are already > kind of doing this for backfill use cases one way or another - some do it in > UI, some do it in planning phase. Would be great to standardize this practice > and provide users with ultimate simplicity. > The assumption here is that users are fully aware of the consequences of > launching two/multiple jobs at the same time. E.g. they need to handle > overlapped results if there's any. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15165) support upsert (Hive ACID) in Hive sink
[ https://issues.apache.org/jira/browse/FLINK-15165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15165: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > support upsert (Hive ACID) in Hive sink > --- > > Key: FLINK-15165 > URL: https://issues.apache.org/jira/browse/FLINK-15165 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Zhenqiu Huang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > Lots of our use cases need to backfill to existing hive table. It requires > support for apache hoodie / Hive ACID. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15856) Various Kubernetes command client improvements
[ https://issues.apache.org/jira/browse/FLINK-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15856: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Various Kubernetes command client improvements > -- > > Key: FLINK-15856 > URL: https://issues.apache.org/jira/browse/FLINK-15856 > Project: Flink > Issue Type: Improvement > Components: Command Line Client, Deployment / Kubernetes >Affects Versions: 1.10.0 >Reporter: Canbin Zheng >Priority: Minor > Labels: auto-deprioritized-major > > This is the umbrella issue for the Kubernetes related command client > improvements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15123) remove uniqueKeys from FlinkStatistic in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15123:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> remove uniqueKeys from FlinkStatistic in blink planner
> ---
>
> Key: FLINK-15123
> URL: https://issues.apache.org/jira/browse/FLINK-15123
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
> Fix For: 1.14.0
>
> Attachments: b_5.txt
>
>
> {{uniqueKeys}} is a kind of constraint, it's unreasonable that {{uniqueKeys}}
> is a kind of statistic. so we should remove uniqueKeys from
> {{FlinkStatistic}} in blink planner. Some temporary solutions (e.g.
> {{RichTableSourceQueryOperation}}) should also be resolved after primaryKey
> is introduced in {{TableSchema}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15183) Use SQL-CLI to TPC-DS E2E test
[ https://issues.apache.org/jira/browse/FLINK-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15183: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Use SQL-CLI to TPC-DS E2E test > -- > > Key: FLINK-15183 > URL: https://issues.apache.org/jira/browse/FLINK-15183 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Tests >Reporter: Jingsong Lee >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > Now SQL-CLI support DDL, we can use SQL-CLI to test tpc-ds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16562) Handle JobManager termination future in place
[
https://issues.apache.org/jira/browse/FLINK-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16562:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Handle JobManager termination future in place
> -
>
> Key: FLINK-16562
> URL: https://issues.apache.org/jira/browse/FLINK-16562
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Zili Chen
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> After FLINK-11843 {{Dispatcher}} becomes a {{PermanentlyFencedRpcEndpoint}}
> and will be created as different instance in difference leader epoch. Thus,
> we don't have {{jobManagerTerminationFutures}} crosses multiple leader epoch
> that should be handled. Given the truth, we can remove
> {{jobManagerTerminationFutures}} field in {{Dispatcher}} and handle those
> futures in place, which will simplify the code and helps on further
> refactoring.
> CC [~trohrmann]
> I will create a branch later this week.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15959) Add min number of slots configuration to limit total number of slots
[ https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15959: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add min number of slots configuration to limit total number of slots > > > Key: FLINK-15959 > URL: https://issues.apache.org/jira/browse/FLINK-15959 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: YufeiLiu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > Flink removed `-n` option after FLIP-6, change to ResourceManager start a new > worker when required. But I think maintain a certain amount of slots is > necessary. These workers will start immediately when ResourceManager starts > and would not release even if all slots are free. > Here are some resons: > # Users actually know how many resources are needed when run a single job, > initialize all workers when cluster starts can speed up startup process. > # Job schedule in topology order, next operator won't schedule until prior > execution slot allocated. The TaskExecutors will start in several batchs in > some cases, it might slow down the startup speed. > # Flink support > [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out > tasks evenly across all available registered TaskManagers], but it will only > effect if all TMs are registered. Start all TMs at begining can slove this > problem. > *suggestion:* > * Add config "taskmanager.minimum.numberOfTotalSlots" and > "taskmanager.maximum.numberOfTotalSlots", default behavior is still like > before. > * Start plenty number of workers to satisfy minimum slots when > ResourceManager accept leadership(subtract recovered workers). > * Don't comlete slot request until minimum number of slots are registered, > and throw exeception when exceed maximum. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20465) Fail globally when not resuming from the latest checkpoint in regional failover
[
https://issues.apache.org/jira/browse/FLINK-20465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20465:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Fail globally when not resuming from the latest checkpoint in regional
> failover
> ---
>
> Key: FLINK-20465
> URL: https://issues.apache.org/jira/browse/FLINK-20465
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major
>
> As a follow up for FLINK-20290 we should assert that we resume from the
> latest checkpoint when doing a regional failover in the
> {{SourceCoordinators}} in order to avoid losing input splits (see
> FLINK-20427). If the assumption does not hold, then we should fail the job
> globally so that we reset the master state to a consistent view of the state.
> Such a behaviour can act as a safety net in case that Flink ever tries to
> recover from not the latest available checkpoint.
> One idea how to solve it is to remember the latest completed checkpoint id
> somewhere along the way to the
> {{SplitAssignmentTracker.getAndRemoveUncheckpointedAssignment}} and failing
> when the restored checkpoint id is smaller.
> cc [~sewen], [~jqin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19358) when submit job on application mode with HA,the jobid will be 0000000000
[ https://issues.apache.org/jira/browse/FLINK-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-19358: --- Labels: auto-deprioritized-major usability (was: stale-major usability) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > when submit job on application mode with HA,the jobid will be 00 > > > Key: FLINK-19358 > URL: https://issues.apache.org/jira/browse/FLINK-19358 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Jun Zhang >Priority: Minor > Labels: auto-deprioritized-major, usability > > when submit a flink job on application mode with HA ,the flink job id will be > , when I have many jobs ,they have the same > job id , it will be lead to a checkpoint error -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18295:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20748) Remap excess slots to outstanding requirements
[
https://issues.apache.org/jira/browse/FLINK-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20748:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Remap excess slots to outstanding requirements
> --
>
> Key: FLINK-20748
> URL: https://issues.apache.org/jira/browse/FLINK-20748
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> The {{DeclarativeSlotPool}} maintains a mapping of slots to requirements that
> they fulfill. This mapping is currently determined when the slot is offered
> to the pool, and is only modified if the scheduler reserves this slot for a
> different requirement.
> This currently implies 2 things:
> * a slot can be prematurely released, although it could fulfill a different
> outstanding requirement
> * a new slot could be requested, despite there being another slot around
> that could fulfill the requirement
> We should remedy this by checking whether excess slots can be re-mapped to a
> different requirement, whenever the requirements change or a slot is freed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[
https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20714:
---
Labels: auto-deprioritized-major keberos (was: keberos stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Hive delegation token is not obtained when using `kinit` to submit Yarn
> per-job
>
>
> Key: FLINK-20714
> URL: https://issues.apache.org/jira/browse/FLINK-20714
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Affects Versions: 1.11.2, 1.11.3, 1.12.0
> Environment: Flink 1.11.2 on Yarn
>Reporter: jackwangcs
>Priority: Minor
> Labels: auto-deprioritized-major, keberos
>
> Hive delegation token is not obtained when using `kinit` to submit Yarn
> per-job.
> In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor
> to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase
> tokens currently, since the Hive integration is supported, the Hive
> delegation should be obtained also.
> Otherwise, it will throw the following error when it tries to connect to
> Hive metastore:
> {code:java}
> Caused by: MetaException(message:Could not connect to meta store using any of
> the URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: GSS initiate failedCaused
> by: MetaException(message:Could not connect to meta store using any of the
> URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: GSS initiate failed at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187)
> at
> org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97)
> at
> org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240)
> at
> org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71)
> at
> org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35)
> at
> org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74)
> at
> org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68)
> at
> org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65)
> at
> org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43)
> at
> org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61)
> at
> org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97)
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20681) Support specifying the hdfs path when ship archives or files
[ https://issues.apache.org/jira/browse/FLINK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20681: --- Labels: auto-deprioritized-major auto-unassigned pull-request-available pull-requests-available (was: auto-unassigned pull-request-available pull-requests-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support specifying the hdfs path when ship archives or files > - > > Key: FLINK-20681 > URL: https://issues.apache.org/jira/browse/FLINK-20681 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0 >Reporter: Ruguo Yu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available, pull-requests-available > Attachments: image-2020-12-23-20-58-41-234.png, > image-2020-12-24-01-01-10-021.png > > > Currently, our team try to submit flink job that depends extra resource with > yarn-application target, and use two options: "yarn.ship-archives" and > "yarn.ship-files". > But above options only support specifying local resource and shiping them to > hdfs, besides if it can support remote resource on distributed filesystem > (such as hdfs), then get the following benefits： > * client will exclude the local resource uploading to accelerate the job > submission process > * yarn will cache them on the nodes so that they doesn't need to be > downloaded for application -- This message was sent by Atlassian Jira (v8.3.4#803005)
