[jira] [Created] (FLINK-18966) Support key_by() on ConnectedStreams for Python DataStream API
Hequn Cheng created FLINK-18966: --- Summary: Support key_by() on ConnectedStreams for Python DataStream API Key: FLINK-18966 URL: https://issues.apache.org/jira/browse/FLINK-18966 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Assignee: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Flink S3 Hadoop dependencies
Filesystems are supposed to be used as plugins (by putting the jars under plugins/ instead of lib/), in which case they are loaded separately from other classes, specifically user-code. https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/plugins.html On 14/08/2020 20:25, Satish Saley wrote: Hi team, Was there a reason for not shading hadoop-common https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44 ? This is leaking lots of classes such as guava and causing issues in our flink application. I see that hadoop-common classes were shaded in earlier versions https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0 Stacktrace : Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V at io.grpc.Metadata$Key.validateName(Metadata.java:742) at io.grpc.Metadata$Key.(Metadata.java:750) at io.grpc.Metadata$Key.(Metadata.java:668) at io.grpc.Metadata$AsciiKey.(Metadata.java:959) at io.grpc.Metadata$AsciiKey.(Metadata.java:954) at io.grpc.Metadata$Key.of(Metadata.java:705) at io.grpc.Metadata$Key.of(Metadata.java:701) at io.grpc.internal.GrpcUtil.(GrpcUtil.java:80) at io.grpc.internal.AbstractManagedChannelImplBuilder.(AbstractManagedChannelImplBuilder.java:90) at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42) at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23) at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)
Re: [DISCUSS] Planning Flink 1.12
What about the work of migrating some Flink sources to the new FLIP-27 source interface? They are not listed in the 1.12 release wiki page. On Thu, Aug 13, 2020 at 6:51 PM Dian Fu wrote: > Hi Rodrigo, > > Both FLIP-130 and FLIP-133 will be in the list of 1.12. Besides, there are > also some other features from PyFlink side in 1.12. More details could be > found in the wiki page( > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release < > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release>). > > Regards, > Dian > > > 在 2020年8月14日，上午9:37，rodrigobrochado > 写道： > > > > Hi, > > > > I hope it's not too late to ask, but would FLIP-130 [1] and FLIP-133 [2] > be > > considered? I think that it would be nice to have some details of pyFlink > > Datastreams API (FLIP-130) on the roadmap, giving us (users) more > insights > > into what we can expect from pyFlink in the near future. > > > > > > [1] > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html > > [2] > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-133-Rework-PyFlink-Documentation-tt43570.html > > > > > > Thanks, > > Rodrigo > > > > > > > > -- > > Sent from: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ > >
Flink S3 Hadoop dependencies
Hi team, Was there a reason for not shading hadoop-common https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44 ? This is leaking lots of classes such as guava and causing issues in our flink application. I see that hadoop-common classes were shaded in earlier versions https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0 Stacktrace : Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V at io.grpc.Metadata$Key.validateName(Metadata.java:742) at io.grpc.Metadata$Key.(Metadata.java:750) at io.grpc.Metadata$Key.(Metadata.java:668) at io.grpc.Metadata$AsciiKey.(Metadata.java:959) at io.grpc.Metadata$AsciiKey.(Metadata.java:954) at io.grpc.Metadata$Key.of(Metadata.java:705) at io.grpc.Metadata$Key.of(Metadata.java:701) at io.grpc.internal.GrpcUtil.(GrpcUtil.java:80) at io.grpc.internal.AbstractManagedChannelImplBuilder.(AbstractManagedChannelImplBuilder.java:90) at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42) at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23) at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)
[jira] [Created] (FLINK-18964) package org.apache.flink.sql.parser.impl does not exist
weizihan created FLINK-18964: Summary: package org.apache.flink.sql.parser.impl does not exist Key: FLINK-18964 URL: https://issues.apache.org/jira/browse/FLINK-18964 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.12.0 Reporter: weizihan org.apache.flink.sql.parser.utils.ParserResource class *cannot resolve* org.apache.flink.sql.parser.impl.ParseException class, because it has not org.apache.flink.sql.parser.impl package. The following is the error message: /flink/flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/utils/ParserResource.java Error:(21, 40) java: package org.apache.flink.sql.parser.impl does not exist Error:(34, 26) java: cannot find symbol symbol: class ParseException location: interface org.apache.flink.sql.parser.utils.ParserResource Error:(37, 26) java: cannot find symbol symbol: class ParseException location: interface org.apache.flink.sql.parser.utils.ParserResource -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18963) Added Copyright information to coding style guide
Matthias created FLINK-18963: Summary: Added Copyright information to coding style guide Key: FLINK-18963 URL: https://issues.apache.org/jira/browse/FLINK-18963 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Matthias Assignee: Matthias Add Copyright as a requirement to [https://flink.apache.org/contributing/code-style-and-quality-common.html] Add Copyright profile instructions to ide_setup.md (including the Chinese version). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Kinesis Performance Issue (was [VOTE] Release 1.11.0, release candidate #4)
Thanks Thomas for reporting the problem, analysing which commit has caused
and now for the verification that it was fixed :) Much appreciated.
Piotrek
czw., 13 sie 2020 o 18:18 Thomas Weise napisał(a):
> Hi Roman,
>
> Thanks for working on this! I deployed the change and it appears to be
> working as expected.
>
> Will monitor over a period of time to compare the checkpoint counts and get
> back to you if there are still issues.
>
> Thomas
>
>
> On Thu, Aug 13, 2020 at 3:41 AM Roman Khachatryan >
> wrote:
>
> > Hi Thomas,
> >
> > The fix is now merged to master and to release-1.11.
> > So if you'd like you can check if it solves your problem (it would be
> > helpful for us too).
> >
> > On Sat, Aug 8, 2020 at 9:26 AM Roman Khachatryan <
> ro...@data-artisans.com>
> > wrote:
> >
> >> Hi Thomas,
> >>
> >> Thanks a lot for the detailed information.
> >>
> >> I think the problem is in CheckpointCoordinator. It stores the last
> >> checkpoint completion time after checking queued requests.
> >> I've created a ticket to fix this:
> >> https://issues.apache.org/jira/browse/FLINK-18856
> >>
> >>
> >> On Sat, Aug 8, 2020 at 5:25 AM Thomas Weise wrote:
> >>
> >>> Just another update:
> >>>
> >>> The duration of snapshotState is capped by the Kinesis
> >>> producer's "RecordTtl" setting (default 30s). The sleep time in
> flushSync
> >>> does not contribute to the observed behavior.
> >>>
> >>> I guess the open question is why, with the same settings, is 1.11 since
> >>> commit 355184d69a8519d29937725c8d85e8465d7e3a90 processing more
> checkpoints?
> >>>
> >>>
> >>> On Fri, Aug 7, 2020 at 9:15 AM Thomas Weise wrote:
> >>>
> Hi Roman,
>
> Here are the checkpoint summaries for both commits:
>
>
>
> https://docs.google.com/presentation/d/159IVXQGXabjnYJk3oVm3UP2UW_5G-TGs_u9yzYb030I/edit#slide=id.g86d15b2fc7_0_0
>
> The config:
>
> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
>
> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> checkpointConfig.setCheckpointInterval(*10_000*);
> checkpointConfig.setMinPauseBetweenCheckpoints(*10_000*);
>
>
> checkpointConfig.enableExternalizedCheckpoints(DELETE_ON_CANCELLATION);
> checkpointConfig.setCheckpointTimeout(600_000);
> checkpointConfig.setMaxConcurrentCheckpoints(1);
> checkpointConfig.setFailOnCheckpointingErrors(true);
>
> The values marked bold when changed to *60_000* make the symptom
> disappear. I meanwhile also verified that with the 1.11.0 release
> commit.
>
> I will take a look at the sleep time issue.
>
> Thanks,
> Thomas
>
>
> On Fri, Aug 7, 2020 at 1:44 AM Roman Khachatryan <
> ro...@data-artisans.com> wrote:
>
> > Hi Thomas,
> >
> > Thanks for your reply!
> >
> > I think you are right, we can remove this sleep and improve
> > KinesisProducer.
> > Probably, it's snapshotState can also be sped up by forcing records
> > flush more often.
> > Do you see that 30s checkpointing duration is caused
> > by KinesisProducer (or maybe other operators)?
> >
> > I'd also like to understand the reason behind this increase in
> > checkpoint frequency.
> > Can you please share these values:
> > - execution.checkpointing.min-pause
> > - execution.checkpointing.max-concurrent-checkpoints
> > - execution.checkpointing.timeout
> >
> > And what is the "new" observed checkpoint frequency (or how many
> > checkpoints are created) compared to older versions?
> >
> >
> > On Fri, Aug 7, 2020 at 4:49 AM Thomas Weise wrote:
> >
> >> Hi Roman,
> >>
> >> Indeed there are more frequent checkpoints with this change! The
> >> application was configured to checkpoint every 10s. With 1.10 ("good
> >> commit"), that leads to fewer completed checkpoints compared to 1.11
> >> ("bad
> >> commit"). Just to be clear, the only difference between the two runs
> >> was
> >> the commit 355184d69a8519d29937725c8d85e8465d7e3a90
> >>
> >> Since the sync part of checkpoints with the Kinesis producer always
> >> takes
> >> ~30 seconds, the 10s configured checkpoint frequency really had no
> >> effect
> >> before 1.11. I confirmed that both commits perform comparably by
> >> setting
> >> the checkpoint frequency and min pause to 60s.
> >>
> >> I still have to verify with the final 1.11.0 release commit.
> >>
> >> It's probably good to take a look at the Kinesis producer. Is it
> >> really
> >> necessary to have 500ms sleep time? What's responsible for the ~30s
> >> duration in snapshotState?
> >>
> >> As things stand it doesn't make sense to use checkpoint intervals <
> >> 30s
> >> when using the Kinesis producer.
> >>
> >> Thanks,
> >> Thomas
> >>
> >> On
[jira] [Created] (FLINK-18962) Improve error message if checkpoint directory is not writable
Nico Kruber created FLINK-18962:
---
Summary: Improve error message if checkpoint directory is not
writable
Key: FLINK-18962
URL: https://issues.apache.org/jira/browse/FLINK-18962
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing
Affects Versions: 1.11.1
Reporter: Nico Kruber
If the checkpoint directory from {{state.checkpoints.dir}} is not writable by
the user that Flink is running with, checkpoints will be declined, but the real
cause is not mentioned anywhere:
* the Web UI says: "Cause: The job has failed" (the Flink job is running though)
* the JM log says:
{code}
2020-08-14 12:13:18,820 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Triggering
checkpoint 2 (type=CHECKPOINT) @ 159738819 for job
2c567b14e8d0833404931ef47dfec266.
2020-08-14 12:13:18,921 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Decline
checkpoint 2 by task 0d4fd75374ad16c8d963679e3c2171ec of job
2c567b14e8d0833404931ef47dfec266 at a184deea621e3923fbfcb1d899348448 @
Nico-PC.lan (dataPort=35531).
{code}
* the TM log says:
{code}
2020-08-14 12:13:14,102 INFO
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] -
Checkpoint 1 has been notified as aborted, would not trigger any checkpoint.
{code}
And that's it. It should have a real error message indicating that the
checkpoint (sub)-directory could not be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18961) In the case of FlatMap linking map, if map returns null, an exception will be thrown in FlatMap
Ryan created FLINK-18961:
Summary: In the case of FlatMap linking map, if map returns null,
an exception will be thrown in FlatMap
Key: FLINK-18961
URL: https://issues.apache.org/jira/browse/FLINK-18961
Project: Flink
Issue Type: Bug
Components: API / DataSet
Affects Versions: 1.11.0
Environment: Mac OS 10.13.6
Kubernetes 1.16.8
Flink 1.11.0
Reporter: Ryan
Attachments: Lark20200814-173817.png, Lark20200814-173821.png,
Lark20200814-173824.png
I found a DateSet problem. In the case of FlatMap linking map, if map returns
null, an exception will be thrown in FlatMap.I think it's a problem with the
operator chain.I will post a screenshot of the corresponding stack call in the
attachment.
{code:java}
text.filter(value -> value.f0.contains("any")).flatMap(new
FlatMapFunction, String>() {
@Override
public void flatMap(Tuple2 value,
Collector out) throws Exception {
Pattern pattern = Pattern.compile("\".*\"");
Matcher matcher = pattern.matcher(value.f0);
if(matcher.find()){
String match = matcher.group(0);
out.collect(match); // here throw Exception
}
}
}).map(value -> {
try {
String jsonS = value.replace("\"\"","\"");
jsonS = jsonS.substring(1,jsonS.length()-1);
JSONObject json = JSONObject.parseObject(jsonS);
String result =
json.getJSONObject("body").getJSONObject("message").getString("data");
return result; // this is null
}catch (Exception e){
return value;
}
}).print();{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18960) flink sideoutput union
xiaohang.li created FLINK-18960:
---
Summary: flink sideoutput union
Key: FLINK-18960
URL: https://issues.apache.org/jira/browse/FLINK-18960
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.10.1
Environment: val side = new OutputTag[String]("side")
val side2 = new OutputTag[String]("side2")
val side3 = new OutputTag[String]("side3")
val ds = env.socketTextStream("master",9001)
val res = ds.process(new ProcessFunction[String,String] {
override def processElement(value: String, ctx: ProcessFunction[String,
String]#Context, out: Collector[String]): Unit = {
if(value.contains("hello")){
ctx.output(side,value)
}else if(value.contains("world")){
ctx.output(side2,value)
}else if(value.contains("flink")){
ctx.output(side3,value)
}
out.collect(value)
}
})
val res1 = res.getSideOutput(side)
val res2 = res.getSideOutput(side2)
val res3 = res.getSideOutput(side3)
println( ">"+res1.getClass)
println( ">"+res2.getClass)
res1.print("res1")
res2.print("res2")
res3.print("res3")
res2.union(res1).union(res3).print("all")
在socket端口分别输入
hello
world
flink
idea显示数据如下
res1> hello
res2> world
res3> flink
all> flink
all> flink
all> flink
可见在all输出流显示的是最后一个union的侧输出流*union的次数，实际显示应为
all>flink
Reporter: xiaohang.li
flink sideoutput
union操作时数据出现问题。从主流分出来的侧输出流进行union操作时，显示输出的是以最后一个union的数据流结果*union的次数
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18959) Fail to archiveExecutionGraph because job is not finished when dispatcher close
Liu created FLINK-18959:
---
Summary: Fail to archiveExecutionGraph because job is not finished
when dispatcher close
Key: FLINK-18959
URL: https://issues.apache.org/jira/browse/FLINK-18959
Project: Flink
Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Liu
When job is cancelled, we expect to see it in flink's history server. But I can
not see my job after it is cancelled.
After digging into the problem, I find that the function archiveExecutionGraph
is not executed. Below is the brief log:
{panel:title=log}
2020-08-14 15:10:06,412 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph
[flink-akka.actor.default-dispatcher-15] -
2-2.1_Window(TumblingProcessingTimeWindows(60), ProcessingTimeTrigger,
WindowFunction$1) (4/5) (14a86b2a2b4debe6ba61bf4551cb3619) switched from
RUNNING to CANCELING.
2020-08-14 15:10:06,415 DEBUG
org.apache.flink.runtime.dispatcher.MiniDispatcher
[flink-akka.actor.default-dispatcher-3] - Shutting down per-job cluster because
the job was canceled.
2020-08-14 15:10:06,629 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher
[flink-akka.actor.default-dispatcher-3] - Stopping dispatcher
akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
2020-08-14 15:10:06,629 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher
[flink-akka.actor.default-dispatcher-3] - Stopping all currently running jobs
of dispatcher akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
2020-08-14 15:10:06,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
[flink-akka.actor.default-dispatcher-29] - Stopping the JobMaster for job
EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
2020-08-14 15:10:06,632 DEBUG org.apache.flink.runtime.jobmaster.JobMaster
[flink-akka.actor.default-dispatcher-29] - Disconnect TaskExecutor
container_e144_1590060720089_2161_01_06 because: Stopping JobMaster for job
EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
2020-08-14 15:10:06,646 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph
[flink-akka.actor.default-dispatcher-29] - Job EtlAndWindow
(6f784d4cc5bae88a332d254b21660372) switched from state CANCELLING to CANCELED.
2020-08-14 15:10:06,664 DEBUG
org.apache.flink.runtime.dispatcher.MiniDispatcher
[flink-akka.actor.default-dispatcher-4] - There is a newer JobManagerRunner for
the job 6f784d4cc5bae88a332d254b21660372.
{panel}
>From the log, we can see that job is not finished when dispatcher close. The
>process is as following:
* Receive cancel command and send it to all tasks async.
* In MiniDispatcher, begin to shutting down per-job cluster.
* Stopping dispatcher and remove job.
* Job is cancelled and callback is executed in method startJobManagerRunner.
* Because job is removed before, so currentJobManagerRunner is null which not
equals to the original jobManagerRunner. In this case, archivedExecutionGraph
will not be uploaded.
In normal cases, I find that job is cancelled first and then dispatcher is
stopped so that archivedExecutionGraph will succeed. But I think that the order
is not constrained and it is hard to know which comes first.
Above is what I suspected. If so, then we should fix it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18958) Lose column comment when create table
Shengkai Fang created FLINK-18958:
-
Summary: Lose column comment when create table
Key: FLINK-18958
URL: https://issues.apache.org/jira/browse/FLINK-18958
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Shengkai Fang
Currently, table column will not store column comment and user can't see column
comment when use {{describe table}} sql.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18957) Implement bulk fulfil-ability timeout tracking for shared slots
Andrey Zagrebin created FLINK-18957: --- Summary: Implement bulk fulfil-ability timeout tracking for shared slots Key: FLINK-18957 URL: https://issues.apache.org/jira/browse/FLINK-18957 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Andrey Zagrebin Assignee: Andrey Zagrebin Fix For: 1.12.0 Track fulfil-ability of required physical slots for all SharedSlot(s) (no matter whether they are created at this bulk or not) with timeout. This ensures we will not wait indefinitely if the required slots for this bulk cannot be fully fulfilled at the same time. # Create a LogicalSlotRequestBulk to track all physical requests and logical slot requests (logical slot requests only which belong to the bulk) # Mark physical slot request fulfilled in LogicalSlotRequestBulk, once its future is done # If any physical slot request fails then clear the LogicalSlotRequestBulk to stop the fulfil-ability check # Schedule a fulfil-ability check in LogicalSlotRequestBulkChecker for the LogicalSlotRequestBulk # In case of timeout: # cancel/fail the logical slot futures of the bulk in SharedSlot(s) # remove -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18956) StreamTask.invoke should catch Throwable instead of Exception
Dian Fu created FLINK-18956:
---
Summary: StreamTask.invoke should catch Throwable instead of
Exception
Key: FLINK-18956
URL: https://issues.apache.org/jira/browse/FLINK-18956
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.11.0
Reporter: Dian Fu
Assignee: Dian Fu
Fix For: 1.12.0, 1.11.2
In StreamTask.invoke, we should catch Throwable. Otherwise, cleanUpInvoke()
will not be called if Error is thrown:
{code}
@Override
public final void invoke() throws Exception {
try {
beforeInvoke();
// final check to exit early before starting to run
if (canceled) {
throw new CancelTaskException();
}
// let the task do its work
runMailboxLoop();
// if this left the run() method cleanly despite the
fact that this was canceled,
// make sure the "clean shutdown" is not attempted
if (canceled) {
throw new CancelTaskException();
}
afterInvoke();
}
catch (Exception invokeException) {
failing = !canceled;
try {
cleanUpInvoke();
}
// TODO: investigate why Throwable instead of Exception
is used here.
catch (Throwable cleanUpException) {
Throwable throwable =
ExceptionUtils.firstOrSuppressed(cleanUpException, invokeException);
throw (throwable instanceof Exception ?
(Exception) throwable : new Exception(throwable));
}
throw invokeException;
}
cleanUpInvoke();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18955) Add snapshot path to job startup message
Nico Kruber created FLINK-18955:
---
Summary: Add snapshot path to job startup message
Key: FLINK-18955
URL: https://issues.apache.org/jira/browse/FLINK-18955
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 1.11.1
Reporter: Nico Kruber
When a job is started from a checkpoint or savepoint (I'm using snapshot as the
unanimous term below), the {{CheckpointCoordinator}} prints a log line like
this:
{code}
2020-08-13 13:50:51,418 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Restoring job
220d8a4953cd40198b6eb3b1ec0cece0 from latest valid checkpoint: Checkpoint 357 @
1597326576925 for 220d8a4953cd40198b6eb3b1ec0cece0.
{code}
I propose to add the path to the snapshot to this message because which
snapshot is taken for restore may actually not be that obvious for the user:
even if a savepoint was specified in the job start command, e.g. in a
Kubernetes pod spec, an HA store could overrule the decision and take a more
recent snapshot instead. If that snapshot is a savepoint, it is not that easy
to map this to checkpoint IDs and find out which savepoint the job actually
started from.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18954) Add documentation for Metrics in Python DataStream API.
Hequn Cheng created FLINK-18954: --- Summary: Add documentation for Metrics in Python DataStream API. Key: FLINK-18954 URL: https://issues.apache.org/jira/browse/FLINK-18954 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Assignee: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18953) Add documentation for DataTypes in Python DataStream API
Hequn Cheng created FLINK-18953: --- Summary: Add documentation for DataTypes in Python DataStream API Key: FLINK-18953 URL: https://issues.apache.org/jira/browse/FLINK-18953 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18952) Add 10 minutes to DataStream API documentation
Hequn Cheng created FLINK-18952: --- Summary: Add 10 minutes to DataStream API documentation Key: FLINK-18952 URL: https://issues.apache.org/jira/browse/FLINK-18952 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18950) Add documentation for Operations in Python DataStream API.
Shuiqiang Chen created FLINK-18950: -- Summary: Add documentation for Operations in Python DataStream API. Key: FLINK-18950 URL: https://issues.apache.org/jira/browse/FLINK-18950 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18951) Add documentation for Configurations in Python DataStream API.
Shuiqiang Chen created FLINK-18951: -- Summary: Add documentation for Configurations in Python DataStream API. Key: FLINK-18951 URL: https://issues.apache.org/jira/browse/FLINK-18951 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18949) Support Streaming File Sink for Python DataStream API
Hequn Cheng created FLINK-18949: --- Summary: Support Streaming File Sink for Python DataStream API Key: FLINK-18949 URL: https://issues.apache.org/jira/browse/FLINK-18949 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18948) Add end to end test for Python DataStream API
Hequn Cheng created FLINK-18948: --- Summary: Add end to end test for Python DataStream API Key: FLINK-18948 URL: https://issues.apache.org/jira/browse/FLINK-18948 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18947) Support partitionCustom() operation for Python DataStream API
Hequn Cheng created FLINK-18947: --- Summary: Support partitionCustom() operation for Python DataStream API Key: FLINK-18947 URL: https://issues.apache.org/jira/browse/FLINK-18947 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18946) Support Cassandra connector for Python DataStream API
Shuiqiang Chen created FLINK-18946: -- Summary: Support Cassandra connector for Python DataStream API Key: FLINK-18946 URL: https://issues.apache.org/jira/browse/FLINK-18946 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18945) Support key_by() on ConnectedStreams for Python DataStream API
Hequn Cheng created FLINK-18945: --- Summary: Support key_by() on ConnectedStreams for Python DataStream API Key: FLINK-18945 URL: https://issues.apache.org/jira/browse/FLINK-18945 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18944) Support JDBC connector for Python DataStream API
Shuiqiang Chen created FLINK-18944: -- Summary: Support JDBC connector for Python DataStream API Key: FLINK-18944 URL: https://issues.apache.org/jira/browse/FLINK-18944 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18943) Support connect() operation for Python DataStream API
Hequn Cheng created FLINK-18943: --- Summary: Support connect() operation for Python DataStream API Key: FLINK-18943 URL: https://issues.apache.org/jira/browse/FLINK-18943 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Hequn Cheng Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18942) HiveTableSink shouldn't try to create BulkWriter factory when using MR writer
Rui Li created FLINK-18942: -- Summary: HiveTableSink shouldn't try to create BulkWriter factory when using MR writer Key: FLINK-18942 URL: https://issues.apache.org/jira/browse/FLINK-18942 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.11.1 Reporter: Rui Li Fix For: 1.11.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
