Hi,
I submit a stream job from the SQL gateway. The stream job keeps outputting
results to the SQL gateway. If the SQL gateway restarts or crashes, the stream
job will continue running. After the SQL gateway restarts, how to get the
results of the steam job?
Thanks.
Jack
Thanks for the details Chesnay!
By “alias” I mean to respect the original definition made in FLIP-33 for
numRecordsOut, which is the number of records written to the external system,
and keep numRecordsSend as the same value as numRecordsOut for compatibility.
I think keeping numRecordsOut for
Hi
我理解应该是任务恢复的时候从上一次成功的 checkpoint 或者你指定的 checkpoint 里记录的 offset
开始消费，所以此时的统计值应该是有短暂的下跌，因为数据相当于回复到之前重复计算了一部分。这个应该是符合预期的，可能需要在业务上做一些处理。
Best
JasonLee
回复的原邮件
| 发件人 | 天下五帝东 |
| 发送日期 | 2022年10月10日 13:34 |
| 收件人 | user-zh@flink.apache.org |
| 主题 | Flink sql从ck恢复，统计数据波动问题 |
Hi:
当前flinlk cdc master分支的snapshot版本最高支持到flink 1.14.4，尝试使用flink 1.15.2编译会出错，请问flink
cdc什么时候支持flink 1.15.x?
可以给一些hints吗？看哪些类?
在 2022-10-11 10:22:07，"yuxia" 写道：
>用 datastream api，自己解析一下 DDL。
>
>Best regards,
>Yuxia
>
>- 原始邮件 -
>发件人: "yh z"
>收件人: "user-zh"
>发送时间: 星期二, 2022年 10 月 11日 上午 10:23:43
>主题: Re: flink cdc能否同步DDL语句?
>
>目前，社区的 Flink CDC，是可以读取到 binlog 中的 DDL
用 datastream api，自己解析一下 DDL。
Best regards,
Yuxia
- 原始邮件 -
发件人: "yh z"
收件人: "user-zh"
发送时间: 星期二, 2022年 10 月 11日 上午 10:23:43
主题: Re: flink cdc能否同步DDL语句?
目前，社区的 Flink CDC，是可以读取到 binlog 中的 DDL 语句的，但是不会传递给下游，即下游无法感知 schema 的变化。
Xuyang 于2022年10月10日周一 16:46写道：
> Hi， 目前应该是不行的
> 在
目前，社区的 Flink CDC，是可以读取到 binlog 中的 DDL 语句的，但是不会传递给下游，即下游无法感知 schema 的变化。
Xuyang 于2022年10月10日周一 16:46写道：
> Hi， 目前应该是不行的
> 在 2022-09-26 23:27:05，"casel.chen" 写道：
> >flink cdc能否同步DDL语句以实现schema同步? 例如create table , create index, truncate
> table等
>
Hi，
调大 taskmanager.memory.task.off-heap.size 应该能解决部分问题，
我这里还有些疑问，部署的session集群，每次集群只跑这一个任务，执行结束才开始下一个任务，如果是前面的任务read/write申请的堆外内存，执行结束的时候，会立即释放吗？
执行几次任务之后，才出现这种异常，前面任务都是成功的，后面任务就异常了，感觉有内存泄漏的现象。Flink
taskmanager的off-heap内存管理有更多的介绍吗？（官网的看过了）
Thanks
在 2022-10-10 12:34:55，"yanfei lei" 写道：
退订
--
发自我的网易邮箱手机智能版
Hi,
去除掉后运行是没有问题的，所以这种lib的冲突问题，能在官网上加个说明吗，避免后续其他人也遇到这种问题。
Thanks
在 2022-10-10 12:50:33，"yanfei lei" 写道：
>Hi, 从table store的pom中看，table store的dist包shade了一份connector-kafka。
>https://repo1.maven.org/maven2/org/apache/flink/flink-table-store-dist/0.2.0/flink-table-store-dist-0.2.0.pom
数据库的统计值
> 下面是被转发的邮件：
>
> 发件人: Hangxiang Yu
> 主题: 回复：Flink sql从ck恢复，统计数据波动问题
> 日期: 2022年10月10日 GMT+8 下午2:03:50
> 收件人: user-zh@flink.apache.org
> 回复－收件人: user-zh@flink.apache.org
>
> 是什么值下跌呢？哪个metric吗？
>
> On Mon, Oct 10, 2022 at 1:34 PM 天下五帝东 wrote:
>
>> Hi:
>>
I am writing Flink applications in Java and I need to do data type
conversions between SQL/Table `DataType` and `TypeInformation`. According
to Flink's documentation, type mapping methods in TypeConversions
Hey Flinksters,
I was wondering if you had any ideas for how to preserve the rowtime across
an INNER equi join so that the output can be used in a temporal join.
I've attached an example based on the TemporalJoinTest where I'm creating
two views by deduplicating underlying streams (to rates_pk
Hi Sucheth,
The short answer is no, when deserializing Avro messages you have to
provide the schema somehow, either directly or using the Schema Registry.
On Mon, Oct 10, 2022 at 10:00 AM Sucheth S wrote:
> Hi,
>
> I'm trying to deserialize avro messages from the kafka topic as a
> consumer.
>
Hi,What you mean by holding the schema? You don't have to hold it as schema registry does it for you.20:00, 10 октября 2022 г., Sucheth S :Hi,I'm trying to deserialize avro messages from the kafka topic as a consumer. As a kafka consumer, I do not want to hold the reader schema on my end.Is there
Hi,
I'm trying to deserialize avro messages from the kafka topic as a consumer.
As a kafka consumer, I do not want to hold the reader schema on my end.
Is there a way to deserialize avro messages to GenericRecord
without providing the reader schema ?
Regards,
Sucheth Shivakumar
website :
Thanks for the confirmation :)
Regards,
Alexis.
On Sun, 9 Oct 2022, 10:37 Hangxiang Yu, wrote:
> Hi, Alexis.
> I think you are right. It also applies for a global window with a custom
> trigger.
> If you apply a ReduceFunction or AggregateFunction, the window state size
> usually is smaller
+1
Am 10. Oktober 2022 14:39:28 MESZ schrieb "guenterh.lists"
:
>really very sad - as far as I know this happens for the first time, attitude
>of new Ververica?
>
>Hopefully immerok may resume the open mentality of data artisans.
>
>Günter
>
>On 10.10.22 11:26, Martijn Visser wrote:
>> Hi
+1 for reverting these changes in Flink 1.16, so I will cancel 1.16.0-rc1.
+1 for `numXXXSend` as the alias of `numXXXOut` in 1.15.3.
Best,
Xingbo
Chesnay Schepler 于2022年10月10日周一 19:13写道：
> > I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut
>
> But that's not possible. If
really very sad - as far as I know this happens for the first time,
attitude of new Ververica?
Hopefully immerok may resume the open mentality of data artisans.
Günter
On 10.10.22 11:26, Martijn Visser wrote:
Hi Günter,
I've understood that only the keynotes were recorded and not the other
> I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut
But that's not possible. If it were that simple there would have never
been a need to introduce another metric in the first place.
It's a rather fundamental issue with how the new sinks work, in that
they emit data to the
On 10/10/2022 11:24, Martijn Visser wrote:
Sidenote: metric names are not mentioned in the FLIP process as a
public API. Might make sense to have a separate follow-up to add that
to the list (I do think we should list them there).
That's a general issue we have. There's a lot of things we _
Thanks everyone for joining the discussion!
> Do you have any idea what has happened in the process here?
The discussion in this PR [1] shows some details and could be helpful to
understand the original motivation of the renaming. We do have a test case for
guarding metrics but unfortunaly the
I think everyone would be happier with the videos published on Youtube but
it's unfortunately at the discretion of the organizer.
At this time they decided against it for some reason.
Gyula
On Mon, Oct 10, 2022 at 11:57 AM Martin wrote:
> Hey,
>
> that's sad. Is it possible for future Flink
Hey,
that's sad. Is it possible for future Flink Forwards to record again and
publish all sessions?
Best regards
Martin
Am 10. Oktober 2022 11:26:26 MESZ schrieb Martijn Visser
:
>Hi Günter,
>
>I've understood that only the keynotes were recorded and not the other
>sessions.
>
>Best regards,
+1 for reverting these changes in Flink 1.16.
For 1.15.3, can we make these metrics available via both names (numXXXOut
and numXXXSend)? In this way we don't break it for those who already
migrated to 1.15 and numXXXSend. That means we still need to change
SinkWriterOperator to use another metric
Hi Günter,
I've understood that only the keynotes were recorded and not the other
sessions.
Best regards,
Martijn
On Sun, Oct 9, 2022 at 4:10 PM guenterh.lists
wrote:
> Sorry if this question was already posted
>
> By now only a few videos of the conference were published (mainly the
>
Hi Qingsheng,
Do you have any idea what has happened in the process here? Do we know why
they were changed? I was under the impression that these metric names were
newly introduced due to the new interfaces and because it still depends on
each connector implementing these.
Sidenote: metric names
Thanks Qingsheng for starting this thread.
+1 on reverting sink metric name and releasing 1.15.3 to fix this inconsistent
behavior.
Best,
Leonard
> 2022年10月10日 下午3:06，Jark Wu 写道：
>
> Thanks for discovering this problem, Qingsheng!
>
> I'm also +1 for reverting the breaking changes.
>
Hi，不重启作业的情况下，修改配置，实时改变ExecutionGraph目前是不支持的。
--
Best！
Xuyang
在 2022-09-27 08:36:53，"casel.chen" 写道：
>我有一个数据同步场景是希望通过修改配置来实时动态修改数据同步的目标，例如使用flink
>cdc将mysql中的变更数据实时同步进kafka，如果后来业务又要求同一份数据再同步进mongodb的话，我是否可以通过修改同步配置来达到不停止原来作业来动态修改数据同步的目标（由一个变多个）？又或者是flink
>
Hi， 目前应该是不行的
在 2022-09-26 23:27:05，"casel.chen" 写道：
>flink cdc能否同步DDL语句以实现schema同步? 例如create table , create index, truncate table等
ck“打”完是啥意思。
Congxian Qiu 于2022年10月10日周一 15:11写道：
>
> Hi
> 可以的话也同步下相关的计算逻辑，从 checkpoint 恢复后的统计结果可能会和计算逻辑有关
> Best,
> Congxian
>
>
> Hangxiang Yu 于2022年10月10日周一 14:04写道：
>
> > 是什么值下跌呢？哪个metric吗？
> >
> > On Mon, Oct 10, 2022 at 1:34 PM 天下五帝东 wrote:
> >
> > > Hi:
> > >
Thanks both for your advice, I will give them a try!
From: Schwalbe Matthias
Sent: 10 October 2022 08:35
To: 仙路尽头谁为峰 ; Qing Lim
Cc: User
Subject: RE: Re:Question about Flink Broadcast State event ordering
Hi Qing again,
Another point to consider: broadcast streams are subject to
Hi Qing again,
Another point to consider: broadcast streams are subject to watermarking. i.e.
* You can wait to process the broadcast records only after the watermark
passed, then
* order those records by time
* keep all broadcast records where the watermark not yet passed in some
Hi
可以的话也同步下相关的计算逻辑，从 checkpoint 恢复后的统计结果可能会和计算逻辑有关
Best,
Congxian
Hangxiang Yu 于2022年10月10日周一 14:04写道：
> 是什么值下跌呢？哪个metric吗？
>
> On Mon, Oct 10, 2022 at 1:34 PM 天下五帝东 wrote:
>
> > Hi:
> > 各位大佬们，flink任务从状态恢复时，统计值有一个明显的下跌，等ck打完后，统计值才会上升，这种情况有什么好的解决方案吗？
>
>
>
> --
> Best,
> Hangxiang.
Thanks for discovering this problem, Qingsheng!
I'm also +1 for reverting the breaking changes.
IIUC, currently, the behavior of "numXXXOut" metrics of the new and old
sink is inconsistent.
We have to break one of them to have consistent behavior. Sink V2 is an
evolving API which is just
是什么值下跌呢？哪个metric吗？
On Mon, Oct 10, 2022 at 1:34 PM 天下五帝东 wrote:
> Hi:
> 各位大佬们，flink任务从状态恢复时，统计值有一个明显的下跌，等ck打完后，统计值才会上升，这种情况有什么好的解决方案吗？
--
Best,
Hangxiang.
37 matches
Mail list logo
