参考
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#starting-a-flink-session-on-kubernetes
基于命令创建k8s flink session集群（flink1.13.6）：./bin/kubernetes-session.sh
-Dkubernetes.cluster-id=my-first-flink-cluster 。创建成功，最后提示一句 Create
flink
Hi Peter,
Have you compared the DAT topologies in 1.15 / 1.14?
I think it's expected that "Records Received", "Bytes Sent" and "Records
Sent" are 0. These metrics trace the internal data exchanges between Flink
tasks. External data changes, i.e., source reading / sink writing data from
/ to
这里有 flink table store 的设计文档，你可以了解下。
https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage
Best regards,
Xianxun
On 04/6/2022 16:56，LuNing Wang wrote：
Hi,
Table store是存储，应和数据湖类似
Best,
LuNing Wang
yidan zhao 于2022年4月6日周三 16:55写道：
看官网出来个
You have not configured the tumbling window at all. Please refer to [1] for
more details.
Regards,
Dian
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/queries/window-agg/#group-window-aggregation
On Wed, Apr 6, 2022 at 10:46 PM ivan.ros...@agilent.com <
Thank you for the clarification! After some discussion, I think we'll be
using processing time as an alternative for our use case.
Just for my education, if I really need ingestion-time. It seem like I can
get it by either of the below approach?
// 1. an ingestion time watermark strategy
new
Hello,
We are using Flink 1.13.6, Application Mode, k8s HA. To upgrade job, we use
POST, url=http://gsp-jm:8081/jobs//savepoints,
then we wait for up to 5 minutes for completion, periodically pulling status
(GET,
Hi there,
I just successfully upgraded our Flink cluster to 1.15.0 rc0 - also the
corresponding job is running on this version. Looks great so far!
In the Web UI I noticed some metrics are missing, especially "Records
Received", "Bytes Sent" and "Records Sent". Those were shown in v 1.14.4.
See
Hello,
I'm trying to understand tumbling windows at the level of the python table api.
For this short example:
Input csv
Print output
2022-01-01 10:00:23.0, "data line 3"
2022-01-01 10:00:24.0, "data line 4"
2022-01-01 10:00:18.0, "data line 1"
2022-01-01
Hi,
We have an in-house platform that we want to integrate with external
clients via HDFS. They have lots of existing files and they continuously
put more data to HDFS. Ideally, we would like to have a Flink job that
takes care of ingesting data as one of the requirements is to execute SQL
on top
HI, Kevin
I have not reproduced this problem.
What is the impact of this problem? Can't get this parameter correctly in user
main method?
Could you provide a screenshot of the JobManager configuration on the UI
> 2022年4月2日 上午10:23，Kevin Lee 写道：
>
> It's a typo
>
> I run this demo on yarn
Hey Matthew,
Thanks also for sharing the code that you're working on. What are your
plans with the connector? I could imagine that others would also be
interested, so perhaps you wanted to add it to https://flink-packages.org/
?
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
Happy to help,
Let us know if it helped in your use case.
On Tue, Apr 5, 2022 at 1:34 AM Yaroslav Tkachenko
wrote:
> Hi Marios,
>
> Thank you, this looks very promising!
>
> On Mon, Apr 4, 2022 at 2:42 AM Marios Trivyzas wrote:
>
>> Hi again,
>>
>> Maybe you can use the
>>
Hi,
Hive sink不支持 upsert写入，只能INSERT写入，你怎么设置都不行，一般这种情况可以使用hudi和iceberg作为Sink接受
upsert数据。
Best,
LuNing Wang
799590...@qq.com.INVALID <799590...@qq.com.invalid> 于2022年4月6日周三 14:37写道：
> 问题：flink1.1.36用SQL方式如何设置输出到hive表为upsert模式？
>
> flink：1.13.6
> hive：1.1.1
> hadoop：2.6.0-cdh5.16.2
>
>
Thanks Guowei! I'll check it out.
Best,
Zhanghao Chen
From: Guowei Ma
Sent: Wednesday, April 6, 2022 16:01
To: Zhanghao Chen
Cc: user@flink.apache.org
Subject: Re: Why first op after union cannot be chained?
Hi Zhanghao
AFAIK, you might to see the
Hi,
Table store是存储，应和数据湖类似
Best,
LuNing Wang
yidan zhao 于2022年4月6日周三 16:55写道：
> 看官网出来个 flink table store，和 flink table、flink sql 那一套有啥区别呢？
>
看官网出来个 flink table store，和 flink table、flink sql 那一套有啥区别呢？
Dear all,
I'm trying to submit SQL jobs to session cluster on k8s, which need
external local udfs jars.
I meet some proplems below:
- 'pipeline.jars' will be overwritten by '-j' option which only accepts
one jar.
- 'pipeline.classpaths' will not be uploaded, so local files could not
Hi Zhanghao
AFAIK, you might to see the `StreamingJobGraphGenerator` not the
`JobGraphGenerator` which is only used by the old flink stream sql stack.
>From comment of the `StreamingJobGraphGenerator::isChainableInput` the `an
union operator` does not support chain currently.
Best,
Guowei
On
I'm not quite familiar with ES conector. However, I guess you could check
if there is data going into the sink connector. One way to achieve this is
to set the pipeline.operator-chaining as false and then you could see the
count of input elements for the sink operator.
PS: Just removed the
I think you should use [1] or [2] instead. See [3] for more details.
PS: This question is more fit for the user mailing list.
Regards,
Dian
[1]
https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6_2.11/1.14.4/flink-sql-connector-elasticsearch6_2.11-1.14.4.jar
问题：flink1.1.36用SQL方式如何设置输出到hive表为upsert模式？
flink：1.13.6
hive：1.1.1
hadoop：2.6.0-cdh5.16.2
纯SQL的方式，使用kafka作为source，中间的转换会有DISTINCT 或者 GROUP 操作，将计算结果sink到hive表，会报下面的错误
doesn't support consuming update changes which is produced by node
GroupAggregate(groupBy=[id, user_id, status, EXPR$3]
Thanks Robert!
All working now.
Turns out an incorrect log4j2 configuration was slurping up a warning about
converting org.apache.avro.util.Utf8 to the internal flink data type.
I've drawn some inspiration from flink-avro and added a converter that
converts and re-orders the Avro encoded
22 matches
Mail list logo
