Just to add a bit of context: The first-level members all-exceptions,
root-exceptions, truncated and timestamp have been around for a longer
time. The exceptionHistory was added in Flink 1.13. As part of this change,
the aforementioned members were deprecated (see [1]). We kept them for
Hi John,
Going with processing time is perfectly sound if the results meet your
requirements and you can easily live with events misplaced into the wrong time
window.
This is also quite a bit cheaper resource-wise.
However you might want to keep in mind situations when things break down
当然不会。
杨浩 于2021年11月25日周四 下午6:09写道：
> keyed state重新指定并行度后，比如2->4或者4->2，请问下是不是会丢失部分老的状态？
可以配置开启SSL认证的，只不过开启之后登陆会很复杂，所以为了方便登陆，推荐用nginx做一层代理，这样可以将认证方式替换为http basic
authentication等。然后nginx和flink之间仍然采用SSL双向认证即可。
如果不介意浏览器本身登陆复杂，可以直接开启SSL双向认证，但是需要浏览器安装本地client证书。
Mabin 于2021年11月26日周五 上午10:25写道：
> flink web UI无认证登陆问题是怎样解决的呢？
>
> 官网上写的是：Simple mutual authentication may be enabled by
我认为这个应该是bug。
yidan zhao 于2021年11月26日周五 上午11:18写道：
> 如题，注意，非检查点本身失败，而是检查点完成后导致任务失败。
>
> 目前跟进报错是PartitionTimeCommitTrigger.committablePartitions部分如下代码报的异常：
>
> if (!watermarks.containsKey(checkpointId)) {
> throw new IllegalArgumentException(
> String.format(
>
同样的代码逻辑，是用flink 1.10.1版本，在flink
ui界面可以看到local聚合和global聚合。但是更新为1.12.1版本，则看不到，想了解一下，如何确认开启minibatch功能成功呢？
Hi Rahul
>From your description I guess maybe you could try different flink.yaml(one
for server and another for client).
I am not an expert about SSL and security stuff. So please correct me if I
am wrong.
Best,
Guowei
On Wed, Nov 24, 2021 at 3:54 AM Rahul wrote:
> Hello,
> I am trying to
Hi Joseph
Would you like to give more details about the error message?
Best,
Guowei
On Thu, Nov 25, 2021 at 2:59 AM Joseph Lorenzini
wrote:
> Hi all,
>
>
>
> I have an implementation of KafkaDeserializationSchema interface that
> deserializes a kafka consumer record into a generic record.
>
>
你好：
我在over窗口中使用了RANGE BETWEEN INTERVAL '1000'
SECOND(4)，运行时报错，错误如邮件标题，但是官网中这个精度要超过4，有人知道为什么吗？
您好，远程调用这样操作是不行的呢，这样只适合本地调用的。感谢。
| |
王健
|
|
13166339...@163.com
|
签名由网易邮箱大师定制
在2021年11月26日 11:18，liuyehan 写道：
您好：
可以尝试下在/etc/profile文件里配置
export HADOOP_CLASS PATH=｀hadoop classpath`
我是这样解决的。
| |
lyh1067341434
|
|
邮箱：lyh1067341...@163.com
|
发自 网易邮箱大师
回复的原邮件
| 发件人 |
如题，注意，非检查点本身失败，而是检查点完成后导致任务失败。
目前跟进报错是PartitionTimeCommitTrigger.committablePartitions部分如下代码报的异常：
if (!watermarks.containsKey(checkpointId)) {
throw new IllegalArgumentException(
String.format(
"Checkpoint(%d) has not been snapshot. The
watermark information
您好：
可以尝试下在/etc/profile文件里配置
export HADOOP_CLASS PATH=｀hadoop classpath`
我是这样解决的。
| |
lyh1067341434
|
|
邮箱：lyh1067341...@163.com
|
发自 网易邮箱大师
回复的原邮件
| 发件人 | 王健<13166339...@163.com> |
| 日期 | 2021年11月26日 10:59 |
| 收件人 | user-zh |
| 抄送至 | |
| 主题 | flink远程调用时环境变量问题 |
大佬们：
您好，ssh远程调用，/etc/profile配置是不起作用的呢
| |
王健
|
|
13166339...@163.com
|
签名由网易邮箱大师定制
在2021年11月26日 11:12，Mabin 写道：
在/etc/profile里面配的
发自我的iPhone
在 2021年11月26日，上午11:07，王健 <13166339...@163.com> 写道：
大佬们：
Hi Jonas,
Previously Flink indeed does not support checkpoints after some tasks finished.
In 1.14 we implement a first version for this feature (namely
https://issues.apache.org/jira/browse/FLINK-2491),
and it could be enabled by set
在/etc/profile里面配的
发自我的iPhone
> 在 2021年11月26日，上午11:07，王健 <13166339...@163.com> 写道：
>
>
>
> 大佬们：
>
> 远程调用flink启动任务，如何解决hadoop的环境变量问题呢，像java，hbase其他的环境变量都可以通过在flink-conf.yaml配置文件里配置，但是hadoop配置env.hadoop.conf.dir不起作用。
>可能是需要增加export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop
>
大佬们：
远程调用flink启动任务，如何解决hadoop的环境变量问题呢，像java，hbase其他的环境变量都可以通过在flink-conf.yaml配置文件里配置，但是hadoop配置env.hadoop.conf.dir不起作用。
可能是需要增加export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop
classpath`，但是这个是不能在flink-conf.yaml配置吧
急求解决，万分感谢。
| |
王健
|
|
13166339...@163.com
|
签名由网易邮箱大师定制
大佬们：
远程调用flink启动任务，如何解决hadoop的环境变量问题呢，像java，hbase其他的环境变量都可以通过在flink-conf.yaml配置文件里配置，但是hadoop配置env.hadoop.conf.dir不起作用。
可能是需要增加export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop
classpath`，但是这个是不能在flink-conf.yaml配置吧
急求解决，万分感谢
| |
王健
|
|
13166339...@163.com
|
签名由网易邮箱大师定制
I'm trying to track down a couple errors I've hit related to key groups. I
want to verify that all of my keys have stable hashes. I tried to print
out the execution plan but it doesn't contain enough info.
大佬们：
远程调用flink启动任务，如何解决hadoop的环境变量问题呢，像java，hbase其他的环境变量都可以通过在flink-conf.yaml配置文件里配置，但是hadoop配置env.hadoop.conf.dir不起作用。
可能是需要增加export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop
classpath`，但是这个是不能在flink-conf.yaml配置吧
急求解决，万分感谢
| |
王健
|
|
13166339...@163.com
|
签名由网易邮箱大师定制
flink web UI无认证登陆问题是怎样解决的呢？
官网上写的是：Simple mutual authentication may be enabled by configuration if
authentication of connections to the REST endpoint is required, but we
recommend to deploy a “side car proxy”: Bind the REST endpoint to the loopback
interface (or the pod-local interface in
Well what I'm thinking for 100% accuracy no data loss just to base the
count on processing time. So whatever arrives in that window is counted. If
I get some events of the "current" window late and they go into another
window it's ok.
My pipeline is like so
browser(user)->REST
I included the user ML in the thread.
@users Are you still using Zookeeper 3.4? If so, were you planning to
upgrade Zookeeper in the near future?
I'm not sure about ZK compatibility, but we'd also upgrade Curator to
5.x, which doesn't support ookeeperK 3.4 anymore.
On 25/11/2021 21:56,
Hi community,
I am using Flink 1.11 + Java 8 and I was updating my application from
Spring boot 1 to spring boot 2.6. Then my Integration Test of Flink + Kafka
started giving me this error: "java.lang.NoClassDefFoundError:
scala/concurrent/ExecutionContext$parasitic$". The older version of spring
Hi all,
I have been struggling with this issue for a couple of days now.
Checkpointing appears to fail as the Task Source ( kinesis stream in this
case) appears to be in a FINISHED state.
Excerpt from Jobmanager logs:
2021-11-25 12:52:00,479 INFO
Hi John,
… just a short hint:
With datastream API you can
* hand-craft a trigger that decides when an how often emit intermediate,
punctual and late window results, and when to evict the window and stop
processing late events
* in order to process late event you also need to specify
Can you clarify when each exception occurs? Is the latter causing the
first one?
There are a few possible explanations.
One could be an implementation issue in Micrometer where they use the
JVMs common pool. In this case a thread may or may not use the user-code
classloader.
Regarding
Thanks. Using, data streaming.
On Wed, 24 Nov 2021 at 20:56, Caizhi Weng wrote:
> Hi!
>
> Are you using the datastream API or the table / SQL API? I don't know if
> datastream API has this functionality, but in table / SQL API we have the
> following configurations [1].
>
>-
root-exception: The last exception that caused a job to fail.
all-exceptions: All exceptions that occurred the last time a job failed.
This is primarily useful for completed jobs.
exception-history: Exceptions that previously caused a job to fail.
On 25/11/2021 11:52, Mahima Agarwal wrote:
Hi
Hi Team,
Please find the query below regarding exceptions
API(/jobs/:jobid/exceptions)
In response of above rest api:
Users are getting 3 types of exceptions:
1. exceptionHistory
2. all-exceptions
3. root-exception
What is the purpose of the above 3 exceptions?
Any leads would be
keyed state重新指定并行度后，比如2->4或者4->2，请问下是不是会丢失部分老的状态？
You're welcome!
Piotrek
śr., 24 lis 2021 o 17:48 Shazia Kayani napisał(a):
> Hi Piotrek,
>
> Thanks for you message!
>
> Ok that does sound interesting and is a approach I had not considered
> before, will take a look into and further investigate
>
>
> Thank you!
>
> Best wishes,
>
> Shazia
>
Will Flink loss some old Keyed State when changing the parallelism, like 2 ->
5, or 5->3?
我根据webui的task2节点（coordinator）的watermark来看，部分情况会出现task2被反压。
我目前理解是，ckpt完成后，才开始compcat。如果task2出现反压，意味着ckpt2发生时，ckpt1对应的compact可能还未完成。
不清楚理解是否正确呢？
yidan zhao 于2021年11月25日周四 下午3:34写道：
> Task3就是compact，具体jstack可能有点复杂，本身每个算子在每个机器上都有，jstack也不好分辨具体那个线程。
>
>
Hello,all
请教下各位大佬，能不能提供一些思路，能够采用简单的方式去构建一些，数据倾斜，source、sink join 反压的现象。
| |
稚远
|
|
drewfrank...@163.com
行远自逊，笃行不怠
|
签名由网易邮箱大师定制
34 matches
Mail list logo
