Hi;I am trying to run simple worcount application using the kafka consumer so
that the data sources remain available and I can trigger checkpoints. When
deploying the application over flink 1.7.2 cluster, I am facing the
*java.lang.NoClassDefFoundError:
Hi
Could you please check whether the key and namespace are the same? when
getting a result for the value state, Flink will use current key and
current namespace to retrieve the result.
Best,
Congxian
Parth Sarathy 于2019年12月22日周日 下午9:54写道：
> Hi,
>
> I am using flink 1.8.2, event time stream
Hi:
I wanted to find out what's the best way of collecting Flink metrics using
Prometheus in a streaming application on EMR/Hadoop.
Since the Flink streaming jobs could be running on any node - is there any
Prometheus configuration or service discovery option available that will
dynamically
Hi
I encountered same problem before and found that the `NoClassDefFoundError`
is not the root cause, maybe you can check the jm/tm log to see if this is
your case.
Best,
Congxian
syed 于2019年12月24日周二 下午7:30写道：
> Hi; I am trying to run simple worcount application using the kafka
> consumer so
Hi all,
I use Kafka 0.10.0, Flink 1.9.0, why I can’t get flink consumer10 group
which I had configured. And I use KafkaConsumer not with Flink to consumer the
same topic, I can get the consumer group metadata. Thanks.
Kafka/bin/kafka-run-class kafka.admin.ConsumerGroupCommand
Typically , *NoClassDefFoundError* is caused by jar conflicts , it
means, there are two same class and not same version jar in your
classpath,
I suggest you can check which jar is this class from ,and then to confirm
is there more too than one jar in u classpath. if is and remove unuse jar .
The shutdown happened after the massive IO wait. I don't use any state
Checkpoints are disk based...
On Mon., Dec. 23, 2019, 1:42 a.m. Zhijiang,
wrote:
> Hi John,
>
> Thanks for the positive comments of Flink usage. No matter at least-once
> or exactly-once you used for checkpoint, it would
If you use rocksDB state backend, it might consume extra native memory.
Some resource framework cluster like yarn would kill the container if the
memory usage exceeds some threshold. You can also double check whether it
exists in your case.
Hi Mans,
IMO, the mechanism of metrics reporter does not depend on any deployment
mode.
>> is there any Prometheus configuration or service discovery option
available that will dynamically pick up the metrics from the Filnk job and
task managers running in cluster ?
Can you share more
Hi zjfplayer,
First, your user fat jar should not contains any flink dependents, them
should be "provided".
If your goal is to reduce the deployment time of task, and you are
currently using session mode, you can consider putting jars directly under
lib.
Best,
Jingsong Lee
On Mon, Dec 23, 2019
??flink stream api??ETL
1??tumble??watermark??10s??
topic 5000/s,??topic4000/s
CEP的 sql中order by ， 不会因为乱序导致不匹配。
在api中没用过，可以看看是否有对应的api
qishang zhong 于2019年12月23日周一 下午9:37写道：
> HI，大家好。
>
> 咨询一个问题，flink-training-exercises练习的工程里面
> com.ververica.flinktraining.solutions.datastream_java.cep.LongRidesSolution
>
> Pattern completedRides =
> Pattern.begin("start")
> .where(new
??slotremove???CPU??FULL
GC??
org.apache.flink.util.FlinkException: The assigned slot
bae00218c818157649eb9e3c533b86af_11 was removed.
at
Hi faaron,
Flink 1.9 中 -yn参数应该是不生效的，后续版本中已经删除了这个参数。
根据你的参数，在每个 TM 的内存为30G不变的情况下，每个 TM
中的slot个数（-ys）从5变成10，也就意味着平均每个slot占用的内存变为了原来的一半。
Flink 1.9 的sql batch 算子对 flink managed memory 是有确定的需求的，很可能是这个变化导致单个 slot
的managed memory无法满足算子的资源需求了。
Thank you~
Xintong Song
On Wed, Dec 25, 2019 at 11:09 AM
??metricrecordsInrecordsOut
??window??processfunction??filter??
??
.window(TumblingEventTimeWindows.of(org.apache.flink.streaming.api.windowing.time.Time.minutes(1)))
.process(new ProcessWindowFunction
??
window+processfunctionwatermark??
??inputprocessfunction??
??
https://www.imageoss.com/image/sTn2U
跑tpcds的query1: flink run -m yarn-cluster -d -p 100 -yn 20 -ys 5 -yjm 60g -ytm
30g 任务可以正常执行 flink run -m yarn-cluster -d -p 100 -yn 10 -ys 10 -yjm 60g -ytm
30g 任务在做hashjoin的时候就会失败 报错是No pooled slot available and request to
ResourceManager for new slot failed 搞不懂这有啥关系，求指教 faaron zheng
window算子会丢弃迟到数据，可以看看是不是这个原因。如果是的话，可以调整一下watermark生成策略，适当增加点延迟，可以看一下：BoundedOutOfOrdernessTimestampExtractor
> 在 2019年12月25日，上午10:39，1530130567 <1530130...@qq.com> 写道：
>
> 大佬们好：
>
> 最近使用window+processfunction处理数据，并加了watermark延迟处理。
>
> 发现当数据input到达一定峰值后就会出现数据丢失的现象，请问是processfunction处理能力不够导致的吗？
>
Hi zjfplayer,
First, your user fat jar should not contains any flink dependents, them
should be "provided".
If your goal is to reduce the deployment time of task, and you are
currently using session mode, you can consider putting jars directly under
lib.
Best,
Jingsong Lee
On Mon, Dec 23, 2019
这个应该不是root cause，slot was removed通常是tm挂掉了导致的，需要找下对应的tm日志看下挂掉的原因。
Thank you~
Xintong Song
On Tue, Dec 24, 2019 at 10:06 PM hiliuxg <736742...@qq.com> wrote:
> 偶尔发现，分配好的slot突然就被remove了，导致作业重启，看不出是什么原因导致?CPU和FULL GC都没有，异常信息如下：
>
> org.apache.flink.util.FlinkException: The assigned slot
>
使用基于EventTime 的 watermark处理数据通常会碰到两这么两种情况：
1. 数据因为乱序，迟到严重，会被丢弃，这个可以查看Side Out API [1]
2.
数据产生的事件时间比当前系统时间大，我称之为“超自然数据”，比如当前系统时间是10：37：55，但数据产生的事件时间可能是10：38：55，那么一旦有这类数据到达，将会使窗口提前触发计算，导致正常数据被当做迟到数据，因而被丢弃，这个处理方式是在assignWaterMark
之前过滤掉。
3. 建议: 如果是简单的ETL，尽量不要用EventTime 来处理数据
[1]
项目简述：从kafka取数据，每10秒一批，sink到mysql中的ETL
环境相关信息
flink运行模式：local
mysql的global variables中wait_timeout=28800
mysql客户端mysql-connector-java版本5.1.42
报错
org.apache.flink.streaming.runtime.tasks.TimerException:
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
Could not
hi，各位好：
hdfs目前是配置了高可用， namenode分别为 nn1(active),nn2(standby)
flink-conf.yaml中的配置如下：
state.checkpoints.dir: hdfs://nn1的ip:8020/flink/flink-checkpoints/
然后发布一个job，等checkpoint生成后，把 nn1 namenode 杀掉。
nn2会自动切换成active,nn1无法访问。
job就会不停的restart。
日志显示 failed on connection
不要写地址 写nameservice
--原始邮件--
发件人："LJY "
24 matches
Mail list logo
