Hi Alexey,
Thanks for confirming.
Can you send me a copy of the exception stack trace? That could help me
pinpoint the exact issue.
Cheers,
Gordon
On Fri, Mar 5, 2021 at 2:02 PM Alexey Trenikhun wrote:
> Hi Gordon,
> I was using RocksDB backend
> Alexey
>
> --
>
补充一下，我是继承Trigger自己实现了一个定时器，然后只重写了三个抽象方法
-- 原始邮件 --
发件人: smq <374060...@qq.com
发送时间: 2021年3月8日 15:30
收件人: user-zh
Trigger 抽象类其中有两个方法，onProcessingTime （）和OnEventTime （），在源码描述中，当trigger
context设置的定时器触发时会调用上面两个方法。但是在实际测试中，发现，就算没有设置定时器，上面的方法也能在水印超过窗口最大时间之后触发。请问有人知道了解过这个吗。
Hi,
可以排查下 GC 情况，频繁 FGC 也会导致这些情况。
Best,
jjiey
> 2021年3月8日 14:37，yidan zhao 写道：
>
> 如题，我有个任务频繁发生该异常然后重启。今天任务启动1h后，看了下WEB-UI的检查点也没，restored达到了8已经。然后Exception页面显示该错误，估计大多数都是因为该错误导致的restore。
> 除此外，就是 ‘Job leader for job id eb5d2893c4c6f4034995b9c8e180f01e lost
> leadership’ 错导致任务重启。
>
>
Hi!
I'm running a backfill Flink stream job over older data. It has multiple
interval joins. I noticed my checkpoint is regularly gaining in size. I'd
expect my checkpoints to stabilize and not grow.
Is there a setting to prune useless data from the checkpoint? My top guess
is that my
如题，我有个任务频繁发生该异常然后重启。今天任务启动1h后，看了下WEB-UI的检查点也没，restored达到了8已经。然后Exception页面显示该错误，估计大多数都是因为该错误导致的restore。
除此外，就是 ‘Job leader for job id eb5d2893c4c6f4034995b9c8e180f01e lost
leadership’ 错导致任务重启。
下面给出刚刚的一个错误日志（环境flink1.12，standalone集群，5JM+5TM，JM和TM混部在相同机器）：
2021-03-08 14:31:40
Hi,
I tried with the standalone session (sorry I do not have a yarn cluster in
hand) and it seems that
the flink cluster could startup normally. Could you check the log of
NodeManager to see the detail
reason that the container does not get launched? Also have you check if there
are some spell
还有没有大佬解释下，我最近又遇到这个问题了，而且很频繁。任务启动1小时restored达到了8。
yidan zhao 于2021年3月3日周三 下午2:58写道：
> Mark下。这个问题我也遇到多次，看过一个xintognsongn的回复，由于网络、zk可用性等问题会导致。不够一般会自动恢复。
>
> 史 正超 于2020年12月7日周一 下午10:13写道：
>
>> 8 个slot，8个并行度，jm是2G，tm配置的是8G，其它的任务配置是
>> ```
>> SET 'execution.checkpointing.interval' = '5min';
>> SET
您好！
hive.metastore.sasl.enabled 是true
启动sql client的时候，可以正常读取到认证信息，并读取metastore的表名。
读和写，认证就失败了。
---原始邮件---
发件人: "Rui Li"
The company has provided an algorithm written in C++, which has been packaged
into a.so file. I have built a SpringBoot project, which uses JNI to operate
the algorithm written in C++. Could you please tell me how to call it in Flink?
Do i need to define operators, chains of operators?
我现在有个需求，公司提供了一个用c++写的算法，已经包成了.so文件，我已经建了一个SpringBoot工程，使用JNI操作C++写的算法，请问我怎样在Flink里调用呢？需要定义算子、算子链吗？
Hi Yun,
Thanks for the response. I checked the mounts and only the JM's and TM's
are mounted with this EFS. Not sure how to debug this.
Thanks
On Sun, Mar 7, 2021 at 8:29 PM Yun Gao wrote:
> Hi Navneeth,
>
> It seems from the stack that the exception is caused by the underlying EFS
> problems
Hi Navneeth,
It seems from the stack that the exception is caused by the underlying EFS
problems ? Have you checked
if there are errors reported for EFS, or if there might be duplicate mounting
for the same EFS and others
have ever deleted the directory?
Best,
Yun
也可能是需要这个fix：https://issues.apache.org/jira/browse/FLINK-20913
On Mon, Mar 8, 2021 at 12:11 PM Rui Li wrote:
> Hi，
>
>
> 从你发的stacktrace来看，走到了set_ugi方法说明client认为server没有开启kerberos。确认一下你HiveCatalog这边指定的hive-site.xml是否配置正确呢，像hive.metastore.sasl.enabled是不是设置成true了？
>
> On Sun, Mar 7, 2021 at 5:49 PM
Hi，
从你发的stacktrace来看，走到了set_ugi方法说明client认为server没有开启kerberos。确认一下你HiveCatalog这边指定的hive-site.xml是否配置正确呢，像hive.metastore.sasl.enabled是不是设置成true了？
On Sun, Mar 7, 2021 at 5:49 PM 861277...@qq.com <861277...@qq.com> wrote:
> 环境：
> flink1.12.1
> hive2.1.0
> CDH6.2.0
>
>
> 【问题描述】
>
有没有大佬有思路可以参考下？
--
Sent from: http://apache-flink.147419.n8.nabble.com/
Hi,
不好意思回复这么晚。关于pandas
udaf，我有专门测试过框架层的开销(函数用普通的均值计算)。和java相比，差距也就3，4倍左右，具体可以参考代码[1]。关于你这个代码，我怀疑是因为你函数实现的问题。你这个函数构造df是会有额外的开销。你为啥不直接使用j来进行计算。当然了，你也可以根据调整一些参数来提高性能，比如python.fn-execution.bundle.size和python.fn-execution.bundle.time，具体可以参考文档[2]。
[1]
个人理解是不会重复驱动执行的，具体你可以测试一下，但是从底层原理上讲，我也讲不了。
发件人： lp
发送时间： 2021-03-05 17:31
收件人： user-zh
主题： flink dataStream多次sink DAG重复驱动执行？
有个疑问，
如下程序片段：
--
Properties properties = new Properties();
properties.setProperty("bootstrap.servers",kafkaAddr);
I think you want to submit a Flink python job to the existing session
cluster.
Please ensure the session cluster is created with proper service exposed
type[1].
* LoadBalancer for the cloud environment
* NodePort for self managed K8s cluster
* ClusterIP for the K8s internal submission, which means
Hi, Yuval, Jark, Timo.
Currently the watermark push down happens in the logical rewrite phase but
the filter push down happens in the local phase, which means the planner
will first check the Filter push down and then check the watermark push
down.
I think we need a rule to transpose between the
Hi Hemant,
I don't see any problem in your settings. Any exceptions suggesting why TM
containers are not coming up?
Thank you~
Xintong Song
On Sat, Mar 6, 2021 at 3:53 PM bat man wrote:
> Hi Xintong Song,
> I tried using the java options to generate heap dump referring to docs[1]
> in
Hi,
退订请发邮件到 user-zh-unsubscr...@flink.apache.org
详细的可以参考 [1]
[1] https://flink.apache.org/zh/community.html#section-1
发件人： zenglong chen
发送时间： 2021-03-08 10:00
收件人： user-zh
主题： 退订
退订
退订
如题
hezongji...@qq.com
from pyflink.table import EnvironmentSettings, StreamTableEnvironment
env_settings =
EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
table_env = StreamTableEnvironment.create(environment_settings=env_settings)
table = table_env.from_elements([(1, 'Hi'), (2,
Hey Rion,
you don't need special access to Flink's Jira: Any JIra user is assignable
to tickets, but only committers can assign people.
For low hanging fruits, we have a "starter" label to tag those tickets. I
also recommend keeping an eye on Jira tickets about topics you are
experienced with /
Hey folks,
The community here has been awesome with my recent questions about Flink, so
I’d like to give back. I’m already a member of the ASF JIRA but I was wondering
if I could get access to the Flink Project.
I’ve contributed a good bit to Apache Beam in the past, but I figured that I’ll
Hi Jark,
Even after implementing both, I don't see the watermark being pushed to the
tablesource in the logical plan and avoids predicate pushdown from running.
On Sun, Mar 7, 2021, 15:43 Jark Wu wrote:
> Hi Yuval,
>
> That's correct you will always get a LogicalWatermarkAssigner if you
>
是这样的，可以产生，问题从watermarksState里面拿出来的时间戳，会变成是默认值，全局除过这里有设置过，再无任何关于watermark的逻辑。
[image: image.png]
val dataLoadStream = data
.map(new EventMapFunction(config))
// Add watermark
.assignTimestampsAndWatermarks(
WatermarkStrategy
可以中途产生，走这个接口
org.apache.flink.streaming.api.datastream.DataStream#assignTimestampsAndWatermarks(org.apache.flink.api.common.eventtime.WatermarkStrategy)
麻烦贴一下你加 watermark 的代码和 pipeline 看一下啥情况
Best,
tison.
Xavier 于2021年3月7日周日 下午7:51写道：
>
Hi Yuval,
That's correct you will always get a LogicalWatermarkAssigner if you
assigned a watermark.
If you implement SupportsWatermarkPushdown, the LogicalWatermarkAssigner
will be pushed
into TableSource, and then you can push Filter into source if source
implement SupportsFilterPushdown.
想问下社区，watermark必须加在数据源上吗？顺便想问下一般用什么方式来调watermark，我自己本地有发现加在map
function之后，watermark会自动重置为默认值的情况。
谢谢！
--
Best Regards,
*Xavier*
Thanks Tamir,
I was having some issues connecting from my IDE (solved) but this is really
helpful.
On Sat, Mar 6, 2021, 23:04 Tamir Sagi wrote:
> I had a typo in my previous answer, the env name was missing an 'S'
>
> ENABLE_BUILT_IN_PLUGIN --> ENABLE_BUILT_IN_PLUGIN*S*
> once again, the value
环境：
flink1.12.1
hive2.1.0
CDH6.2.0
【问题描述】
在没开启Kerberos认证时，可以正常读写hive表
开启Kerberos认证后，
启动时可以正常读取到hive metastore的元数据信息，读写不了表。
【sql-client.sh embedded】
Flink SQL show tables;
dimension_table
dimension_table1
test
Flink SQL select * from test;
[ERROR] Could not execute SQL statement. Reason:
可以看下flink源码的flink k8s模块，里面的test pachage下有需求flink k8s的使用姿势，希望对你有帮助。
DanielGu <610493...@qq.com> 于2021年3月7日周日 下午4:34写道：
> >有的，通过 FLINK 和 YARN 或 k8s
> 的接口进行编程，管理元数据，管理用户文件，支持提交作业及之后管理作业状态，这是许多公司应用Flink
> 的实现方式。
>
> 请问有什么可以参考的资料吗?有相关意愿,不知道从哪里下手,希望整个 flink on k8s
>
>
>
> --
> Sent from:
>有的，通过 FLINK 和 YARN 或 k8s 的接口进行编程，管理元数据，管理用户文件，支持提交作业及之后管理作业状态，这是许多公司应用Flink
的实现方式。
请问有什么可以参考的资料吗?有相关意愿,不知道从哪里下手,希望整个 flink on k8s
--
Sent from: http://apache-flink.147419.n8.nabble.com/
36 matches
Mail list logo
