Re: Custom log appender for YARN

Hi Gyula, I guess it should work if you put log appender jar under $flink/lib folder. There are two different kinds of classloader in your case. One is for Flink framework, the other is for user code. The framework classloader is the parent of user classloader. The parent classloader could not

Re: Error while running flink job on local environment

Hi Vinayak, If `StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(2)` works for your case, you could try as below. `StreamExecutionEnvironment.setDefaultLocalParallelism(2);` `StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();`

Fwd: Flink metrics with parallel operator

Hi Sibendu, Have you checked the document of metrics[1]? It might be helpful. BTW, I think it's better to ask in user mailing list. So forward it there. 1. https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#system-scope -- Forwarded message - From:

请教 如何使用TableAPI connector将一个字段定义为rowtime属性

代码如下： StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment(); executionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); StreamTableEnvironment tableEnvironment = StreamTableEnvironment.create(executionEnvironment); String[]

Re: Why is the size of each checkpoint increasing?

Hi Andrew >From Flink doc[1], there is "Flink guarantees removal only for time-based windows and not for other types, *e.g.* global windows (see Window Assigners ). ", Seems the state of the

Improve data type validation in Flink inputformats according to avro schemas

Hi all, Following this JIRA ticket opened last year : https://issues.apache.org/jira/browse/FLINK-9813 I use to run Flink, currently 1.8, in my professional activity (at DCbrain, a French B2B software editor for physical fluid networks operators) and we wish to share some of our experience with

Custom log appender for YARN

Hi All! We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem. We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path. However when the YARN cluster

Re: Support priority of the Flink YARN application in Flink 1.9

Thanks for bringing this up, Boxiu. The problem make sense to me. For me the concern is should we limit the priorities to 1-9 or not. I think it would be good to open a jira issue and have the discussion there. Thank you~ Xintong Song On Wed, Jul 31, 2019 at 12:22 PM tian boxiu wrote:

Support priority of the Flink YARN application in Flink 1.9

To: user@flink.apache.org Title: Support priority of the Flink YARN application in Flink 1.9 Hello everyone, Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted

Re: Checkpoints very slow with high backpressure

Hi, For Flink 1.8 (and 1.9) the only thing that you can do, is to try to limit amount of data buffered between the nodes (check Flink network configuration [1] for number of buffers and or buffer pool sizes). This can reduce maximal throughput (but only if the network transfer is a significant

Re: 请教Flink SQL watermark遇到未来时间的处理问题

hi,智笙： 感谢提供解决思路，目前我这边还尝试了几种可行的方案： 1.在kafka反序列化的时候，判断kafka中日期字段的值，如果超过当前时间太多，则丢弃，或者重置为当前时间（重置其实可能导致正常数据丢失）。 2.自定义一个watermark，当时间大于当前时间太多的时候，不更新当前的watermark，这样在watermark达到这条未来时间的时间点后，也会将这条数据纳入窗口计算，这种其实是比较理想的。但是这种没有完全的测试，感觉数据会一直存放在内存中，不知道会不会引起其他问题。

Re: Error while running flink job on local environment

Hi Andrey and Jeff, Thank you for the reply. I agree with Jeff. My concern is to use different code for local and non-local deployments. It would help if StreamExecutionEnvironment.getExecutionEnvironment() works for both local and cluster deployments. Thanks & Regards, Vinayak On Wed,