Re: Is there a better way to debug ClassNotFoundException from FlinkUserCodeClassLoaders?

Thanks Congxian for the tip. Arthas looks great Hao Sun Team Lead 1019 Market St. 7F San Francisco, CA 94103 On Fri, Jan 4, 2019 at 5:42 PM Congxian Qiu wrote: > Hi, Hao Sun > > For debugging the `ClassNotFoundException`, maybe the Arthas[1] tool can > help. > > [1] Arthas

Re: S3 StreamingFileSink never completes multipart uploads

Hi Addison, >From the information that Nick provides, how can you be sure that the root cause is the same? Cheers, Kostas On Fri, Jan 4, 2019, 22:10 Addison Higham Hi Nick, > > This is a known issue with 1.7.0, I have an issue opened up here: > https://issues.apache.org/jira/browse/FLINK-11187

Re: Is there a better way to debug ClassNotFoundException from FlinkUserCodeClassLoaders?

Hi, Hao Sun For debugging the `ClassNotFoundException`, maybe the Arthas[1] tool can help. [1] Arthas : https://github.com/alibaba/arthas Hao Sun 于2019年1月3日周四 下午10:08写道： > > I am on Flink 1.7.1 and K8S. > I said "suddenly" because my program worked fine

Re: ConnectTimeoutException when createPartitionRequestClient

Hi Till, Thanks for your reply and help on this issue. I increased taskmanager.network.netty.client.connectTimeoutSec to 1200 which is 20 minutes. But it seems the connection not respects this timeout. In addition, I increase both taskmanager.network.request-backoff.max and

Re: Change Window Size during runtime

Hi, Rad We can not change the window size during runtime as Chesnay said. Could you please share the reason why would you want to change the window size during runtime? Rad Rad 于2019年1月3日周四 上午8:03写道： > Hi All, > > I have one stream is consumed by FlinkKafkaConsumer which will be joined > with

Kerberos error when restoring from HDFS backend after 24 hours

Hello and happy new year to all flink users, I have a streaming application (flink v1.7.0) on a Kerberized cluster, using a flink configuration file where the following parameters are set : security.kerberos.login.use-ticket-cache: false security.kerberos.login.keytab: X

Re: ConnectTimeoutException when createPartitionRequestClient

Hi Wenrui, from the logs I cannot spot anything suspicious. Which configuration parameters have you changed exactly? Does the JobManager log contain anything suspicious? The current Flink version changed quite a bit wrt 1.4. Thus, it might be worth a try to run the job with the latest Flink

Re: Unable to restore the checkpoint on restarting the application!!

Hi Puneet, what exactly is the problem when you try to resume from a checkpoint? Cheers, Till On Fri, Jan 4, 2019 at 2:31 PM Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Hi All > > I am creating a poc where i am trying the out of box feature of flink > for managed state of

Re: same parallelism with different taskmanager and slots, skew occurs

Hi, could you tell me how exactly you started the cluster and with which parameters (configured memory, maybe vcores, etc.)? Cheers, Till On Thu, Jan 3, 2019 at 2:37 AM varuy322 wrote: > Hi, Till > It's very kind of your reply. I got your point, I'm sorry to not make it > clear about my

Re: Unable to restore the checkpoint on restarting the application!!

The List it returns is blank On Fri, Jan 4, 2019 at 7:15 PM Till Rohrmann wrote: > Hi Puneet, > > what exactly is the problem when you try to resume from a checkpoint? > > Cheers, > Till > > On Fri, Jan 4, 2019 at 2:31 PM Puneet Kinra < > puneet.ki...@customercentria.com> wrote: > >> Hi All >>

Re: The way to write a UDF with generic type

Hi Yinhua, Flink needs to know how to serialize and deserialize a type `T`. If you are using a type variable here, Flink can not derive the type information. You need to override org.apache.flink.table.functions.AggregateFunction#getResultType and return type information that matches.

Unable to restore the checkpoint on restarting the application!!

Hi All I am creating a poc where i am trying the out of box feature of flink for managed state of operator . I am able to create the checkpoint while running my app in eclipse but when i am trying to restart the app . I am unable to restore the state. Please find attached below snippet. step

Re: using updating shared data

Another solution to the watermarking issue is to write an AssignerWithPeriodicWatermarks for the control stream that always returns Watermark.MAX_WATERMARK as the current watermark. This produces watermarks for the control stream that will effectively be ignored. On Thu, Jan 3, 2019 at 9:18 PM

Re: The way to write a UDF with generic type

Hi Chesnay, Maybe you misunderstand my question. I have below code: public class MyMaxAggregation extends AggregateFunction { @Override public MyAccumulator createAccumulator() { return new MyAccumulator(); } @Override public T getValue(MyAccumulator accumulator) { return null;

Re: Is there an example of flink cluster "as a job" deployment on k8s ?

Awesome. If https://issues.apache.org/jira/browse/FLINK-9953. is on we would rather wait. In the interim we might go for the static pool route. https://github.com/kubernetes/community/blob/master/sig-big-data/resources.md#flink is the big data link I am following. On Thu, Jan 3, 2019 at 4:46 PM

Re: The way to write a UDF with generic type

I believe you have to extend "org.apache.flink.table.functions.AggregateFunction" instead for it to work with SQL (or more generally speaking, any sub-class of "org.apache.flink.table.functions.UserDefinedFunction". On 04.01.2019 05:18, yinhua.dai wrote: Hi Community, I tried to write a UDF

Re: Using port ranges to connect with the Flink Client

Hi, Thanks Chesnay my problem was fixed it was related to enabling port ranges for the rest client it turned out. Gyula On Fri, 4 Jan 2019 at 10:26, Chesnay Schepler wrote: > @Gyula: From what I can tell your custom client is still relying on > akka, and should be using the RestClusterClient

Re: Unable to restore the checkpoint on restarting the application!!

When starting a job from within the IDE using the LocalEnvironment, it is not possible to specify a checkpoint from which to resume. That's why your recovered state is empty. Flink won't automatically pick up the latest checkpoint it finds in some checkpoint directory. You can test it though by

Re: S3 StreamingFileSink never completes multipart uploads

Hi Nick, This is a known issue with 1.7.0, I have an issue opened up here: https://issues.apache.org/jira/browse/FLINK-11187 On Wed, Jan 2, 2019 at 5:00 PM Martin, Nick wrote: > I’m running on Flink 1.7.0 trying to use the StreamingFileSink with an S3A > URI. What I’m seeing is that whenever