Re: on YARN question

Do you mean to run it in detach mode? If so, you could add "-d". Best, Yangze Guo On Fri, Apr 10, 2020 at 1:05 PM Ethan Li wrote: > > I am not a Flink expert. Just out of curiosity, > > I am seeing > > “YARN application has been deployed successfully“ > > Does it not mean it’s working properly?

Re: on YARN question

I am not a Flink expert. Just out of curiosity, I am seeing “YARN application has been deployed successfully“ Does it not mean it’s working properly? Best, Ethan > On Apr 9, 2020, at 23:01, 罗杰 wrote: > >  > Hello, could you please tell me how to solve the problem that when I use > yarn

on YARN question

Hello, could you please tell me how to solve the problem that when I use yarn-session.sh, the card will not run when it reaches the following place? Hadoop2.7.2  flink 1.10.0   have?? flink/lib/ flink-shaded-hadoop-2-uber-2.7.5-10.0.jar [root@hadoop131 bin]# ./yarn-session.sh -n 2 -s 2 -jm 1024 -t

Re: Re: fink sql client not able to read parquet format table

Hi lei, I think the reason is that our `HiveMapredSplitReader` not supports name mapping reading for parquet format. Can you create a JIRA for tracking this? Best, Jingsong Lee On Fri, Apr 10, 2020 at 9:42 AM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > I am using Hive 3.1.1

[Stateful Functions] Using statefun for E2E testing

Hi there, Today we were designing a test for a workflow that involved 3 different systems talking to each other async. My colleague came with the idea that we could use Flink for E2E, which we got excited about. We came with a quick implementation, within our existing Flink application, after som

Re: Re: fink sql client not able to read parquet format table

I am using Hive 3.1.1 The table has many fields, each field is corresponded to a feild in the RobotUploadData0101 class. CREATE TABLE `robotparquet`(`robotid` int, `framecount` int, `robottime` bigint, `robotpathmode` int, `movingmode` int, `submovingmode` int, `xlocation` int,

Re: End to End Latency Tracking in flink

An Operator like below will expose lag between current time and event time passing the operator. I add that after the source and before the sink, and calculate sink_delay - source_delay in grafana. would that be a generic solution to solve the problem? ``` public class EmitLagOperator extends Abstr

Re: [PROPOSAL] Contribute training materials to Apache Flink

Hi David, +1 to add to the project. I agree that flink.apache.org and flink playgrounds respectively are the best places to host this content. On Thu, Apr 9, 2020 at 2:56 PM Niels Basjes wrote: > Hi, > > Sounds like a very nice thing to have as part of the project ecosystem. > > Niels > > On T

Re: [PROPOSAL] Contribute training materials to Apache Flink

Hi, Sounds like a very nice thing to have as part of the project ecosystem. Niels On Thu, Apr 9, 2020 at 8:10 PM David Anderson wrote: > Dear Flink Community! > > For some time now Ververica has been hosting some freely available Apache > Flink training materials at https://training.ververica.

Re: [PROPOSAL] Contribute training materials to Apache Flink

Hello David, This is a nice move. Im pretty sure the more the resources at one place the better it is for reference, especially for starters. Thanks, Hemant On Thu, Apr 9, 2020 at 11:40 PM David Anderson wrote: > Dear Flink Community! > > For some time now Ververica has been hosting some freel

[PROPOSAL] Contribute training materials to Apache Flink

Dear Flink Community! For some time now Ververica has been hosting some freely available Apache Flink training materials at https://training.ververica.com. This includes tutorial content covering the core concepts of the DataStream API, and hands-on exercises that accompany those explanations. We

Re: Passing checkpoint lock object to StreamSourceContexts.getSourceContext after StreamTask.getCheckpointLock deprecation

One thing that had just occurred to me is that the context is later used to emit watermarks and send elements downstream using ctx.collect. Perhaps all these operations should now be switched to the new mailbox executor instead? On Thu, Apr 9, 2020 at 8:52 PM Yuval Itzchakov wrote: > Hi, > > I h

Passing checkpoint lock object to StreamSourceContexts.getSourceContext after StreamTask.getCheckpointLock deprecation

Hi, I have an implementation of a custom source, which uses StreamSourceContexts.getSourceContext ( https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/StreamSourceContexts.java#L46) which has a checkpointLock argument that's used

Re: FlinkRuntimeException: Unexpected list element deserialization failure

Hi I think you have missed the "caused by" exception [1] , which could tell us the truth. Could you please check the full stack trace? [1] https://github.com/apache/flink/blob/6846522ed67343c665f7e1dd02b7c06c05c1eb1d/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink

FlinkRuntimeException: Unexpected list element deserialization failure

Hi flink team, I see below exception . What could be the reason of the failure ? Please share your thoughts? Caused by: org.apache.flink.util.FlinkRuntimeException: Unexpected list element deserialization failure at org.apache.flink.contrib.streaming.state.RocksDBListState.deserializeNe

Re: Flink job didn't restart when a task failed

For future reference, here is the issue to track the reconciliation logic [1]. [1] https://issues.apache.org/jira/browse/FLINK-17075 Cheers, Till On Thu, Apr 9, 2020 at 6:47 PM Till Rohrmann wrote: > Hi Bruce, > > what you are describing sounds indeed quite bad. Quite hard to say whether > we

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Sounds like a plan Arvid! Taking note of it, this is gold! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink job didn't restart when a task failed

Hi Bruce, what you are describing sounds indeed quite bad. Quite hard to say whether we fixed such an issue in 1.10. It is definitely worth a try to upgrade, though. In order to further debug the problem, it would be really great if you could provide us with the log files of the JobMaster and the

Re: Inserting nullable data into NOT NULL columns

Hi Gyula, some disclaimer: the type system rework is still ongoing and there a a couple of known issues and missing end-to-end tests around this topic. I would therefore recommend to declare the sink as `STRING NULL` for now. Can you open an issue for your concrete use case with some example

Re: Possible memory leak in JobManager (Flink 1.10.0)?

For further reference, I've created this issue [1] to track the problem. [1] https://issues.apache.org/jira/browse/FLINK-17073 Cheers, Till On Thu, Apr 9, 2020 at 1:20 PM Yun Tang wrote: > Hi Marc > > The left 'chk-X' folders, which should be discarded when removing > checkpoint at the final s

Inserting nullable data into NOT NULL columns

Hi All! We ran into a problem while trying to insert data read from kafka into a table sink where some of the columns are not nullable. The problem is that from Kafka we can only read nullable columns in JSON format otherwise you get the following error: org.apache.flink.table.api.ValidationExce

Re: Re: fink sql client not able to read parquet format table

Hi lei, Which hive version did you use? Can you share the complete hive DDL? Best, Jingsong Lee On Thu, Apr 9, 2020 at 7:15 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > I am using the newest 1.10 blink planner. > > Perhaps it is because of the method i used to write the pa

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

I would say so, yes. Also could you set the paths where you want to use Presto to "s3p", as described in [1], just to be sure that there is not ambiguity. You could also make use of [2]. And thanks for looking into it! Cheers, Kostas [1] https://ci.apache.org/projects/flink/flink-docs-stable/d

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

Great work! Thanks Gordon for the continuous efforts for enhancing stateful functions and the efficient release! Wish stateful functions becoming more and more popular in users. Best, Zhijiang -- From:Yun Tang Send Time:2020 Apr.

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

Btw, I ran the same exact code on a local Flink cluster run with `./bin/start-cluster.sh` on my local machine. With `s3a` it did not work, the part files do not roll over; however, with the local filesystem it works perfectly. Should I be looking at the S3Committer in Flink to see if there is somet

Re: Making job fail on Checkpoint Expired?

Hello again Congxian, Thank you so much for your advice, it is really helpful! We have managed to pinpoint that most of our problems occur because of disk pressure, most likely due to the usage of EBS, we will try again with local SSDs. Digging deeper into the "snowball effect on incremental check

Re: Upgrade of Cassandra driver in connector

Just for info the Cassandra client version 4 driver does not leak guava anymore, but represents a considerable API change so maybe worth to explore that as an alternative to 'modernize' the connector. On Thu, Apr 9, 2020 at 1:54 PM Aljoscha Krettek wrote: > > Hi Thomas! > > On 09.04.20 11:35, Thm

Re: Flink job didn't restart when a task failed

Hi, this indeed seems very strange! @Gary Could you maybe have a look at this since you work/worked quite a bit on the scheduler? Best, Aljoscha On 09.04.20 05:46, Hanson, Bruce wrote: Hello Flink folks: We had a problem with a Flink job the other day that I haven’t seen before. One task

Re: Upgrade of Cassandra driver in connector

Hi Thomas! On 09.04.20 11:35, Thms Hmm wrote: Hey, are there any plans to upgrade the version of the Cassandra driver to a newer one? There is this Jira issue, along with a PR: https://issues.apache.org/jira/browse/FLINK-8424. But no-one has worked on it for a while. There were some difficul

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

Nope just the s3a. I'll keep looking around to see if there is anything else I can see. If you think of anything else to try, let me know. On Thu, Apr 9, 2020, 7:41 AM Kostas Kloudas wrote: > It should not be a problem because from what you posted, you are using > "s3a" as the scheme for s3. > A

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

It should not be a problem because from what you posted, you are using "s3a" as the scheme for s3. Are you using "s3p" for Presto? This should also be done in order for Flink to understand where to use the one or the other. On Thu, Apr 9, 2020 at 1:30 PM Roshan Punnoose wrote: > > Lastly, could i

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

Lastly, could it be the way I built the flink image for kube? I added both the presto and Hadoop plugins On Thu, Apr 9, 2020, 7:29 AM Roshan Punnoose wrote: > Sorry realized this came off the user list by mistake. Adding the thread > back in. > > On Thu, Apr 9, 2020, 7:26 AM Roshan Punnoose wro

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

Sorry realized this came off the user list by mistake. Adding the thread back in. On Thu, Apr 9, 2020, 7:26 AM Roshan Punnoose wrote: > Yes sorry, no errors on the task manager. However, I am new to flink so > don't know all the places to look for the logs. Been looking at the task > manager log

Re: Possible memory leak in JobManager (Flink 1.10.0)?

Hi Marc The left 'chk-X' folders, which should be discarded when removing checkpoint at the final stage, could also prove that those not discarded completed checkpoint meta occupied the memory. If we treat your average checkpoint meta size as 30KB, 2 not-discarded complete checkpoints woul

Re: Re: fink sql client not able to read parquet format table

I am using the newest 1.10 blink planner. Perhaps it is because of the method i used to write the parquet file. Receive kafka message, transform each message to a Java class Object, write the Object to HDFS using StreamingFileSink, add the HDFS path as a partition of the hive table No matter

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

I was wondering if you could actually really use AsyncWaitOperator in the following way. - Use a rather big timeout (so if callbacks usually take 1s, use 10). - Use UNORDERED mode. - Use a rather big queue size that would not cause any backpressure (you could just experiment with different setting

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

Hi Roshan, Your logs refer to a simple run without any failures or re-running from a savepoint, right? I am asking because I am trying to reproduce it by running a modified ParquetStreamingFileSinkITCase [1] and so far I cannot. The ITCase runs against the local filesystem, and not S3, but I adde

Re: Possible memory leak in JobManager (Flink 1.10.0)?

Thanks for reporting this issue Marc. From what you've reported, I think Yun is right and that the large memory footprint is caused by CompletedCheckpoints which cannot be removed fast enough. One way to verify this is to enable TRACE logging because then Flink will log for every CompletedCheckpoin

Upgrade of Cassandra driver in connector

Hey, are there any plans to upgrade the version of the Cassandra driver to a newer one? Currently the driver v3.0 is integrated which dates back to 2016 and we are having problems getting this working with AD authentication and Cassandra V4.x. Also the shading makes it not possible to easily exch

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Perfectly understood, thanks a lot for your reply/patience . I will take a look at AsyncWaitOperator and adapt from there if I really need that. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi, With: > Can not you take into account the pending element that’s stuck somewhere in > the transit? Snapshot it as well and during recovery reprocess it? This is > exactly that’s AsyncWaitOperator is doing. I didn’t mean for you to use AsynWaitOperator, but what both me and Arvid suggested

Re: Possible memory leak in JobManager (Flink 1.10.0)?

Hello Yun, Thank you for your feedback, please find below my answers to your questions: 1. I am using incremental state checkpointing with RocksDB backend and AWS S3 as a distributed file system, everything is configured in flink-conf.yaml as follows: state.backend: rocksdb state.backend.increm

Re: Complex graph-based sessionization (potential use for stateful functions)

Hi All, One way to try to think about it with StateFun, is to represent the Graph vertices as stateful functions instances. Unlike other frameworks an instance of a function does not take any resources while idle, and potentially you can have many millions of those. A state for each vertex might be

Re: ListState with millions of elements

On 08.04.20 20:14, Seth Wiesman wrote: There is a limitation in RocksDB's JNI bridge that will cause applications to fail if list state exceeds 2GB. I am not aware of anyone working on this issue. That is correct, and here's the Jira issue for it: https://issues.apache.org/jira/browse/FLINK-67

Re: Upgrading Flink

Hey Stephen, 1. You should be able to migrate from 1.8 to 1.10: https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/upgrading.html#compatibility-table 2. Yes, you need to recompile (but ideally you don't need to change anything). On Mon, Apr 6, 2020 at 10:19 AM Stephen Connolly <

Re: Checkpoint Error Because "Could not find any valid local directory for s3ablock-0001"

Hey, Others have experienced this as well, yes: https://lists.apache.org/thread.html/5cfb48b36e2aa2b91b2102398ddf561877c28fdbabfdb59313965f0a%40%3Cuser.flink.apache.org%3EDiskErrorException I have also notified the Hadoop project about this issue: https://issues.apache.org/jira/browse/HADOOP-15915

Re: Fwd: Complex graph-based sessionization (potential use for stateful functions)

Hey Max, 1) Stateful functions has been released now: https://mvnrepository.com/artifact/org.apache.flink/statefun-flink-core See also: https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html Getting Started: https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/getting