Re: Support for detached mode for Flink1.5 SQL Client

The INSERT INTO [1] statement will allow to submit queries detached. So your can close the client and let the Flink program do it's job sinking into external systems. Regards, Timo [1] https://issues.apache.org/jira/browse/FLINK-8858 Am 12.07.18 um 02:47 schrieb Rong Rong: Is the Gateway

Re: Need to better way to create JSON If we have TableSchema and Row

Hi Shivam, Flink 1.5 provides full Row-JSON-Row conversions. You can take a look at the `flink-json` module. A table schema can be converted into a TypeInformation (Types.ROW(schema.getColumns(), schema.getTypes())) which can be used to configure JsonRowSerialization/DeserializationSchemas.

Re: Want to write Kafka Sink to SQL Client by Flink-1.5

Hi Shivam, a Kafka sink for the SQL Client will be part of Flink 1.6. For this we need to do provide basic interfaces that sinks can extends as Rong mentioned (FLINK-8866). In order to support all formats that also sources support we also working on separating the connector from the formats

Questions about high-availability directory

Hi all Recently, we use flink with high-availability. We found that there are three kinds of directories in ha.baseDir. They are applicationID/blob, submittedJobGraph and completedcheckpoint. It's used to restore users' jars, submitted job graphs and completed checkpoint. When old Jobmanager is

Re: Some question about document

Hi Yuta, It seems Chesnay is right. The "fallback" in flink's documentation is in terms of the types flink supported. But for all the other arbitrary types kryo is the first choice. 2018-07-12 9:55 GMT+08:00 Yuta Morisawa : > Thank you for your answer. > > > For POJOs Flink has a custom

Re: How to create User Notifications/Reminder ?

Hi Hequen, I was more interested in solving using CEP. I want to have a window of 2 weeks and in the Timeout Handler I want to create Notification/Reminder. Is this doable in Flink 1.4.2.? Thanks On Wed, Jul 11, 2018 at 6:14 PM, Hequn Cheng wrote: > Hi shyla, > > There is a same question[1]

Re: Some question about document

Thank you for your answer. > For POJOs Flink has a custom serializer. For arbitrary objects we use > kryo, and can use Avro as a fallback. https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types It may be the reverse. Kryo is for

Re: Need to better way to create JSON If we have TableSchema and Row

Hi shivam, It seems there is no such a function but you can write one by yourself, maybe use the com.fasterxml.jackson.databind.ObjectMapper. Best, Hequn On Thu, Jul 12, 2018 at 1:56 AM, Shivam Sharma <28shivamsha...@gmail.com> wrote: > Hi All, > > I have TableSchema >

Re: How to create User Notifications/Reminder ?

Hi shyla, There is a same question[1] asked two days ago. Maybe it is helpful for you. Let me know if you have any other concern. Best, Hequn [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-trigger-a-function-on-the-state-periodically-td21311.html On Thu, Jul 12,

Re: Support for detached mode for Flink1.5 SQL Client

Is the Gateway Mode [1] in the FLIP-24 SQL Client road map what you are looking for? -- Rong [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client On Tue, Jul 10, 2018 at 3:37 AM Shivam Sharma <28shivamsha...@gmail.com> wrote: > Hi All, > > Is there any way to run

Re: Want to write Kafka Sink to SQL Client by Flink-1.5

Hi Shivam, Thank you for interested in contributing to Kafka Sink for SQL client. Could you share your plan for implementation. I have some questions as there might have been some overlap with current implementations. On a higher level, 1. Are you using some type of metadata store to host topic

How to create User Notifications/Reminder ?

I need to create User Notification/Reminder when I don’t see a specific event (high volume) from that user for more than 2 weeks. Should I be using windowing or CEP or ProcessFunction? I am pretty new to Flink. Can anyone please advise me what is the best way to solve this? Thank you for your

Re: Filter columns of a csv file with Flink

Ok Hequn, I'll open 2 Jira for this issue, and maybe propose a draft of CsvTableSource class handling avro schemas FLINK-9813 and FLINK-9814 Thank you for your answers and best regards François 2018-07-11 8:11 GMT+02:00 Hequn Cheng : > Hi francois, > > > Is there any plan to give avro schemas

Need to better way to create JSON If we have TableSchema and Row

Hi All, I have TableSchema object and a Flink Row object(or list). Do we have any straightforward way to convert Row object into JSON by using Schema? For Example:- TableSchema-

Re: Flink job hangs using rocksDb as backend

If this is about too many timers and your application allows it, you may also try to reduce the timer resolution and thus frequency by coalescing them [1]. Nico [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/process_function.html#timer-coalescing On

Re: Flink job hangs using rocksDb as backend

Hi shishal! I think there is an issue with cancellation when many timers fire at the same time. These timers have to finish before shutdown happens, this seems to take a while in your case. Did the TM process actually kill itself in the end (and got restarted)? On Wed, Jul 11, 2018 at 9:29

Re: flink JPS result changes

Generally speaking no, the DIspatcher (here called StandaloneSessionClusterEntrypoint) will spawn a jobmanager internally when a job is submitted On 11.07.2018 16:42, Will Du wrote: In this case, do i need to add a jobManager On Jul 11, 2018, at 10:14 AM, miki haiat

Re: flink JPS result changes

In this case, do i need to add a jobManager > On Jul 11, 2018, at 10:14 AM, miki haiat wrote: > > Flink 6 changed the execution model compactly > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 >

Re: flink JPS result changes

Flink 6 changed the execution model compactly https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 https://docs.google.com/document/d/1zwBP3LKnJ0LI1R4-9hLXBVjOXAkQS5jKrGYYoPz-xRk/edit#heading=h.giuwq6q8d23j On Wed, Jul 11, 2018 at 5:09 PM Will Du wrote: > Hi folks > Do

flink JPS result changes

Hi folks Do we have any information about the process changes after v1.5.0? I used to see jobManager and TaskManager process once the start-cluster.sh is being called. But, it shows below in v1.5.0 once started. Everything works, but no idea where is the jobManager. $jps 2523 TaskManagerRunner

Re: Custom metrics reporter classloading problem

Thanks for the explanation, that makes sense. For some reason I thought that in Yarn all stuff goes into the classpath. Gy Chesnay Schepler ezt írta (időpont: 2018. júl. 11., Sze, 15:16): > Reporters do not have access to libraries provided with user-jars. > They are instantiated when JM/TM

Re: Custom metrics reporter classloading problem

Reporters do not have access to libraries provided with user-jars. They are instantiated when JM/TM starts, i.e. before any user-code is even accessible. My recommendation would be to either put the kafka dependencies in the /lib folder or try to relocate the kafka code in the reporter. On

Re: Confusions About JDBCOutputFormat

Cool. I will take a look. Thanks. On Wed, Jul 11, 2018 at 7:08 PM, wangsan wrote: > Well, I see. If the connection is established when writing data into DB, > we need to cache received rows since last write. > > IMO, maybe we do not need to open connections repeatedly or introduce > connection

Custom metrics reporter classloading problem

Hi all, I have ran into the following problem and I want to double check wether this is intended behaviour. I have a custom metrics reporter that pushes things to Kafka (so it creates a KafkaProducer in the open method etc.etc.) for my streaming job. Naturally as my Flink job consumes from

Re: Need assistance : creating remote environment

Based on the logs your client is using the RestClusterClient, which means that the client is either a) running 1.4 with the flip6 profile enabled b) running 1.5. Please ensure that both the flink versions match for client and server, and that both run/do not run in flip6 mode. On 11.07.2018

Re: Need assistance : creating remote environment

Attached log file. (Log level : Trace) Is this the issue ? Trying with very minimal graph (execution plan is printed in log file) WARN akka.remote.transport.netty.NettyTransport- Remote connection to [/127.0.0.1:44322] failed with

Re: Need assistance : creating remote environment

Did/could you enable logging in the submitting code? On 11.07.2018 13:57, Mohan mohan wrote: Hi, I have started flink in cluster mode. ..flink1.4.2/bin/$ ./start-cluster.sh (no config changes ie., default settings) And trying to connect to it,

Need assistance : creating remote environment

Hi, I have started flink in cluster mode. ..flink1.4.2/bin/$ ./start-cluster.sh (no config changes ie., default settings) And trying to connect to it, ExecutionEnvironment.createRemoteEnvironment("localhost", 6123, "xxx.jar"); I am not seeing any response, did not find anything in jobmanager

Re: Confusions About JDBCOutputFormat

Well, I see. If the connection is established when writing data into DB, we need to cache received rows since last write. IMO, maybe we do not need to open connections repeatedly or introduce connection pools. Test and refresh the connection periodically can simply solve this problem. I’ve

Re: Some question about document

1) TypeInformation are used to create serializers, comparators and to verify correctness of certain operations (like projections on tuple datasets). 2) see https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html 3) Flink comes with a number of serializers for varying

Re: Cancelling job with savepoint fails sometimes

My guess is that this is related to https://issues.apache.org/jira/browse/FLINK-2491. The relevant bit is "Failed to trigger savepoint. Decline reason: Not all required tasks are currently running." So, if one task has already finished (for example a source with a small finite input) then

Re: 答复: How to find the relation between a running job and the original jar?

Hi Chesnay I have create an issue https://issues.apache.org/jira/browse/FLINK-9800 Please let me know if I can do anything to help implementing it. Lasse 2018-07-11 9:08 GMT+02:00 Chesnay Schepler : > For a running job there is no way to determine which jar was originally > used for it. > > I

Examples about complex analysis

Hi I would want to find some Flink examples about complex analysis (like CEP) and its log files. I have already found logs for TaxiRides and how to find the long rides using by Flink and CEP [1], but it is little simple analysis case (only two sequential events). Do you know any more complex

Cancelling job with savepoint fails sometimes

I notice that sometimes when I try to cancel a Flink job with savepoint, the cancel fails with the following error: org.apache.flink.util.FlinkException: Could not cancel job 3be3d380dca9bb6a5cf0d559d54d7ff8. at org.apache.flink.client.cli.CliFrontend.lambda$cancel$4(CliFrontend.java:581)

Re: Checkpointing in Flink 1.5.0

As a workaround, we commented out state.backend.rocksdb.localdir since it defaults to the taskmanager.tmp.dirs location. Now, we are having only these state backend configs in our flink-conf.yaml: state.backend: rocksdb state.checkpoints.dir: file:///home/demo/checkpoints/ext_checkpoints

Flink job hangs using rocksDb as backend

Hi, I am using flink 1.4.2 with rocksdb as backend. I am using process function with timer on EventTime. For checkpointing I am using hdfs. I am trying load testing so Iam reading kafka from beginning (aprox 7 days data with 50M events). My job gets stuck after aprox 20 min with no error.

Some question about document

Hi all Now, I'm reading Flink document and I have some points to feel difficult to get an idea. I'd appreciate if you tell it me. 1,TypeInformation I understand TypeInformation is used for selecting relevant serializer and comparator. But, the ducument doesn't specify if it has another

Re: 答复: How to find the relation between a running job and the original jar?

For a running job there is no way to determine which jar was originally used for it. I remember a previous request for this feature, but I don't think a JIRA was created for it. Might be a good time to create one now. Could you open one and specify your requirements? On 11.07.2018 06:33,

Re: Confusions About JDBCOutputFormat

Hi wangsan, What I mean is establishing a connection each time write data into JDBC, i.e. establish a connection in flush() function. I think this will make sure the connection is ok. What do you think? On Wed, Jul 11, 2018 at 12:12 AM, wangsan wrote: > Hi Hequn, > > Establishing a connection

Re: Filter columns of a csv file with Flink

Hi francois, > Is there any plan to give avro schemas a better role in Flink in further versions? Haven't heard about avro for csv. You can open a jira for it. Maybe also contribute to flink :-) On Tue, Jul 10, 2018 at 11:32 PM, françois lacombe < francois.laco...@dcbrain.com> wrote: > Hi