Re: Apache Phenix integration

I opened an issue for this: https://issues.apache.org/jira/browse/FLINK-7605 On Wed, Sep 6, 2017 at 4:24 PM, Flavio Pompermaier wrote: > Maybe this should be well documented also...is there any dedicated page to > Flink and JDBC connectors? > > On Wed, Sep 6, 2017 at 4:12

Re: Flink Job Deployment

Hi Rinat, no, this is unfortunately not possible. When a job is submitted, all required JARs are copied into an HDFS location that's job-specific. Best, Fabian 2017-09-04 13:11 GMT+02:00 Rinat : > Hi folks ! > I’ve got a question about running flink job on the top of

Re: Handle event time

Thank you, effectively I developed also a simple custom solution for watermark looking at flink doc but anyway I see unordered printed streams. I have a doubt about flink behaviour: if I understand, flink doesn't perform automatically reordering of records in a stream, so if for instance a record

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

Hi Kant, no, there is no such functionality. I'm also not sure how well streaming would work together with the JDBC interface. JDBC has not been designed for continuous streaming queries, i.e., queries that never terminate. Challenges would be to have an infinite, streamable ResultSet (which

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

As I said, there is no such functionality built into Flink yet. A client program can be parameterized with a query and turned into a SQL client that way. The submission would work with the regular Flink job client, i.e., it would pickup the regular Flink config. Best, Fabian 2017-09-08 10:05

Re: State Maintenance

Only KeyedState can be used as queryable state. So you cannot query the OperatorState. AFAIK, it should not be a problem if an operator has OperatorState and queryable KeyedState. 2017-09-07 17:01 GMT+02:00 Navneeth Krishnan : > Will I be able to use both queryable

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

Hi Fabian, Thanks for the response. I understand the common approach is to write a client program and run it however this will not allow me to send queries Ad-hoc so Is there anyway for me to submit Calcite SQL to Flink via REST or whatever mechanism? Forgot even, the result set once I know there

Re: Quick checkpointing related question

Hi, the method is only called after the checkpoint completed on the job manager. At this point _all_ work for the checkpoint is done, so doing work in this callback does not add any overhead to the checkpoint. Best, Stefan > Am 08.09.2017 um 10:20 schrieb Martin Eden

Re: Flink Job Deployment (Not enough resources)

Hi Rinat, No, Flink does not have a switch to immediately cancel a job if it cannot allocate enough resources. Maybe YARN has a configuration parameter to define a timeout after which a job is canceled if no resource become available. 2017-09-04 13:29 GMT+02:00 Rinat :

Quick checkpointing related question

Hi all, I have a Flink 1.3.1 job with a source that implements CheckpointingFunction. As I understand it, the notifyCheckpointComplete callback is called when all the downstream operators in the DAG successfully finished their checkpoints. Since I am doing some work in this method, I would like

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

Got it! Thanks a lot for your detailed explanation. On Fri, Sep 8, 2017 at 1:27 AM, Fabian Hueske wrote: > As I said, there is no such functionality built into Flink yet. > > A client program can be parameterized with a query and turned into a SQL > client that way. > The

Re: File System State Backend

Hi, I just tried out checkpoint with FsStateBackend in 1.3.2 and everything works as expected for me. Can you give a bit more detail what you mean by „checkpoint data is not cleaning“? For example, is it not cleaned up while the job is running and accumulating „chk-[ID]“ directories or is

Re: Quick checkpointing related question

Thanks for the prompt reply Stefan! On Fri, Sep 8, 2017 at 9:25 AM, Stefan Richter wrote: > Hi, > > the method is only called after the checkpoint completed on the job > manager. At this point _all_ work for the checkpoint is done, so doing work > in this callback

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi Aljoscha, I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception. Caused by: java.io.FileNotFoundException: Properties file

Error a simple window example

Hi all, I am trying to run a simple example in the Scala shell: case class MonEntier(classe: Int, valeur: Int) val stream =3D senv.socketTextStream("localhost", 9000, '\n') val w =3D stream.map ( { x =3D> Tuple1(x.toInt) } ) .map( {y =3D> MonEntier(y._1 % 3, y._1) } )

Re: File System State Backend

Hi! Checkpoints in Flink need to go to a file system that is accessible across machines. Otherwise there could be no recovery of a data of a failed machine. The cleanup is also triggered by a different node than the node that checkpoints - hence you see no cleanup in your setup. Best, Stephan

BucketingSink never closed

Hi to all, I'm trying to test a streaming job but the files written by the BucketingSink are never finalized (remains into the pending state). Is this caused by the fact that the job finishes before the checkpoint? Shouldn't the sink properly close anyway? This is my code: @Test public void

Re: BucketingSink never closed

Hi Flavio, If I understand correctly, I think you bumped into this issue: https://issues.apache.org/jira/browse/FLINK-2646 There is also a similar discussion on the BucketingSink here:

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi, How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()? Best, Aljoscha > On 8. Sep 2017, at 16:32, Tony Wei wrote: > > Hi, > > I put the my configuration file in

Re: Fwd: HA : My job didn't restart even if task manager restarted.

Hi, sorry for the late response! I'm not familiar with the details of the failure recovery but Till (in CC) knows the code in depth. Maybe he can figure out what's going on. Best, Fabian 2017-09-06 5:35 GMT+02:00 sunny yun : > I am still struggling to solve this problem.

Re: Disable job graph in web UI

Upon further inspection, it appears that the web UI redraws each DOM element with every update. So I think removing the graph won’t fix the page performance issue because each task list item is being redrawn on every refresh. > On Sep 7, 2017, at 2:22 PM, Joshua Griffith

Re: Assigning operators to slots

Hi, For the first question, I think both approaches should work. You only have to be careful about startNewChain() because the behaviour can be somewhat unexpected. What it does is specify, that a new chain should be started with the operator on which you call startNewChain(). For example, in:

Table API and registration of DataSet/DataStream

Hi to all, I have a doubt about Table API. Let's say my code is something like: StreamTableEnvironment te = ...; RowTypeInfo rtf = new RowTypeInfo(...); DataStream myDs = te.registerDataStream("test",myDs,columnNames); Table table = te.sql("SELECT *, (NAME = 'John') as VALID FROM test WHERE

How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi, I put the my configuration file in `./src/main/resources/` and packed it inside my jar. I want to run it on standalone cluster by using web UI to submit my job. No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException`

Assigning operators to slots

Hi, firstly excuse me for the long post. I already read the documentation about parallelism, slots and the API about it but I still have some doubts about practical implementations of them. My program is composed essentially by three operations: - get data from a kafka source - perform a machine

Re: Assigning operators to slots

Nice, thank you for reply. So if I call slotSharedGroup(groupname) on the last operator as here: DataStream stream = env .addSource(new FlinkKafkaConsumer010<>(TOPIC, new CustomDeserializer(), properties)) .assignTimestampsAndWatermarks(new CustomTimestampExtractor()) .map(...)

Re: File System State Backend

Thanks for the response. Thats correct, they do not get purged/deleted while the job is running. I have 3 concurrent jobs running and there are 3 directories in the data directory. /var/data/flink/2375c69006bfeca9644171f31b444dff /var/data/flink/c3264bb6d5e068d6440bbb21069b7d28

Re: Assigning operators to slots

UPDATE: I'm trying to implement the version with one node and two task slots on my laptop. I have also in configured flink-conf.yaml the key: taskmanager.numberOfTaskSlots: 2 but when I execute my program in the IDE: /org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:

Installing Apache Flink on Mesos Cluster without DC/OS

Hi, I am newbie in Apache Flink and our team is trying to set up an Apache Flink Cluster on Apaches Mesos. We have already installed Apache Mesos & Marathon with 3 Master nodes and 3 Slaves and now we are trying to install Apache Flink without DC/OS as mentioned here

Re: Question about Flink internals

Hi, Timo. Thank you for detailed replies. It helps me to understand flink a lot. However, there are misinterpreted points. 2. From a user's perspective you can only see the "real data". Internally, there are different types of records that flow through the topology (namely watermarks,

Re: File System State Backend

Can I utilize disk on the job manager for this or do I need a dedicated disk storage vm? How do I specify not only directory but ip address of the checkpoint data directory? Is there any docs to configure a a state backend without using hdfs or s3? thanks for your help Ryan -- Sent from:

State Issue

Hi, I'm experiencing a wired issue where any data put into map state when retrieved with the same key is returning as null and hence it puts the same value again and again. I used rocksdb state backend but tried with Memory state backend too but the issue still exist. Each time when I set the

question on sideoutput from ProcessWindow function

Hi, Can we have a side output from a process window function ? I am currently genrating a side output from a process function. The main output of the process function is then Windowed and a ProcessWindowFunction is applied on the windows. Can I add to the SideOutpuStream from the

Re: question on sideoutput from ProcessWindow function

Hi Prabhu, That is good question, the short answer is not yet, only ProcessFunction was given flexibility of doing customized sideoutput at the moment. Window Function wasn't given such flexibility partially due to sideoutput initially targeting late arriving event for window use cases.