flink connectors

distribution that contains these connectors? Regards, Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone:

RE: flink connectors

Hi, Is there any alternative to avoiding maven? That is why I was curious if there is a binary distribution of this available for download directly Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Rese

RE: flink connectors

Hi, Thank you for the tips! For future references in case someone else wants to search for the binaries for this, I would like to share the link to the maven repository http://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka Dr. Radu Tudoran Research Engineer IT R&a

key

(GenericType) cannot be used as key. Can you give me a hint about a solution to this? Thanks Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.t

RE: key

expected to go wrong somewhere else without having it? @Marton – the answer was that I need 2 of these fields to be used to do the keyBy (long, string) Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Rese

question about DataStream serialization

et and launch a batch processing while operating in stream mode- in case this is possible, can you give me a reference for it, because it might be the good solution to use in case I could treat the fixed set as a DataSet and just add/remove the incoming event to it and apply an operation Regard

Question about DataStream serialization

thinking that I could keep the fixed set as a DataSet and as each new event comes, transform it into a dataset and then join with reference set and apply an operation Regards, Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOG

RE: Question about DataStream serialization

an empty dataset in the flatmap function while when this finish executing the dataset gets loaded. Is this an error or I am doing something wrong? Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research

RE: Question about DataStream serialization

use ...is empty } @Override public void open( { dataset -> load } }) Dr. Radu Tudoran Research Engineer IT R&D Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +4

RE: Question about DataStream serialization

//System.out.println("dataset value "+dataset.value()); } }); stream.print(); env.execute("test open

RE: Question about DataStream serialization

Hi, The state that is being loaded can very well be partitioned by keys. Assuming this scenario and that you would now that the keys go from 0 to N, is there some possibility to load and partitioned the initial data in the open function? Dr. Radu Tudoran Research Engineer IT R&D Divi

RE: Question about DataStream serialization

Hi, Is the partitioned functioned used by the ".keyBy(Object)" of the form: Object.hash % getNumberOfParallelSubtasks() ? Dr. Radu Tudoran Research Engineer IT R&D Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-m

flink streaming documentation

Hi, I believe i found 2 small inconsistencies in the documentation for the description of Window Apply https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#windows-on-unkeyed-data-streams 1) in the example provided I believe it should be corrected to remove

RE: Behaviour of CountWindowAll

Hi, I believe this question might have been asked before - so sorry for repeating it (I just did not find the discussion on the mailing list). Is it possible somehow to create a new DataStream from the elements that are evicted from a window? A simple use case for this is: We have data

global function over partitions

Hi, I am trying to compute some final statistics over a stream topology. For this I would like to gather all data from all windows and parallel partitions into a single/global window. Could you suggest a solution for this. I saw that the map function has a ".global()" but I end up with the same

RE: global function over partitions

rentTimeMillis()); arg2.collect(new Tuple1(count)); } }).setParallelism(1) .writeAsText("/tmp/testoutput", WriteMode.OVERWRITE); env.execute(&quo

maxtime / watermark for GlobaWindow

rg0, Iterable< Tuple1, > arg1, Collector> arg2) throws Exception { // - get the even timestamp } }) Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [

continous time triger

that timeWindow will end up generating multiple windows and duplicating data, while having the option from the trigger to actually fire the processing at certain times, independent of when the events arrived) would enable to operate with a single window. Regards, Dr. Radu Tudoran Research

RE: continous time triger

Hi, Thank you for sharing your experience and also to Till for the advice. What I would like to do is to be able to fire the window potentially multiple times, even if an event did not arrive. I will look more about how dealing with the processing time could help in this Dr. Radu Tudoran

release of task slot

) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 Münche

RE: Flink writeAsCsv

Hi Radu, It is indeed interesting to know how each window could be registered separately - I am not sure it any of the existing mechanisms in Flink support this. I think you need to create your own output sink. It is a bit tricky to pass the window sequence number (actually I do not think such

RE: release of task slot

like the input flow is faster than the GC collector Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile:

job manager timeout

Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49 891588344173 HUAWEI TEC

events eviction

e some way of creating a state in the evictor that will be check pointed and restore in case of failure. I would be interested if something like an operator state is possible in the evictor. Regards, Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01C

RE: events eviction

. And then to ensure resiliency, the state should be recoverable if a crash happens. Could you approach this without an evictor state? Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 Mün

RE: events eviction

Feb 2016, at 18:07, Radu Tudoran wrote: > > Hi, > > Thanks Aljoscha for the details! > > The warning about performance and evictors is useful, but I am not sure how > it can be put in practice always. Take for example a GlobalWindow that you > would use to aggregate

realtion between operator and task

Hi, Can someone explain how and where a stream operator is mapped to a stream task. I am particularly interested in the way the stream outputs are created and attached to the operators. I saw that this happen in OperatorChain functions but I do not have the picture of the lifecycle of an stream

RE: realtion between operator and task

.." Stephan On Mon, Mar 7, 2016 at 7:36 PM, Radu Tudoran mailto:radu.tudo...@huawei.com>> wrote: Hi, Can someone explain how and where a stream operator is mapped to a stream task. I am particularly interested in the way the stream outputs are created and attached to the op

RE: [ANNOUNCE] Flink 1.0.0 has been released

Hi, Do you have also a linkedin post that I could share - or should I make a blogpost in which I take this announcement? Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-

ype of TypeVariable could not be determined

xecutionEnvironment.java:1107) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1089) ... 1 more Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Ries

RE: ype of TypeVariable could not be determined

I can pass it using the “.returns()” method as Timo suggested Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei

operators

.=> the red items should go on the same machine Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile:

RE: operators

. The best thing usually is location transparency and local affinity (as a performance optimization). Is the file large, i.e., would it hurt to do it on a DFS? Or actually use a Kafka Queue between the operators? Stephan On Wed, Mar 9, 2016 at 5:38 PM, Radu Tudoran mailto:radu.tudo...@huawei.c

jar dependency in the cluster

submit the job to add extra dependencies. Many thanks Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile:

RE: ClasNotFound when submitting job from command line

Hi, In my case the root cause for this was mainly that I was using eclipse to package the jar. Try using mvn instead. Additioanlly you can copy the dependency jars in the lib of the task managers and restart them Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Divi

RE: Custom time window in Flink

would be that you have a field that marks whether a window was created or not (e.g., a Boolean field in the evictor class), which once a window is created you can set it to false and from there one you operate on 5 minutes windows Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&a

RE: Custom time window in Flink

public int evict class. Same story with the triggers https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HU

lost connection

Hi, I am trying to submit a jar via the console (flink run my.jar). The result is that I get an error saying that the communication with the jobmanager failed: Lost connection to the jobmanager. Can you give me some hints/ recommendations about approaching this issue. Thanks Dr. Radu Tudoran

RE: lost connection

. org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Communication with JobManager failed: Lost connection to the JobManager. I do not understand what could be the root cause of this... the IPs look ok and there is not firewall to block things... Dr. Radu Tudoran Research Engineer - Big

RE: lost connection

Yes - it suddenly occurred on something that used to work. I am restarting the deployment to see if this solves the problem Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Ce

RE: What's the advantage of using BroadcastState?

atterns", Types.VOID, Types.POJO(Pattern.class))); // storing in MapState with null as VOID default value bcState.put(null, pattern); If (test) { bcState.put(null, new Pattern(test) ); } } } Dr. Radu Tudoran Staff Research Engineer - Big Data Expert IT R&

FW: What's the advantage of using BroadcastState?

what you think.. Dr. Radu Tudoran Staff Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH German Research Center Munich Office Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@hua

flink build error

Hi, I am trying to build flink 1.6 but cannot build it to run also the tests. Any ideas of why the surefire error to run junits tests? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project flink-test-utils-junit: ExecutionException:

Flink and Calcite

Hi, Can someone point me to the repository where the integration of Calcite with Flink is available? Does this come with the master branch (as indicated by the link in the blog post)? https://github.com/apache/flink/tree/master Thanks Dr. Radu Tudoran Research Engineer - Big Data Expert IT R

error for building flink-runtime from source

org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Best regards, Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D D

RE: error for building flink-runtime from source

Hi, I am building the 1.1 snapshot (should be the latest release). I will try to build the whole project to check if it works Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Ce

SQL for Flink

(DataStreamScanRule.INSTANCE , CUSTOM_TABLE_SCAN_Rule.INSTANCE) - or if the selection of the new translation rule can be handled from the cost than simply configuration.addRule( CUSTOM_TABLE_SCAN_Rule.INSTANCE) What do you think? Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R&D Divi

SQL / Tuple question

Hi, I am trying to create an sql statement that is suppose to return a string and and integer Mytable.sql("select mystringfield,myintfield ) I am trying to give the typefinormation as a class to the TypeInformation> tpinf = new TypeHint>(){}.getTypeInfo(); .toDataStream( , tpinf.getTypeC

RE: SQL / Tuple question

this? TypeInformation> tpinf = new TypeHint>(){}.getTypeInfo(); .toDataStream( , tpinf ) Best, Fabian 2016-09-19 17:53 GMT+02:00 Radu Tudoran mailto:radu.tudo...@huawei.com>>: Hi, I am trying to create an sql statement that is suppose to return a string and and integer Mytable

emit watermarks

Hi, Is there some way to emit a watermark in the trigger? I see that in the evictor there is the option to check the StreamRecord if it is a watermark..so I would hope that there is some option also to create them

org.apache.flink.core.fs.Path error?

/dir/myfile.csv"". The problem is that this is never changed when the file is actually open and accessed which leads to an error. ...not sure if signaling this error here is the best place or if I should have used some other media.. Best regards, Dr. Radu Tudoran Senior Research En

RE: org.apache.flink.core.fs.Path error?

right now i would suggest the usual: check that the path is indeed correct, that you have sufficient permissions to access the file. And yes, you can report problems here ;) Regards, Chesnay On 20.10.2016 13:17, Radu Tudoran wrote: Hi, I am running a program that is suppose to read a CSV file f

RE: org.apache.flink.core.fs.Path error?

Path, but directly generates a URI from it. On 20.10.2016 15:48, Fabian Hueske wrote: The error message suggests that Flink tries to resolve "D:" as a file system schema such as "file:" or "hdfs:". Can you try to use specify your path as "file:/D:/dir/my

question for generic streams

Hi, I am trying to read the types of an input stream from a file and then generate the datastream resulting from parsing a source accordingly (e.g. DataStream>). Finally this will be registered as a table. What would be the best way to do this? I am trying currently to use the generic type of T

TIMESTAMP TypeInformation

Hi, I would like to create a TIMESTAMP type from the data schema. I would need this to match against the FlinkTypeFactory (toTypeInfo()) def toTypeInfo(relDataType: RelDataType): TypeInformation[_] = relDataType.getSqlTypeName match { case BOOLEAN => BOOLEAN_TYPE_INFO case TINYINT => BY

RE: TIMESTAMP TypeInformation

ill be converted to SqlTimeTypeInfo...but it is converted to GenericType. Anythoughts how I could force the type to be recognize as a SqlTimeType? From: Radu Tudoran Sent: Tuesday, October 25, 2016 4:46 PM To: 'user@flink.apache.org' Subject: TIMESTAMP TypeInformation Hi, I would

RE: TIMESTAMP TypeInformation

mechanism called after the returns to recognize TIMESTAMP of type SqlTimeTypeInfo it will not happen but instead a GenericType will be created. It looks like the type parsers were not extended to consider this types Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R&D Divi

RE: TIMESTAMP TypeInformation

TypeInfos. Can you open a JIRA for that? Thanks, Fabian 2016-10-27 9:31 GMT+02:00 Radu Tudoran mailto:radu.tudo...@huawei.com>>: Hi, I dig meanwhile more through this and I think I found a bug actually. The scenario that I was trying to describe was something like 1. You have a g

RE: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Hi, I think the idea of having such a monthly thread is very good and it might even help to further attract new people in the community. In the same time I do not think that 1 extra mail per month is necessary a spam ☺ In the same time – we can also consider a jobs@flink mailing list Dr. Radu

RE: Dummy DataStream

Hi Duck, I am not 100% sure I understand your exact scenario but I will try to give you some pointers, maybe it will help. Typically when you do the split you have some knowledge about the criterion to do the split. For example if you follow the example from the website https://ci.apache.org/pr

concurrency?

Hi, I would like to use a processFunction to accumulate elements. Therefore in the processElement function I will accumulate this element into a state. However, I would like to emit the output only 1ms later. Therefore I would register a timer to trigger one second later and read the state and

RE: concurrency?

: (processElement – executed second) ev3-arrives state=ev3 onTime(executed first): out = ev2 Best regards, Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH German Research Ce

RE: concurrency?

Hi, Yes it does – thanks a lot Knowing that this is the order time = 2, onTimer(2) -> access state with key t=2-1, get A, B time = 2, pocessElement(C) -> put C in state keyed to t=2, registerProcTimer(3) is useful! Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R&

RE: output writer

ach to the appropriate file Might be that other solutions exist Dr. Radu Tudoran Research Engineer IT R&D Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49 891588344173 HUAWE

RE: output writer

I will double check and try to commit this in the next days Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 1520908

RE: output writer

that could be contributed. We should identify a couple of common formats. The first one that comes in my mind is to have something for writing to memory (e.g. memory buffer) Dr. Radu Tudoran Research Engineer IT R&D Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duessel