Re: unclear exception when writing to elasticsearch

etting you know I’m trying your setup right now, and will get back > to you once I confirm the results. > > - Gordon > > > On March 1, 2017 at 9:15:16 PM, Martin Neumann (mneum...@sics.se) wrote: > > I created the project using the maven archetype > <https://ci.apac

Re: unclear exception when writing to elasticsearch

nstall -DskipTestscd flink-dist > mvn clean install > > *Note:* To check your Maven version, run mvn --version. > > On Wed, Mar 1, 2017 at 12:19 PM, Martin Neumann <mneum...@sics.se> wrote: > >> I tried to change the elastic search version to 2.4.1

Re: unclear exception when writing to elasticsearch

ding a dependency to the Elasticsearch client of your > desired version in your project. > > You can also check what Elasticsearch client version the project is using > by checking `mvn dependency:tree` from the base directory of your project. > > Cheers, > Gordon > > >

Re: unclear exception when writing to elasticsearch

org/ > projects/flink/flink-docs-release-1.3/dev/linking.html# > packaging-dependencies-with-your-usercode-with-maven. > > Let me know if the problem remains after trying out the above :-) > > Cheers, > Gordon > > On March 1, 2017 at 12:24:08 AM, Martin Neumann (mneum

unclear exception when writing to elasticsearch

Hej, I'm trying to write to elastic search from a streaming application and I get a weird error message I that I can't decipher. Hopefully, someone here can help me. I'm trying to run the java example

Flink streaming connect and split streams

Hej, I'm currently playing around with some machine learning algorithms in Flink streaming. I have an input stream that I partition by key and then do a map on each of the keys, feeding a model and producing a prediction output. Periodically each operator needs to send model updates to all other

Re: Local Cluster have problem with connect to elasticsearch

Hi, Are you sure the elastic cluster is running correctly? Open a browser and try 127.0.0.1:9200 that should give you the overview of the cluster. If you don't get it there is something wrong with the setup. Its also a good way to double check the cluster.name (I got that wrong more than once)

get start and end time stamp from time window

Hej, I have a windowed stream and I want to run a (generic) fold function on it. The result should have the start and the end time stamp of the window as fields (so I can relate it to the original data). *Is there a simple way to get the timestamps from within the fold function?* I could find

Re: TypeVariable problems

returns(). > > Cheers, > Aljoscha > > On Fri, 29 Apr 2016 at 12:25 Martin Neumann <mneum...@sics.se> wrote: > >> Hej, >> >> I have a construct of different generic classes stacked on each other to >> create a library (so the type variables get handed on). A

Fwd: TypeVariable problems

Hej, I have a construct of different generic classes stacked on each other to create a library (so the type variables get handed on). And I have some trouble getting it to work. The current offender is a Class with 3 type variables internally it calls: .fold(new Tuple3<>(keyInit ,new

Re: Help with generics

initModel.class). If > that doesn't work you can try and pass in a function that gives you a > TypeInformation for your model type M. > > Cheers, > Aljoscha > > On Thu, 21 Apr 2016 at 14:16 Martin Neumann <mneum...@sics.se> wrote: > >> Hej, >> >> I pas

Re: streaming hdfs sub folders

Martin On Fri, Feb 19, 2016 at 2:39 PM, Robert Metzger <rmetz...@apache.org> wrote: > Hi Martin, > > where is the null pointer exception thrown? > I think you didn't call the open() method of the AvroInputFormat. Maybe > that's the issue. > > On Thu, Feb 18, 2016 at 5:0

Re: streaming hdfs sub folders

Record(new EndSongCleanedPq()); if (res != null) collector.collect(res); } } } On Thu, Feb 18, 2016 at 4:06 PM, Martin Neumann <mneum...@sics.se> wrote: > I guess I need to set the parallelism for the FlatMap to 1 to make sure I > read one file at a time. The

Re: streaming hdfs sub folders

do you think? > > Stephan > > > On Wed, Feb 17, 2016 at 9:42 PM, Martin Neumann <mneum...@sics.se> wrote: > >> I forgot to mention I'm using an AvroInputFormat to read the file (that >> might be relevant how the flag needs to be applied) >> See the code

Re: streaming hdfs sub folders

t; > files. If you still want parallel reads, that means you need to sync at > some point, wait for all parallel parts to finish with the "00" work before > anyone may start with the "01" work. > > Is your training program a DataStream or a DataSet program?` > >

streaming hdfs sub folders

Hi, I have a streaming machine learning job that usually runs with input from kafka. To tweak the models I need to run on some old data from HDFS. Unfortunately the data on HDFS is spread out over several subfolders. Basically I have a datum with one subfolder for each hour within those are the

Re: streaming using DeserializationSchema

Feb 12, 2016 at 1:41 PM, Martin Neumann <mneum...@sics.se> wrote: > >> I'm trying the same thing now. >> >> I guess you need to read the file as byte arrays somehow to make it work. >> What read function did you use? The mapper is not hard to write but the >&g

Re: streaming using DeserializationSchema

ram that will feed the data into a topic. This also has the > additional benefit that you are actually testing against kafka. > > Cheers, > Gyula > > Martin Neumann <mneum...@sics.se> ezt írta (időpont: 2016. febr. 12., P, > 0:20): > >> Hej, >> >> I have a s

Re: streaming using DeserializationSchema

munity's example library though. > > Good luck! > -n > > On Fri, Feb 12, 2016 at 2:25 AM, Martin Neumann <mneum...@sics.se> wrote: > >> Its not only about testing, I will also need to run things against >> different datasets. I want to reuse as much of the code as possi

Streaming time window

Hej, Is it possible to extract the start and end window time stamps from within a window operator? I have an event time based window that does a simple fold function. I want to put the output into elasticsearch and want to preserve the start and end timestamp of the data so I can directly

Re: Streaming time window

imeWindow window, Iterable vals, > Collector out) { > // YOUR CODE > window.getEnd() > } > }) > > Best, Fabian > > 2015-12-10 14:41 GMT+01:00 Martin Neumann <mneum...@sics.se>: > >> Hi Fabian, >> >> than

flink with kafka 0.7

Hej, I want to connect Flink streaming to a Kafka 0.7 cluster. Will this work with the latest release or does the Kafka implementation rely on kafka 0.8? cheers Martin

Flink Streaming and Google Cloud Pub/Sub?

Hej, Has anyone tried use connect Flink Streaming to Google Cloud Pub/Sub and has a code example for me? If I have to implement my own sources and sinks are there any good tutorials for that? cheers Martin

Re: Event time in Flink streaming

Hej, I want to give TimeTriggerPolicy a try and see how much of a problem it will be in this use case. Is there any example on how to use it? I looked at the API descriptions but I'm confused now. cheers Martin On Fri, Aug 28, 2015 at 5:35 PM, Martin Neumann <mneum...@sics.se>

Re: Performance Issue

Hej, This sounds like it could be a garbage collection problem. Do you instantiate any classes inside any of the operators (e.g. in the KeySelector). You can also try to run it locally and use something like jstat to rule this out. cheers Martin On Mon, Sep 7, 2015 at 12:00 PM, Rico Bergmann

Splitting Streams

Hej, I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning) The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have

Event time in Flink streaming

Hej, I have a stream of timestamped events I want to process in Flink streaming. Di I have to write my own policies to do so, or can define time based windows to use the timestamps instead of the system time? cheers Martin

Re: Event time in Flink streaming

to implement you own policy. However, this should be be complicated. Have a look at TimeTriggerPolicy. You just need to provide a Timestamp implementation that extracts you ts-attribute from the tuples. -Matthias On 08/28/2015 03:58 PM, Martin Neumann wrote: Hej, I have a stream of timestamped