Re: kafka and flink integration issue
Good to hear. Thanks for letting us know!
On Mon, Feb 29, 2016 at 8:14 PM, Pankaj Kumar wrote:
> yes versioning was issue . Job is working fine on flink 0.10.2.
>
> On Mon, Feb 29, 2016 at 3:15 PM, Stephan Ewen wrote:
>
>> Hi!
>>
>> A "NoSuchMethodError" is always a sign of a version mixup. Please make
>> sure both versions (cluster and client) are exactly the same.
>>
>> Stephan
>>
>>
>> On Sat, Feb 27, 2016 at 11:05 AM, Pankaj Kumar
>> wrote:
>>
>>> Yes Robert ,
>>> i was trying to start Flink on cluster 0.10.1.
>>>
>>> But after changing flink version to 0.10.1 , also i am getting the same
>>> error.
>>>
>>> On Sat, Feb 27, 2016 at 2:47 PM, Robert Metzger
>>> wrote:
>>>
Hi Pankaj,
I suspect you are trying to start Flink on a cluster with Flink 0.10.1
installed?
On Sat, Feb 27, 2016 at 9:20 AM, Pankaj Kumar
wrote:
> I am trying to integrate kafka and flink.
> my pom file is where {flink.version} is 0.10.2
>
>
> org.apache.flink
> flink-java
> ${flink.version}
>
>
> org.apache.flink
> flink-streaming-java
> ${flink.version}
>
>
> org.apache.flink
> flink-clients
> ${flink.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> 2.6.4
>
>
> org.projectlombok
> lombok
> 1.16.6
> provided
>
>
> org.springframework
> spring-web
> 4.2.4.RELEASE
>
>
> org.apache.flink
> flink-connector-kafka
> ${flink.version}
>
>
> when i am running my jar with flink command line using command
>
> bin/flink run ~/apache_flink/target/Application-0.0.1.jar
>
> I am getting error
>
> tatus of job d20986a31f2a8166986d7db354de0ff7 changed to FAILED.
> java.lang.NoSuchMethodError:
> org.apache.flink.streaming.api.operators.StreamingRuntimeContext.isCheckpointingEnabled()Z
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.run(FlinkKafkaConsumer.java:387)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:55)
>
>
> I am building final jar using command *mvn clean package -Pbuild-jar*
>
>
>>>
>>
>
Re: kafka and flink integration issue
yes versioning was issue . Job is working fine on flink 0.10.2.
On Mon, Feb 29, 2016 at 3:15 PM, Stephan Ewen wrote:
> Hi!
>
> A "NoSuchMethodError" is always a sign of a version mixup. Please make
> sure both versions (cluster and client) are exactly the same.
>
> Stephan
>
>
> On Sat, Feb 27, 2016 at 11:05 AM, Pankaj Kumar
> wrote:
>
>> Yes Robert ,
>> i was trying to start Flink on cluster 0.10.1.
>>
>> But after changing flink version to 0.10.1 , also i am getting the same
>> error.
>>
>> On Sat, Feb 27, 2016 at 2:47 PM, Robert Metzger
>> wrote:
>>
>>> Hi Pankaj,
>>>
>>> I suspect you are trying to start Flink on a cluster with Flink 0.10.1
>>> installed?
>>>
>>> On Sat, Feb 27, 2016 at 9:20 AM, Pankaj Kumar
>>> wrote:
>>>
I am trying to integrate kafka and flink.
my pom file is where {flink.version} is 0.10.2
org.apache.flink
flink-java
${flink.version}
org.apache.flink
flink-streaming-java
${flink.version}
org.apache.flink
flink-clients
${flink.version}
com.fasterxml.jackson.core
jackson-databind
2.6.4
org.projectlombok
lombok
1.16.6
provided
org.springframework
spring-web
4.2.4.RELEASE
org.apache.flink
flink-connector-kafka
${flink.version}
when i am running my jar with flink command line using command
bin/flink run ~/apache_flink/target/Application-0.0.1.jar
I am getting error
tatus of job d20986a31f2a8166986d7db354de0ff7 changed to FAILED.
java.lang.NoSuchMethodError:
org.apache.flink.streaming.api.operators.StreamingRuntimeContext.isCheckpointingEnabled()Z
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.run(FlinkKafkaConsumer.java:387)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:55)
I am building final jar using command *mvn clean package -Pbuild-jar*
>>>
>>
>
Re: Iterations problem in command line
Thanks Fabian, I am using in both default options, since I am not testing in a cluster yet, just local in ubuntu, I am not specifying any parallelism. just to test I set in the program env.setParallelism(1) and running with -p 1 (which I guess I would not need) but I am still getting the same issue. Regards, MArcela. On 29.02.2016 16:44, Fabian Hueske wrote: Hi Marcela, do you run the algorithm in both setups with the same parallelism? Best, Fabian 2016-02-26 16:52 GMT+01:00 Marcela Charfuelan mailto:charfuelanol...@tu-berlin.de>>: Hello, I implemented an algorithm that includes iterations (EM algorithm) and I am getting different results when running in eclipse (Luna Release (4.4.0)) and when running in the command line using Flink run; the program does not crash is just that after the first iteration the results are different (wrong in the command line). The solution I am getting in eclipse, for each iteration, is the same that I would get if running the algorithm in octave for example, so I am sure the solution is correct. I have tried using java plus the command-line arguments of eclipse on the command line and that also works ok (local in ubuntu). Has anybody experienced something similar? Any idea why this could happen? how can I fix this? Regards, Marcela.
Re: Iterations problem in command line
Hi Marcela, do you run the algorithm in both setups with the same parallelism? Best, Fabian 2016-02-26 16:52 GMT+01:00 Marcela Charfuelan : > Hello, > > I implemented an algorithm that includes iterations (EM algorithm) and I > am getting different results when running in eclipse (Luna Release (4.4.0)) > and when running in the command line using Flink run; the program does not > crash is just that after the first iteration the results are different > (wrong in the command line). > > The solution I am getting in eclipse, for each iteration, is the same that > I would get if running the algorithm in octave for example, so I am sure > the solution is correct. > > I have tried using java plus the command-line arguments of eclipse on the > command line and that also works ok (local in ubuntu). > > Has anybody experienced something similar? Any idea why this could happen? > how can I fix this? > > Regards, > Marcela. >
Re: filter dataset
Hi Lydia,
you can use a variable outside the scope of the normal filter (the one that
you define inline, if I understand correctly), like this:
*final* int *filterParameter* = 0;
dataset.filter(new FilterFunction () {
@Override
public boolean filter(Integer value) {
return value == *filterParameter*;
}
}
The only catch is that you have to make the variable *final*.
Does this answer your question? Hope I've been helpful.
Best,
Stefano
On Mon, Feb 29, 2016 at 11:14 AM, Lydia Ickler
wrote:
> Hi all,
>
> I have a DataSet and I want to apply a
> filter to only get back all entries with e.g. first Integer in tuple == 0.
> With a normal filter I do not have the possibility to pass an an
> additional argument but I have to set that parameter inside the filter
> function.
> Is there a possibility to send a parameter to make the filter more
> flexible?
> How would be the smartest way to do so?
>
> Best regards,
> Lydia
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
filter dataset
Hi all, I have a DataSet and I want to apply a filter to only get back all entries with e.g. first Integer in tuple == 0. With a normal filter I do not have the possibility to pass an an additional argument but I have to set that parameter inside the filter function. Is there a possibility to send a parameter to make the filter more flexible? How would be the smartest way to do so? Best regards, Lydia
Re: IOException when trying flink-twitter example
Hi! Thanks for figuring this out. Turns out that this issue was caused by a conflict between versions of the Apache HTTP Components library used by the Twitter Client, and used by the Hadoop/HDFS libraries. >From version 1.0 on, we shade that library in Hadoop, so that issue should be solved and the example work without this workaround. Stephan On Sun, Feb 28, 2016 at 11:46 AM, Igor Berman wrote: > > http://stackoverflow.com/questions/32379590/ioexcpetion-while-connecting-to-twitter-streaming-api-with-apache-flink > > there is a problem with http client, > I've solved it by downloading httpclient-4.2.6.jar jar into flink/lib and > renamed it to have "a" (ahttpclient-4.2.6.jar) so it will be added to > classpath of flink runtime first(by bin/config.sh) > > On 21 February 2016 at 22:56, ram kumar wrote: > >> tried running the TwitterFilterSourceExample as java application, works >> fine. >> facing issue when submitting as flink. looking into flink libraries >> >> On Wed, Feb 17, 2016 at 8:22 PM, Stephan Ewen wrote: >> >>> Looks like an issue with the Twitter Client. >>> >>> Maybe the log reveals more that can help you figure out what is >>> happening (loss of connection, etc). >>> >>> On Mon, Feb 15, 2016 at 1:32 PM, ram kumar >>> wrote: >>> org.apache.flink.streaming.connectors.twitter.TwitterFilterSource - Initializing Twitter Streaming API connection 12:27:32,134 INFO com.twitter.hbc.httpclient.BasicClient- New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length 12:27:32,135 INFO org.apache.flink.streaming.connectors.twitter.TwitterFilterSource - Twitter Streaming API connection established successfully 12:27:32,233 INFO com.twitter.hbc.httpclient.ClientBase - twitterSourceClient Establishing a connection 12:27:32,296 WARN com.twitter.hbc.httpclient.ClientBase - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length When i run the job, at first the connection is established, then this log appears 08:25:19,207 WARN com.twitter.hbc.httpclient.ClientBase - twitterSourceClient failed to establish connection properly 08:25:19,207 INFO com.twitter.hbc.httpclient.ClientBase - twitterSourceClient Done processing, preparing to close connection 08:25:19,208 INFO com.twitter.hbc.httpclient.ClientBase - twitterSourceClient Establishing a connection 08:25:19,225 WARN com.twitter.hbc.httpclient.ClientBase - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels wrote: > Hi Ram, > > This is an Exception thrown in the Twitter client. Unfortunately the > exception is caught and not printed to the screen. > > I would suggest to set a breakpoint in the ClientBase to find out > about the reason for the failed connection. > > Cheers, > Max > > On Thu, Feb 11, 2016 at 9:30 AM, ram kumar > wrote: > > Hi, > > > > Got following exception in taskmanager logs when running > > TwitterFilterSourceExample, > > > > 08:25:19,207 WARN com.twitter.hbc.httpclient.ClientBase > > - twitterSourceClient failed to establish connection properly > > 08:25:19,207 INFO com.twitter.hbc.httpclient.ClientBase > > - twitterSourceClient Done processing, preparing to close connection > > 08:25:19,208 INFO com.twitter.hbc.httpclient.ClientBase > > - twitterSourceClient Establishing a connection > > 08:25:19,225 WARN com.twitter.hbc.httpclient.ClientBase > > - twitterSourceClient IOException caught when establishing > connection to > > https://stream.twitter.com/1.1/statuses/filter.json?delimited=length > > > > > > > > Anyone faced this same issue? > > > > Thanks > >>> >> >
Re: kafka and flink integration issue
Hi!
A "NoSuchMethodError" is always a sign of a version mixup. Please make sure
both versions (cluster and client) are exactly the same.
Stephan
On Sat, Feb 27, 2016 at 11:05 AM, Pankaj Kumar wrote:
> Yes Robert ,
> i was trying to start Flink on cluster 0.10.1.
>
> But after changing flink version to 0.10.1 , also i am getting the same
> error.
>
> On Sat, Feb 27, 2016 at 2:47 PM, Robert Metzger
> wrote:
>
>> Hi Pankaj,
>>
>> I suspect you are trying to start Flink on a cluster with Flink 0.10.1
>> installed?
>>
>> On Sat, Feb 27, 2016 at 9:20 AM, Pankaj Kumar
>> wrote:
>>
>>> I am trying to integrate kafka and flink.
>>> my pom file is where {flink.version} is 0.10.2
>>>
>>>
>>> org.apache.flink
>>> flink-java
>>> ${flink.version}
>>>
>>>
>>> org.apache.flink
>>> flink-streaming-java
>>> ${flink.version}
>>>
>>>
>>> org.apache.flink
>>> flink-clients
>>> ${flink.version}
>>>
>>>
>>> com.fasterxml.jackson.core
>>> jackson-databind
>>> 2.6.4
>>>
>>>
>>> org.projectlombok
>>> lombok
>>> 1.16.6
>>> provided
>>>
>>>
>>> org.springframework
>>> spring-web
>>> 4.2.4.RELEASE
>>>
>>>
>>> org.apache.flink
>>> flink-connector-kafka
>>> ${flink.version}
>>>
>>>
>>> when i am running my jar with flink command line using command
>>>
>>> bin/flink run ~/apache_flink/target/Application-0.0.1.jar
>>>
>>> I am getting error
>>>
>>> tatus of job d20986a31f2a8166986d7db354de0ff7 changed to FAILED.
>>> java.lang.NoSuchMethodError:
>>> org.apache.flink.streaming.api.operators.StreamingRuntimeContext.isCheckpointingEnabled()Z
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.run(FlinkKafkaConsumer.java:387)
>>> at
>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:55)
>>>
>>>
>>> I am building final jar using command *mvn clean package -Pbuild-jar*
>>>
>>>
>>
>
Re: flink-storm FlinkLocalCluster issue
Hi Zhang, Please have a look here for the 1.0.0-rc2: Binaries: http://people.apache.org/~rmetzger/flink-1.0.0-rc2/ Maven repository: https://repository.apache.org/content/repositories/orgapacheflink-1064 Cheers, Max On Sat, Feb 27, 2016 at 4:00 AM, #ZHANG SHUHAO# wrote: > Thanks for the confirmation. > > When will 1.0 be ready in maven repo? > > > > From: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] On Behalf Of > Stephan Ewen > Sent: Friday, February 26, 2016 9:07 PM > To: user@flink.apache.org > Subject: Re: flink-storm FlinkLocalCluster issue > > > > Hi! > > > > On 0.10.x, the Storm compatibility layer does not properly configure the > Local Flink Executor to have the right parallelism. > > > > In 1.0 that is fixed. If you try the latest snapshot, or the > 1.0-Release-Candidate-1, it should work. > > > > Greetings, > > Stephan > > > > > > On Fri, Feb 26, 2016 at 12:16 PM, #ZHANG SHUHAO# > wrote: > > Hi till, > > > > Thanks for your reply. > > But it appears that it only started with #slot of 1. > > I have traced down to the source code of flink step by step, where I have > confirmed it. > > > > I'm using flink 0.10.2, source code downloaded from flink website. Nothing > have been changed. I simply try to run the flink-Storm word count local > example. > > > > It just failed to work. > > > > > > Sent from my iPhone > > > On 26 Feb 2016, at 6:16 PM, Till Rohrmann wrote: > > Hi Shuhao, > > the configuration you’re providing is only used for the storm compatibility > layer and not Flink itself. When you run your job locally, the > LocalFlinkMiniCluster should be started with as many slots as your maximum > degree of parallelism is in your topology. You can check this in > FlinkLocalCluster.java:96. When you submit your job to a remote cluster, > then you have to define the number of slots in the flink-conf.yaml file. > > Cheers, > Till > > > > On Fri, Feb 26, 2016 at 10:34 AM, #ZHANG SHUHAO# > wrote: > > Hi everyone, > > > > I’m a student researcher working on Flink recently. > > > > I’m trying out the flink-storm example project, version 0.10.2, > flink-storm-examples, word-count-local. > > > > But, I got the following error: > > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: > Not enough free slots available to run the job. You can decrease the > operator parallelism or increase the number of slots per TaskManager in the > configuration. Task to schedule: < Attempt #0 (tokenizer (2/4)) @ > (unassigned) - [SCHEDULED] > with groupID < b86aa7eb76c417c63afb577ea46ddd72 >> in sharing group < SlotSharingGroup [0ea70b6a3927c02f29baf9de8f59575b, > b86aa7eb76c417c63afb577ea46ddd72, cbc10505364629b45b28e79599c2f6b8] >. > Resources available to scheduler: Number of instances=1, total number of > slots=1, available slots=0 > > at > org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:255) > > at > org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:131) > > at > org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:298) > > at > org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:458) > > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.scheduleAll(ExecutionJobVertex.java:322) > > at > org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:686) > > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply$mcV$sp(JobManager.scala:992) > > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:972) > > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$1.apply(JobManager.scala:972) > > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > > at > akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401) > > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) > > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > > >
