Re: Towards Flink 0.10
Hello everyone. Stephan, will 0.10 include the cache issues for createRemoteEnvironment? That not every addressing of environment will retransmit all the jars? On Tue, Oct 6, 2015 at 1:16 PM, Stephan Ewen <se...@apache.org> wrote: > How about making a quick effort for this: > https://issues.apache.org/jira/browse/FLINK-2313 > > It introduces an execution control handle (returned by > StreamExecutionEnvironment.execute()) for operations like cancel(), > getAccumulators(), scaleIn/Out(), ... > It would be big time API breaking, so would be good to have it before the > release. > > As a first version we could return a control handle that has only a single > method "waitForCompletion()" to emulate the current behavior. > Or we postpone it and later add a method "executeWithControl()" or so, that > returns the control handle. > > > On Mon, Oct 5, 2015 at 6:34 PM, Vasiliki Kalavri < > vasilikikala...@gmail.com> > wrote: > > > Yes, FLINK-2785 that's right! > > Alright, thanks a lot! > > > > On 5 October 2015 at 18:31, Fabian Hueske <fhue...@gmail.com> wrote: > > > > > Hi Vasia, > > > > > > I guess you are referring to FLINK-2785. Should be fine, as there is a > PR > > > already. > > > I'll add it to the list. > > > > > > Would be nice if you could take care of FLINK-2786 (remove Spargel). > > > > > > Cheers, Fabian > > > > > > 2015-10-05 18:25 GMT+02:00 Vasiliki Kalavri <vasilikikala...@gmail.com > >: > > > > > > > Thank you Max for putting the list together and to whomever added > > > > FLINK-2561 to the list :) > > > > I would also add FLINK-2561 (pending PR #1205). It's a sub-task of > > > > FLINK-2561, so maybe it's covered as is. > > > > > > > > If we go for Gelly graduation, I can take care of FLINK-2786 "Remove > > > > Spargel from source code and update documentation in favor of Gelly", > > but > > > > maybe it makes sense to wait for the restructuring, since we're > getting > > > rid > > > > of staging altogether? > > > > > > > > -V. > > > > > > > > On 5 October 2015 at 17:51, Fabian Hueske <fhue...@gmail.com> wrote: > > > > > > > > > Thanks Max. > > > > > I extended the list of issues to fix for the release. > > > > > > > > > > 2015-10-05 17:10 GMT+02:00 Maximilian Michels <m...@apache.org>: > > > > > > > > > > > Thanks Greg, we have added that to the list of API breaking > > changes. > > > > > > > > > > > > On Mon, Oct 5, 2015 at 4:36 PM, Greg Hogan <c...@greghogan.com> > > > wrote: > > > > > > > > > > > > > Max, > > > > > > > > > > > > > > Stephan noted that FLINK-2723 is an API breaking change. The > > > > > > CopyableValue > > > > > > > interface has a new method "T copy()". Commit > > > > > > > e727355e42bd0ad7d403aee703aaf33a68a839d2 > > > > > > > > > > > > > > Greg > > > > > > > > > > > > > > On Mon, Oct 5, 2015 at 10:20 AM, Maximilian Michels < > > > m...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Flinksters, > > > > > > > > > > > > > > > > After a lot of development effort in the past months, it is > > about > > > > > time > > > > > > > > to move towards the next major release. We decided to move > > > towards > > > > > > > > 0.10 instead of a milestone release. This release will > probably > > > be > > > > > the > > > > > > > > last release before 1.0. > > > > > > > > > > > > > > > > For 0.10 we most noticeably have the new Streaming API which > > > comes > > > > > > > > with an improved runtime including exactly-once sources and > > > sinks. > > > > > > > > Additionally, we have a new web interface with support for > > > > > > > > live-monitoring. Not to mention the countless fixes and > > > > improvements. > > > > > > > > > > > > > > > > I've been ransacking the JIRA issues to find out what issues > we > > > > have > > > > > > > > to fix before we can release. I've put these issues on the > 0.10 > > > > wiki > > > > > > > > page: > > > > https://cwiki.apache.org/confluence/display/FLINK/0.10+Release > > > > > > > > > > > > > > > > It would be great to fix all of these issues. However, I > think > > we > > > > > need > > > > > > > > to be pragmatic and pick the most pressing ones. Let's do > that > > on > > > > the > > > > > > > > wiki page on the "To Be Fixed" section. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Max > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Regards* *Alexey Sapozhnikov* CTO& Co-Founder Scalabillit Inc Aba Even 10-C, Herzelia, Israel M : +972-52-2363823 E : ale...@scalabill.it W : http://www.scalabill.it YT - https://youtu.be/9Rj309PTOFA Map:http://mapta.gs/Scalabillit Revolutionizing Proof-of-Concept
Re: Flink ML linear regression issue
Hello everyone. Do you have a sample in Java how to implement Flink MultipleLinearRegression example? Scala is great, however we would like to see the exact example we could invoke it from Java if it is possible. Thanks and sorry for the interrupt. On Thu, Sep 17, 2015 at 4:27 PM, Hanan Meyer <ha...@scalabill.it> wrote: > Hi > > I'm using Flink ML 9.2.1 in order to perform a multiple linear regression > with a csv data file. > > The Scala sample code for it is pretty straightforward: > val mlr = MultipleLinearRegression() > > val parameters = ParameterMap() > > parameters.add(MultipleLinearRegression.Stepsize, 2.0) > parameters.add(MultipleLinearRegression.Iterations, 10) > parameters.add(MultipleLinearRegression.ConvergenceThreshold, 0.001) > val inputDS = env.fromCollection(data) > > mlr.fit(inputDS, parameters) > > When I'm using Java(8) the fit method includes 3 parameters > 1. dataset > 2.parameters > 3. object which implements -fitOperation interface > > multipleLinearRegression.fit(regressionDS, parameters,fitOperation); > > Is there a need to implement the fitOperation interface which have been > already > implemented in Flinks ml source code. > > Another option is using MultipleLinearRegression.fitMLR() method ,but I > haven't found a way to pass the train dataset to it as a parameter or by > setter. > > I'll be more than happy if you could guide me how to implement it in Java > > Thanks > > Hanan Meyer > > > > > -- *Regards* *Alexey Sapozhnikov* CTO& Co-Founder Scalabillit Inc Aba Even 10-C, Herzelia, Israel M : +972-52-2363823 E : ale...@scalabill.it W : http://www.scalabill.it YT - https://youtu.be/9Rj309PTOFA Map:http://mapta.gs/Scalabillit Revolutionizing Proof-of-Concept
Re: Apache Flink:ProgramInvocationException on Yarn
Fully understand. 1.My suggestion is to drop Akka and take something else, since this issue is really big 2.Not hostname not the endpoint are not working, clarifying the VPC topic now. On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen se...@apache.org wrote: Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes annoying, but I have not found a good way around this. The problem is that the address to bind to an the address used by others to send messages to the node is the same. ( https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY) All of that aside, we have been using Flink on Cloud Platforms GCE, Amazon, and it worked. Can you try to use the internal hostname, or the fully qualified external hostname? On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hi. First off - many thanks for your efforts and prompt help. We will try to find how to do it with DNS server on VPC. however, absence of 0.0.0.0 is definitely a huge bug - just think about the current situation : if I dont have a VPC, I cant invoke the Flink functionality remotely in Amazon. We will clarify the VPC situation and I will write my impression of what we get. Thanks, guys On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger rmetz...@apache.org wrote: From this blog post, it seems that this hostname is not resolvable: https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html Can you easily activate a DNS server in the VPC? 0.0.0.0 is not supported because of some requirements of the Akka framework. But you should be able to use the host's IP address. Can you try that as well? On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Here is the exception from the moment we tried to put in jobmanager.rpc.address the hostname of the machine which is ip-172-36-98 looks like it doesnt recognize this address. Why it doesnt support 0.0.0.0 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager - 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager. 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system at ip-172-36-98:6123. 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager - Failed to run JobManager. java.lang.Exception: Could not create JobManager actor system at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882) at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826) at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) @ 21,1 71% On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger rmetz...@apache.org wrote: How is Flink crashing when you start it on the Linux machine in Amazon? Can you post the exception here? On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hello Stephan. We run this Linux machine on Amazon, which I predict, most of the people will do. We tried to put 0.0.0.0 or Public IP of the machine- Flink crashes on start, it doesnt recognize himself. It is very strange that it doesnt work with 0.0.0.0- basically this is a way in Java to make it being seen widely. We tried to put there the hostname - what you get from hostname command. It crashes. It works only with localhost and works only locally So what you suggest we will put there so the remote client could connect. ? Best regards On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen se...@apache.org wrote: Do you start Flink via YARN? In that case the jobmanager.rpc.address is not used, because YARN assigns containers/nodes. If you start Flink in standalone mode, this should be the address of the node that runs the JobManager. It will be used as the host/IP that Flink binds to. The same host should then be used in the RemoteEnvironment. Stephan
Re: Apache Flink:ProgramInvocationException on Yarn
Here is the exception from the moment we tried to put in
jobmanager.rpc.address the hostname of the machine which is ip-172-36-98
looks like it doesnt recognize this address.
Why it doesnt support 0.0.0.0
13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
-
13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
- Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf
13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
- Security is not enabled. Starting non-authenticated JobManager.
13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
- Starting JobManager
13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
- Starting JobManager actor system at ip-172-36-98:6123.
13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
- Failed to run JobManager.
java.lang.Exception: Could not create JobManager actor system
at
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
at
org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
at
org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
@
21,1
71%
On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger rmetz...@apache.org wrote:
How is Flink crashing when you start it on the Linux machine in Amazon?
Can you post the exception here?
On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov ale...@scalabill.it
wrote:
Hello Stephan.
We run this Linux machine on Amazon, which I predict, most of the people
will do.
We tried to put 0.0.0.0 or Public IP of the machine- Flink crashes on
start, it doesnt recognize himself.
It is very strange that it doesnt work with 0.0.0.0- basically this is a
way in Java to make it being seen widely.
We tried to put there the hostname - what you get from hostname
command.
It crashes.
It works only with localhost and works only locally
So what you suggest we will put there so the remote client could connect.
?
Best regards
On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen se...@apache.org wrote:
Do you start Flink via YARN? In that case the jobmanager.rpc.address
is
not used, because YARN assigns containers/nodes.
If you start Flink in standalone mode, this should be the address of
the
node that runs the JobManager. It will be used as the host/IP that
Flink
binds to. The same host should then be used in the RemoteEnvironment.
Stephan
On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer ha...@scalabill.it
wrote:
Hello all.
Firstly- thank you for your valuable advices.
We did some very fine tuned pinpoint test and comes to following
conclusions
1.We run on Ubuntu 14 flink for hadoop 2.7
2.Once we copy our Java client program directy to the machine and run
it
directly there it worked very good
The program is
.
ExecutionEnvironment envRemote
=ExecutionEnvironment.createRemoteEnvironment(localhost, 6123,
\usr\local\HananTestProj.jar);
org.apache.flink.api.java.DataSet text =
(org.apache.flink.api.java.DataSet) envRemote.fromElements(
Who's there?,
I think I hear them. Stand, ho! Who's there?);
org.apache.flink.api.java.DataSetTuple2String, Integer
wordCounts = text
.flatMap(new LineSplitter())
.groupBy(0)
.sum(1);
wordCounts.print();
}
public static class LineSplitter implements
FlatMapFunctionString,
Tuple2String, Integer {
public void flatMap(String line, CollectorTuple2String,
Integer
out) {
for (String word : line.split( )) {
out.collect(new Tuple2String, Integer(word, 1));
}
}
}
.
Program works fine
3.Now we are trying to run this program remotely , from windows
machine
when the first row looks differently
ExecutionEnvironment envRemote
=ExecutionEnvironment.createRemoteEnvironment(1.2.3.4, 6123,
C:\\HananTestProj.jar);
when 1.2.3.4 is IP address of fink machine
4.We got an exception :Jobmanager at 1.2.3.4 cant be reached bla bla
bla
5.in flink configuration we found a following line
jobmanager.rpc.address:
localhost
Flink cant be started with any other value (hostname/ipaddress )
except
the
localhost
6.We assume that probably Flink has a critical bug : it cant be
started
from remote machine, only locally. Are we right
Re: Apache Flink:ProgramInvocationException on Yarn
it will not help, since the internal IPs are changing in AWS from time to time and you should use only Public IP, which is not recognizable by flink. Thats why all app servers, for example JBoss or even Flume are using 0.0.0.0 On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen se...@apache.org wrote: What you can do as a temporary workaround is to actually enter the IP address for jobmanager.rpc.address - that circumvents the DNS. Just saw that Akka 2.4 (released some time in the near future) apparently introduces an option to listen to all network interfaces. On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Fully understand. 1.My suggestion is to drop Akka and take something else, since this issue is really big 2.Not hostname not the endpoint are not working, clarifying the VPC topic now. On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen se...@apache.org wrote: Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes annoying, but I have not found a good way around this. The problem is that the address to bind to an the address used by others to send messages to the node is the same. ( https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY) All of that aside, we have been using Flink on Cloud Platforms GCE, Amazon, and it worked. Can you try to use the internal hostname, or the fully qualified external hostname? On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hi. First off - many thanks for your efforts and prompt help. We will try to find how to do it with DNS server on VPC. however, absence of 0.0.0.0 is definitely a huge bug - just think about the current situation : if I dont have a VPC, I cant invoke the Flink functionality remotely in Amazon. We will clarify the VPC situation and I will write my impression of what we get. Thanks, guys On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger rmetz...@apache.org wrote: From this blog post, it seems that this hostname is not resolvable: https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html Can you easily activate a DNS server in the VPC? 0.0.0.0 is not supported because of some requirements of the Akka framework. But you should be able to use the host's IP address. Can you try that as well? On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Here is the exception from the moment we tried to put in jobmanager.rpc.address the hostname of the machine which is ip-172-36-98 looks like it doesnt recognize this address. Why it doesnt support 0.0.0.0 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager - 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager. 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system at ip-172-36-98:6123. 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager - Failed to run JobManager. java.lang.Exception: Could not create JobManager actor system at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882) at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826) at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) @ 21,1 71% On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger rmetz...@apache.org wrote: How is Flink crashing when you start it on the Linux machine in Amazon? Can you post the exception here? On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hello Stephan. We run this Linux machine on Amazon, which I predict, most of the people will do. We tried to put 0.0.0.0 or Public IP of the machine- Flink crashes on start, it doesnt recognize
Re: Apache Flink:ProgramInvocationException on Yarn
(BatchingExecutor.scala:110) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210) at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) at akka.actor.ActorCell.terminate(ActorCell.scala:369) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.Mailbox.exec(Mailbox.scala:231) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger rmetz...@apache.org wrote: I guess you are getting an entire exception after the org.apache.flink .client.program.ProgramInvocationException: Failed to resolve JobManager. Can you post it here to help us understanding the issue? On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hello all. Some clarification: locally everything works great. However once we run our Flink on remote linux machine and try to run the client program from our machine, using create remote environment- Flink JobManager is raising this exception On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen se...@apache.org wrote: If you start the job via the bin/flink script, then simply use ExecutionEnvironment.getExecutionEnvironment() rather then creating a remote environment manually. That way, hosts and ports are configured automatically. On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger rmetz...@apache.org wrote: Hi, Which values did you use for FLINK_SERVER_URL and FLINK_PORT? Every time you deploy Flink on YARN, the host and port change, because the JobManager is started on a different YARN container. On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer ha...@scalabill.it wrote: Hello All When using Eclipse IDE to submit Flink to Yarn single node cluster I'm getting : org.apache.flink.client.program.ProgramInvocationException: Failed to resolve JobManager Using Flink 0.9.0 The Jar copy a file from one location in Hdfs to another and works fine while executed locally on the single node Yarn cluster - bin/flink run -c Test ./examples/MyJar.jar hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt The code skeleton: ExecutionEnvironment envRemote = ExecutionEnvironment.createRemoteEnvironment (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT); DataSetString data = envRemote.readTextFile(hdfs://localhost:9000/flink/in.txt); data.writeAsText(hdfs
Re: Apache Flink:ProgramInvocationException on Yarn
Stephan it is exactly the same exception -UknownHost bal bla In Jboss for example the external are also not working, only the 0.0.0.0 - this is AWS NAT. We will proceed with VPC and then I will update you about what we get. Thanks for your help. On Sun, Aug 30, 2015 at 6:05 PM, Stephan Ewen se...@apache.org wrote: Why are the external IPs not working? Any kind of exception you can share? On Sun, Aug 30, 2015 at 5:02 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: it will not help, since the internal IPs are changing in AWS from time to time and you should use only Public IP, which is not recognizable by flink. Thats why all app servers, for example JBoss or even Flume are using 0.0.0.0 On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen se...@apache.org wrote: What you can do as a temporary workaround is to actually enter the IP address for jobmanager.rpc.address - that circumvents the DNS. Just saw that Akka 2.4 (released some time in the near future) apparently introduces an option to listen to all network interfaces. On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Fully understand. 1.My suggestion is to drop Akka and take something else, since this issue is really big 2.Not hostname not the endpoint are not working, clarifying the VPC topic now. On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen se...@apache.org wrote: Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes annoying, but I have not found a good way around this. The problem is that the address to bind to an the address used by others to send messages to the node is the same. ( https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY) All of that aside, we have been using Flink on Cloud Platforms GCE, Amazon, and it worked. Can you try to use the internal hostname, or the fully qualified external hostname? On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Hi. First off - many thanks for your efforts and prompt help. We will try to find how to do it with DNS server on VPC. however, absence of 0.0.0.0 is definitely a huge bug - just think about the current situation : if I dont have a VPC, I cant invoke the Flink functionality remotely in Amazon. We will clarify the VPC situation and I will write my impression of what we get. Thanks, guys On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger rmetz...@apache.org wrote: From this blog post, it seems that this hostname is not resolvable: https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html Can you easily activate a DNS server in the VPC? 0.0.0.0 is not supported because of some requirements of the Akka framework. But you should be able to use the host's IP address. Can you try that as well? On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov ale...@scalabill.it wrote: Here is the exception from the moment we tried to put in jobmanager.rpc.address the hostname of the machine which is ip-172-36-98 looks like it doesnt recognize this address. Why it doesnt support 0.0.0.0 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager - 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager. 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system at ip-172-36-98:6123. 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager - Failed to run JobManager. java.lang.Exception: Could not create JobManager actor system at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882) at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826) at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
Re: Apache Flink:ProgramInvocationException on Yarn
Hello all. Some clarification: locally everything works great. However once we run our Flink on remote linux machine and try to run the client program from our machine, using create remote environment- Flink JobManager is raising this exception On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen se...@apache.org wrote: If you start the job via the bin/flink script, then simply use ExecutionEnvironment.getExecutionEnvironment() rather then creating a remote environment manually. That way, hosts and ports are configured automatically. On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger rmetz...@apache.org wrote: Hi, Which values did you use for FLINK_SERVER_URL and FLINK_PORT? Every time you deploy Flink on YARN, the host and port change, because the JobManager is started on a different YARN container. On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer ha...@scalabill.it wrote: Hello All When using Eclipse IDE to submit Flink to Yarn single node cluster I'm getting : org.apache.flink.client.program.ProgramInvocationException: Failed to resolve JobManager Using Flink 0.9.0 The Jar copy a file from one location in Hdfs to another and works fine while executed locally on the single node Yarn cluster - bin/flink run -c Test ./examples/MyJar.jar hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt The code skeleton: ExecutionEnvironment envRemote = ExecutionEnvironment.createRemoteEnvironment (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT); DataSetString data = envRemote.readTextFile(hdfs://localhost:9000/flink/in.txt); data.writeAsText(hdfs://localhost:9000/flink/out.txt); envRemote.execute(); Please advise, Hanan Meyer
