Hi, MIch, thanks for replying. I'm deploying from the same instance from where I showed the logs and commands using --deploy-mode cluster.
The SparkSubmit process only appears while the bin/spark-submit binary is active. When the application starts and the driver takes control, the SparkSubmit process dies. 2016-08-01 16:07 GMT-03:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > OK I can see the Worker (19286 Worker and the executor(6548 > CoarseGrainedExecutorBackend) running on it > > Where is spark-submit? Did you submit your job from another node or used > another method to run it? > > HTH > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 1 August 2016 at 19:08, Maximiliano Patricio Méndez < > mmen...@despegar.com> wrote: > >> I just recently tried again, the port 4040 is not used. And even if it >> were, I think the log would reflect that trying to use the following port >> (4041) as you mentioned. >> >> This is what the driver log says: >> >> 16/08/01 13:55:56 INFO Utils: Successfully started service 'SparkUI' on port >> 4040. >> 16/08/01 13:55:56 INFO SparkUI: Started SparkUI at http://hostname:4040 >> >> >> If I go to {hostname}: >> ~$ jps >> 6548 CoarseGrainedExecutorBackend >> 19286 Worker >> 6843 Jps >> 19182 Master >> >> ~$ netstat -nltp >> Active Internet connections (only servers) >> Proto Recv-Q Send-Q Local Address Foreign Address State >> PID/Program name >> tcp6 0 0 192.168.22.245:43037 :::* >> LISTEN 6548/java >> tcp6 0 0 192.168.22.245:56929 :::* >> LISTEN 19286/java >> tcp6 0 0 192.168.22.245:7077 :::* >> LISTEN 19182/java >> tcp6 0 0 :::33296 :::* >> LISTEN 6548/java >> tcp6 0 0 :::8080 :::* >> LISTEN 19182/java >> tcp6 0 0 :::8081 :::* >> LISTEN 19286/java >> tcp6 0 0 192.168.22.245:6066 :::* >> LISTEN 19182/java >> >> ~$ netstat -nltap | grep 4040 >> >> I'm really lost here and don't know much about spark yet, but shouldn't >> there be a DriverWrapper process which holds the bind on port 4040? >> >> >> 2016-08-01 13:49 GMT-03:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >> >>> Can you check if port 4040 is actually used? If it used the next >>> available one would 4041. For example below Zeppelin uses it >>> >>> >>> *netstat -plten|grep 4040*tcp 0 0 >>> :::4040 :::* LISTEN >>> 1005 73372882 *10699*/java >>> *ps aux|grep 10699* >>> hduser 10699 0.1 3.8 3172308 952932 pts/3 SNl Jul30 5:57 >>> /usr/java/latest/bin/java -cp /data6/hduser/zeppelin-0.6.0/ ... >>> >>> HTH >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 1 August 2016 at 17:44, Maximiliano Patricio Méndez < >>> mmen...@despegar.com> wrote: >>> >>>> Hi, >>>> >>>> Thanks for the answers. >>>> >>>> @Jacek: To verify if the ui is up, I enter to all the worker nodes of >>>> my cluster and run netstat -nltp | grep 4040 with no result. The log of the >>>> driver tells me in which server and on which port should the spark ui be >>>> up, but it isn't. >>>> >>>> >>>> @Mich: I've tried to specify spark.ui.port=nnn but I only manage to >>>> change the log, reporting that the driver should be in another port. >>>> >>>> The ui has no problem to start in that port (4040) when I run my >>>> application in client mode. >>>> >>>> Could there be a network issue making the ui to fail silently? I've >>>> read some of the code regarding those parts of the driver log, but couldn't >>>> find anything weird. >>>> >>>> 2016-07-29 19:45 GMT-03:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >>>> >>>>> why chance it. Best to explicitly specify in spark-submit (or >>>>> whatever) which port to listen to >>>>> >>>>> --conf "spark.ui.port=nnn" >>>>> >>>>> and see if it works >>>>> >>>>> HTH >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 29 July 2016 at 23:37, Jacek Laskowski <ja...@japila.pl> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm curious about "For some reason, sometimes the SparkUI does not >>>>>> appear to be bound on port 4040 (or any other) but the application >>>>>> runs perfectly and finishes giving the expected answer." How do you >>>>>> check that web UI listens to the port 4040? >>>>>> >>>>>> Pozdrawiam, >>>>>> Jacek Laskowski >>>>>> ---- >>>>>> https://medium.com/@jaceklaskowski/ >>>>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >>>>>> Follow me at https://twitter.com/jaceklaskowski >>>>>> >>>>>> >>>>>> On Thu, Jul 28, 2016 at 11:37 PM, Maximiliano Patricio Méndez >>>>>> <mmen...@despegar.com> wrote: >>>>>> > Hi, >>>>>> > >>>>>> > I'm having some trouble trying to submit an application to my spark >>>>>> cluster. >>>>>> > For some reason, sometimes the SparkUI does not appear to be bound >>>>>> on port >>>>>> > 4040 (or any other) but the application runs perfectly and finishes >>>>>> giving >>>>>> > the expected answer. >>>>>> > >>>>>> > And don't know why, but if I restart all the workers at once >>>>>> sometimes it >>>>>> > begins to work and sometimes it doesn't. >>>>>> > >>>>>> > In the driver logs, when it fails to start the SparkUI I see some >>>>>> these >>>>>> > lines: >>>>>> > 16/07/28 16:13:37 INFO Utils: Successfully started service >>>>>> 'SparkUI' on port >>>>>> > 4040. >>>>>> > 16/07/28 16:13:37 INFO SparkUI: Started SparkUI at >>>>>> http://hostname-00:4040 >>>>>> > >>>>>> > but nothing running in those ports. >>>>>> > >>>>>> > I'm attaching the full driver log in which I've activated jetty >>>>>> logs on >>>>>> > DEBUG but couldn't find anything. >>>>>> > >>>>>> > The only properties that I'm not leaving at default at the >>>>>> configuration is >>>>>> > the SPARK_PUBLIC_DNS=$(hostname), SPARK_WORKER_CORES and >>>>>> SPARK_WORKER_MEMORY >>>>>> > >>>>>> > Have anyone faced something similar? >>>>>> > >>>>>> > Thanks >>>>>> > >>>>>> > >>>>>> > >>>>>> --------------------------------------------------------------------- >>>>>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>>> >>>>>> >>>>> >>>> >>> >> >
