On 05/23/2016 07:18 PM, Radoslaw Gruchalski wrote: > Sounds surprisingly close to this: > https://github.com/apache/spark/pull/9608 > I might have overlooked it but bridge mode work appears to make Spark work with docker containers and able to communicate with them when running on more than one machines.
Here I am trying to enable getting information from Spark UI irrespective of Spark running in containers or not. Spark UI's link to workers and application drivers are pointing to internal/protected network. So to get this information from user's machine, he/she has to connect to VPN. Therefore the proposal is to make Spark master UI reverse proxy this information back to user. So only Spark master UI needs to be opened up to internet and there is no need to change anything else how Spark runs internally either in Standalone mode, Mesos or in containers on kubernetes. - Gurvinder > I can ressurect the work on the bridge mode for Spark 2. The reason why > the work on the old one was suspended was because Spark was going > through so many changes at that time that a lot of work done, was wiped > out by the changes towards 2.0. > > I know that Lightbend was also interested in having bridge mode. > > – > Best regards, > Radek Gruchalski > ra...@gruchalski.com <mailto:ra...@gruchalski.com> > de.linkedin.com/in/radgruchalski > > *Confidentiality: > *This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > > On May 23, 2016 at 7:14:51 PM, Timothy Chen (tnac...@gmail.com > <mailto:tnac...@gmail.com>) wrote: > >> This will also simplify Mesos users as well, DCOS has to work around >> this with our own proxying. >> >> Tim >> >> On Sun, May 22, 2016 at 11:53 PM, Gurvinder Singh >> <gurvinder.si...@uninett.no> wrote: >> > Hi Reynold, >> > >> > So if that's OK with you, can I go ahead and create JIRA for this. As it >> > seems this feature is missing currently and can benefit not just for >> > kubernetes users but in general Spark standalone mode users too. >> > >> > - Gurvinder >> > On 05/22/2016 12:49 PM, Gurvinder Singh wrote: >> >> On 05/22/2016 10:23 AM, Sun Rui wrote: >> >>> If it is possible to rewrite URL in outbound responses in Knox or other >> >>> reverse proxy, would that solve your issue? >> >> Any process which can keep track of workers and application drivers IP >> >> addresses and route traffic to those will work. Considering Spark Master >> >> does exactly this due to all workers and application has to register to >> >> the master, therefore I propose master to be the place to add such a >> >> functionality. >> >> >> >> I am not aware with Knox capabilities but Nginx or any other normal >> >> reverse proxy will not be able to this on its own due to dynamic nature >> >> of application drivers and to some extent workers too. >> >> >> >> - Gurvinder >> >>>> On May 22, 2016, at 14:55, Gurvinder Singh <gurvinder.si...@uninett.no> >> >>>> wrote: >> >>>> >> >>>> On 05/22/2016 08:32 AM, Reynold Xin wrote: >> >>>>> Kubernetes itself already has facilities for http proxy, doesn't it? >> >>>>> >> >>>> Yeah kubernetes has ingress controller which can act the L7 load >> >>>> balancer and router traffic to Spark UI in this case. But I am referring >> >>>> to link present in UI to worker and application UI. Replied in the >> >>>> detail to Sun Rui's mail where I gave example of possible scenario. >> >>>> >> >>>> - Gurvinder >> >>>>> >> >>>>> On Sat, May 21, 2016 at 9:30 AM, Gurvinder Singh >> >>>>> <gurvinder.si...@uninett.no <mailto:gurvinder.si...@uninett.no>> wrote: >> >>>>> >> >>>>> Hi, >> >>>>> >> >>>>> I am currently working on deploying Spark on kuberentes (K8s) and >> >>>>> it is >> >>>>> working fine. I am running Spark with standalone mode and >> >>>>> checkpointing >> >>>>> the state to shared system. So if master fails K8s starts it and >> >>>>> from >> >>>>> checkpoint it recover the earlier state and things just works fine. >> >>>>> I >> >>>>> have an issue with the Spark master Web UI to access the worker and >> >>>>> application UI links. In brief, kubernetes service model allows me >> >>>>> to >> >>>>> expose the master service to internet, but accessing the >> >>>>> application/workers UI is not possible as then I have to expose >> >>>>> them too >> >>>>> individually and given I can have multiple application it becomes >> >>>>> hard >> >>>>> to manage. >> >>>>> >> >>>>> One solution can be that the master can act as reverse proxy to >> >>>>> access >> >>>>> information/state/logs from application/workers. As it has the >> >>>>> information about their endpoint when application/worker register >> >>>>> with >> >>>>> master, so when a user initiate a request to access the information, >> >>>>> master can proxy the request to corresponding endpoint. >> >>>>> >> >>>>> So I am wondering if someone has already done work in this direction >> >>>>> then it would be great to know. If not then would the community >> >>>>> will be >> >>>>> interesting in such feature. If yes then how and where I should get >> >>>>> started as it would be helpful for me to have some guidance to start >> >>>>> working on this. >> >>>>> >> >>>>> Kind Regards, >> >>>>> Gurvinder >> >>>>> >> >>>>> >> >>>>> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >>>>> <mailto:dev-unsubscr...@spark.apache.org> >> >>>>> For additional commands, e-mail: dev-h...@spark.apache.org >> >>>>> <mailto:dev-h...@spark.apache.org> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >>>> For additional commands, e-mail: dev-h...@spark.apache.org >> >>>> >> >>> >> >>> >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >>> For additional commands, e-mail: dev-h...@spark.apache.org >> >>> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> > For additional commands, e-mail: dev-h...@spark.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
