hi Taylor, If I guess correctly, Usually you submit jobs by scp jars to cluster, ssh to cluster and launch jobs by spart-submit CLI?
We will discuss whether/when to support this use case in our dev list. Thanks, William On Mon, Apr 30, 2018 at 8:22 PM, Darrell Taylor < [email protected]> wrote: > Hi William, > > > > Thanks for the info. I understand the potential firewall issues, we > already addressed this for applications we deploy around the edge of our > cluster. > > > > Currently the main reason for not using Livy is because it’s not currently > available in our cluster. > > > > I was just curious how easy it would be to make it optional? > > > > Thanks > > Darrell. > > > > > > *From: *William Guo <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, 30 April 2018 at 12:35 > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Disable Livy option? > > > > Hi Taylor, > > > > The job submission is triggered by our job scheduler(quartz), by quartz, > we periodically call livy service to submit spark jobs. > > > > We have thought submit jobs directly, but in many cases, that will cause > firewall/security/auth issues. > > > > We leverage livy to submit jobs from http layer, because most cluster are > accessible from http layer. > > > > What is your issue for using livy? > > > > Thanks, > > William > > > > On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor <darrell.taylor@ > skybettingandgaming.com> wrote: > > Hi, > > > > I was wondering if it’s feasible to disable the Livy option and have the > ability to submit Spark jobs directly? > > > > Having had a brief look through the code it seems like all the SparkConf > and SparkContext is handled as normal, but couldn’t quite figure out where > the job submission gets routed to Livy. > > > Thanks > > -- > > > Information in this email including any attachments may be privileged, > confidential and is intended exclusively for the addressee. The views > expressed may not be official policy, but the personal views of the > originator. If you have received it in error, please notify the sender by > return e-mail and delete it from your system. You should not reproduce, > distribute, store, retransmit, use or disclose its contents to anyone. > Please note we reserve the right to monitor all e-mail communication > through our internal and external networks. The SKY trademarks are owned by > Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No. > 1110) with registered address Office 1, 1 The Crusher, Braye Harbour, > Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with > registered address 2 Wellington Place, Leeds, LS1 4AP > <https://maps.google.com/?q=2+Wellington+Place,+Leeds,+LS1+4AP&entry=gmail&source=g> > and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address > Century House, 12 Victoria Street, Alderney, GY9 3UF > <https://maps.google.com/?q=12+Victoria+Street,+Alderney,+GY9+3UF&entry=gmail&source=g> > operate the Sky Betting and Gaming and Oddschecker services. >
