Hi Taylor, The job submission is triggered by our job scheduler(quartz), by quartz, we periodically call livy service to submit spark jobs.
We have thought submit jobs directly, but in many cases, that will cause firewall/security/auth issues. We leverage livy to submit jobs from http layer, because most cluster are accessible from http layer. What is your issue for using livy? Thanks, William On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor < [email protected]> wrote: > Hi, > > > > I was wondering if it’s feasible to disable the Livy option and have the > ability to submit Spark jobs directly? > > > > Having had a brief look through the code it seems like all the SparkConf > and SparkContext is handled as normal, but couldn’t quite figure out where > the job submission gets routed to Livy. > > > Thanks > > -- > > *Darrell Taylor* > > Principal Engineer (Data Science) > > > > > > > > > > > Information in this email including any attachments may be privileged, > confidential and is intended exclusively for the addressee. The views > expressed may not be official policy, but the personal views of the > originator. If you have received it in error, please notify the sender by > return e-mail and delete it from your system. You should not reproduce, > distribute, store, retransmit, use or disclose its contents to anyone. > Please note we reserve the right to monitor all e-mail communication > through our internal and external networks. The SKY trademarks are owned by > Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No. > 1110) with registered address Office 1, 1 The Crusher, Braye Harbour, > Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with > registered address 2 Wellington Place, Leeds, LS1 4AP > <https://maps.google.com/?q=2+Wellington+Place,+Leeds,+LS1+4AP&entry=gmail&source=g> > and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address > Century House, 12 Victoria Street, Alderney, GY9 3UF > <https://maps.google.com/?q=12+Victoria+Street,+Alderney,+GY9+3UF&entry=gmail&source=g> > operate the Sky Betting and Gaming and Oddschecker services. >
