Hi William, Thanks for the info. I understand the potential firewall issues, we already addressed this for applications we deploy around the edge of our cluster.
Currently the main reason for not using Livy is because it’s not currently available in our cluster. I was just curious how easy it would be to make it optional? Thanks Darrell. From: William Guo <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Monday, 30 April 2018 at 12:35 To: "[email protected]" <[email protected]> Subject: Re: Disable Livy option? Hi Taylor, The job submission is triggered by our job scheduler(quartz), by quartz, we periodically call livy service to submit spark jobs. We have thought submit jobs directly, but in many cases, that will cause firewall/security/auth issues. We leverage livy to submit jobs from http layer, because most cluster are accessible from http layer. What is your issue for using livy? Thanks, William On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor <[email protected]<mailto:[email protected]>> wrote: Hi, I was wondering if it’s feasible to disable the Livy option and have the ability to submit Spark jobs directly? Having had a brief look through the code it seems like all the SparkConf and SparkContext is handled as normal, but couldn’t quite figure out where the job submission gets routed to Livy. Thanks -- Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. The SKY trademarks are owned by Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No. 1110) with registered address Office 1, 1 The Crusher, Braye Harbour, Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with registered address 2 Wellington Place, Leeds, LS1 4AP and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address Century House, 12 Victoria Street, Alderney, GY9 3UF operate the Sky Betting and Gaming and Oddschecker services.
