I agree that users should be blocked if they don't install livy, livy is easy for spark job submitting, but we also need some other approach like submitting in shell way or using APIs of yarn.
However, there're more spikes to adapt to different spark clusters, like yarn or mesos, which is not the problem we want to solve in data quality domain. In my opinion, livy focuses on connecting to spark cluster and works well, it could be the default solution. We can also provide a simple way to submit spark jobs, for the basic functions only, to support the specific environment, that'll be enough. If we want to get job state, retry the job if failed, it will cost more efforts. Thanks, Lionel On Mon, Apr 30, 2018 at 10:47 PM, William Guo <[email protected]> wrote: > hi all, > > In some cluster environment, administrator doesn't install livy in their > cluster. > > How can we support these users, should we support it by launching spark > jobs by CLI? > > > What are your comments for this issue? > > > Thanks, > William >
