If you are using PigServer and submitting programmatically via same jvm, it should automatically reuse the application if the requested AM resources are same.
https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java#L242-L245 On Fri, Jan 18, 2019 at 12:20 PM Diego Pereira <diego.ns.pere...@gmail.com> wrote: > Hi! > > We are developing an application that is looking for new files on a folder, > running a few Pig Scripts to prepare those files and, finally, loading them > into our database. > > The problem is that, for small files, the time that Pig / Tez / Yarn take > to create a new application master and spawn new containers is way longer > than the time it takes processing. > > Since Tez Sessions already allows a single Pig script to run multiple DAGs > against the same application master, is there a way to reuse that > application master and it´s containers for multiple Pig Scripts submissions > ? > > Regards, > > Diego >
