Jacques, please see the comment below. Correct me if I am wrong: > On May 6, 2015, at 4:23 PM, Jacques Nadeau <jacq...@apache.org> wrote: > > We currently don't start fragments until we have enough input for them. > The fragment manager is responsible for watching how much incoming data we > have and then once there is enough that it is possible to start working, > the fragment is executed. When it is started, I believe it is added to the > running fragments. Imagine an n-way merge. Until we get at least one > batch from each of the n-way inputs, we don't start the fragment. > > Now, this made more sense when we didn't have the fast schema behavior. > Now that we do, everything starts basically immediately anyway (except when
By “everything starts basically immediately”, you mean the fragment is executed as soon as it receives the first batch. It happens so that with fast schema behavior, the first batch arrives quicker than before. If this is the case, isn’t it safe to start executing a fragment as soon as a Drillbit receives a fragment from the Foreman? The assumption that the fragment is running will simplify things, e.g. cancellation. > schema is indeterminate, as in the case of CONVERT_FROM(xx, 'JSON'). > > On Wed, May 6, 2015 at 3:50 PM, Abdel Hakim Deneche <adene...@maprtech.com> > wrote: > >> Hi all, >> >> In WorkManager there are two methods that can be used to start a fragment >> executor addFragmentRunner(FragmentExecutor) and >> addFragmentPendingRemote(FragmentManager) >> >> both methods will run the fragment executor but the second one will make >> sure the fragment executor is added to WorkManager.runningFragments. >> >> Why not add all fragment that are running in a drillbit to >> "runningFragments" and limit ourselves to the fragments started through >> addFragmentRunner() ? >> >> Thanks! >> >> -- >> >> Abdelhakim Deneche >> >> Software Engineer >> >> <http://www.mapr.com/> >> >> >> Now Available - Free Hadoop On-Demand Training >> < >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>> >>
