All the ci-hadoop users, I apologize for starving the resources by my misconfiguration. Aborted all the hadoop-multibranch jobs and modified the configuration as follows:
- Disabled the periodic scanning. - Set "Discover pull requests from origin/forks" to "The current pull request revision" Thank you Sean for the report and thanks Zoltan for your advice. Best regards, Akira On Sat, Aug 1, 2020 at 1:07 AM Zoltan Haindrich <k...@rxd.hu> wrote: > Hey All! > > For Hive we also use the multi-branch pipeline plugin [1] - and in the > last couple months I've seen some issues with it. > > Probably most importantly w.r.t to the current issue: the > hadoop-multibranch job right now is configured "merge the PR with the > master" which is a handy feature - but it > has a backstab: in case branch indexing happens - it will re-discover all > PRs as well in case the master has new commits! > As a quick fix to this I would recommend to disable the periodic scanning > (which is set to 30 minutes right now). > > Second: it is hard to limit the number of active jobs of a multi-branch > pipeline job ; for Hive we use resource locking [2] to ensure that at most > 5 runs are executing at > any given time. > > I've cc-ed the hadoop-common mailing list. > > cheers, > Zoltan > > [1] http://ci.hive.apache.org/job/hive-precommit/ > [2] https://plugins.jenkins.io/lockable-resources/ > > > On 7/31/20 5:35 PM, Sean Busbey wrote: > > would the maintainers of > > > > https://ci-hadoop.apache.org/job/hadoop-multibranch/ > > > > please configure it to limit the number of concurrent executions and > clean up the current backlog? y'all have saturate all workers for the last > 10 hours and it is not apparent if the running jobs can actually complete. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
