[ https://issues.apache.org/jira/browse/IMPALA-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751703#comment-16751703 ]
Michael Brown commented on IMPALA-8115: --------------------------------------- I noticed this because I posted https://gerrit.cloudera.org/#/c/12272/ at 2:42 PT and at 3:45 PT there is still no update that the pre-review job completed. https://jenkins.impala.io/job/gerrit-code-review-checks/1879/console shows a slow timeline: {noformat} Triggered by Gerrit: http://gerrit.cloudera.org:8080/12272 Running in Durability level: MAX_SURVIVABILITY [Pipeline] timestamps [Pipeline] { [Pipeline] ansiColor [Pipeline] { [Pipeline] timeout 22:42:09 Timeout set to expire in 10 hr [Pipeline] { [Pipeline] parallel [Pipeline] { (Branch: Tidy) [Pipeline] { (Branch: BuildOnly) [Pipeline] { (Branch: Python26Compatibility) [Pipeline] { (Branch: Rat) [Pipeline] build (Building clang-tidy-ub1604) 22:42:09 Scheduling project: clang-tidy-ub1604 [Pipeline] build (Building ubuntu-16.04-build-only) 22:42:09 Scheduling project: ubuntu-16.04-build-only [Pipeline] build (Building python26-incompatibility-check) 22:42:09 Scheduling project: python26-incompatibility-check [Pipeline] build (Building rat-check-ub1604) 22:42:09 Scheduling project: rat-check-ub1604 22:44:22 Starting building: python26-incompatibility-check #1548 [Pipeline] } 23:35:15 Starting building: ubuntu-16.04-build-only #5027 23:36:34 Starting building: rat-check-ub1604 #5404 [Pipeline] } 23:37:14 Starting building: clang-tidy-ub1604 #4922 {noformat} It's taken nearly an hour to start rat-check-ub1604 #5404 and clang-tidy-ub1604 #4922. My 15 minutes math doesn't account for all of this. I'm not yet sure where else we could accelerate/improve the process. > some jenkins workers slow to do real work due to dpkg lock conflicts > -------------------------------------------------------------------- > > Key: IMPALA-8115 > URL: https://issues.apache.org/jira/browse/IMPALA-8115 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Reporter: Michael Brown > Priority: Major > > A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start > doing real work. I noticed that it was retrying {{apt-get update}}: > {noformat} > ++ sudo apt-get --yes install openjdk-8-jdk > E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily > unavailable) > E: Unable to lock the administration directory (/var/lib/dpkg/), is another > process using it? > ++ date > Thu Jan 24 23:37:33 UTC 2019 > ++ sudo apt-get update > ++ sleep 10 > ++ sudo apt-get --yes install openjdk-8-jdk > [etc] > {noformat} > I ssh'd into a host and saw that, yes, something else was holding onto the > dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was > the culprit) > {noformat} > root 1750 0.0 0.0 4508 1664 ? Ss 23:21 0:00 /bin/sh > /usr/lib/apt/apt.systemd.daily > root 1804 12.3 0.1 141076 80452 ? S 23:22 1:24 \_ > /usr/bin/python3 /usr/bin/unattended-upgrade > root 3263 0.0 0.1 140960 72896 ? S 23:23 0:00 \_ > /usr/bin/python3 /usr/bin/unattended-upgrade > root 11459 0.6 0.0 45920 25184 pts/1 Ss+ 23:24 0:03 \_ > /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure > /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb > /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb > /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb > /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb > /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > > /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb > /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb > /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb > /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb > /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb > /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb > /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb > /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb > /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb > /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb > /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb > /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb > /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb > /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb > /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb > /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb > /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb > /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb > /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb > /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb > /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb > /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb > /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb > /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb > /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb > /var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb > /var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb > /var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb > /var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb > /var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb > /var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb > /var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb > /var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb > /var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb > /var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb > /var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb > /var/cache/apt/archives/apport_2.20.1-0ubuntu2.18_all.deb > /var/cache/apt/archives/dns-root-data_2018013001~16.04.1_all.deb > /var/cache/apt/archives/dnsmasq-base_2.75-1ubuntu0.16.04.5_amd64.deb > /var/cache/apt/archives/git-man_1%3a2.7.4-0ubuntu1.6_all.deb > /var/cache/apt/archives/git_1%3a2.7.4-0ubuntu1.6_amd64.deb > /var/cache/apt/archives/libmspack0_0.5-1ubuntu0.16.04.3_amd64.deb > /var/cache/apt/archives/libpolkit-agent-1-0_0.105-14.1ubuntu0.4_amd64.deb > /var/cache/apt/archives/libpolkit-backend-1-0_0.105-14.1ubuntu0.4_amd64.deb > /var/cache/apt/archives/linux-image-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb > /var/cache/apt/archives/linux-aws_4.4.0.1074.76_amd64.deb > /var/cache/apt/archives/linux-image-aws_4.4.0.1074.76_amd64.deb > /var/cache/apt/archives/linux-aws-headers-4.4.0-1074_4.4.0-1074.84_all.deb > /var/cache/apt/archives/linux-headers-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb > /var/cache/apt/archives/linux-headers-aws_4.4.0.1074.76_amd64.deb > /var/cache/apt/archives/policykit-1_0.105-14.1ubuntu0.4_amd64.deb > /var/cache/apt/archives/python3-cryptography_1.2.3-1ubuntu0.2_amd64.deb > /var/cache/apt/archives/python3-jwt_1.3.0-1ubuntu0.1_all.deb > /var/cache/apt/archives/python3-requests_2.9.1-3ubuntu0.1_all.deb > /var/cache/apt/archives/unattended-upgrades_0.90ubuntu0.10_all.deb > root 12555 0.0 0.0 4500 740 pts/1 S+ 23:24 0:00 > \_ /bin/sh /var/lib/dpkg/info/unattended-upgrades.prerm upgrade > 0.90ubuntu0.10 > root 12556 0.0 0.0 4500 1736 pts/1 S+ 23:24 0:00 > \_ /bin/sh /usr/sbin/invoke-rc.d unattended-upgrades stop > root 12594 0.0 0.0 24884 1388 pts/1 S+ 23:24 0:00 > \_ systemctl stop unattended-upgrades.service > {noformat} > After about 15 minutes, the retry to install openjdk-8-jdk succeeded. > I'm not sure how to make this better except rebuild AMIs with updated > packaging. Over time, the system's automatic apt-get update / upgrades are > going to take longer and longer and our own stuff is going to block for > longer and longer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org