** Description changed: [Impact] - On our tentacle packages, several manager modules fail to load due to a - missing dependency (cherrypy_mgr). For more details, see: - https://github.com/canonical/microceph/issues/874 + On tentacle/edge rev 1956 (20.2.4+snap2b1a2cf17b), the prometheus and + dashboard mgr modules fail to import: + + ModuleNotFoundError: No module named 'cherrypy_mgr' + + cherrypy_mgr.py is not shipped in the snap. The only top-level files in + share/ceph/mgr/ are: + + $ unsquashfs -l microceph_1956.snap | grep -E 'share/ceph/mgr/[^/]+\.py$' + squashfs-root/share/ceph/mgr/mgr_module.py + squashfs-root/share/ceph/mgr/mgr_util.py + squashfs-root/share/ceph/mgr/object_format.py + + latest/edge rev 1967 has the same content. + + The Ceph exporter never starts: nothing listens on :9283, and ceph mgr + services returns {}. No Ceph metrics or alerts reach COS. + + The dashboard cannot start, now for this reason rather than the missing + smb module in Tentacle 20.2.1: Dashboard fails to start due to missing + mgr/smb module (ModuleNotFoundError: No module named 'smb') #811. + + With the microceph charm (rev 327, the same revision on tentacle/stable, candidate and edge), a fresh deployment never forms a cluster. The leader bootstraps, then its config-changed hook fails on every retry, and the other units stay Leader not ready. There are two failures in sequence: + ceph mgr module enable prometheus returns ENOENT: module 'prometheus' reports that it cannot run on the active manager daemon: No module named 'cherrypy_mgr'. + After ceph mgr module enable prometheus --force: ceph config set mgr mgr/prometheus/exclude_perf_counters False returns EINVAL: unrecognized config option. The option is only registered once the module loads. + + The charm's cos-agent refresh runs both calls on config-changed whether + or not a cos-agent relation exists + (lib/charms/ceph_mon/v0/ceph_cos_agent.py _on_refresh, src/microceph.py + cos_agent_mgr_config_set_cb). No charm config value avoids them, so + there is no operator-side workaround. + + The first error in the log, Unable to initialize cluster: Database is + online, is not the cause. The first bootstrap succeeded, and every hook + retry repeats it harmlessly. + + Suggested fix: + + Snap: ship usr/share/ceph/mgr/cherrypy_mgr.py (fix the ceph-mgr install list in the PPA packaging, or stage the file explicitly). + Charm (secondary, for charm-microceph): do not fail the hook when the prometheus module cannot run. Catch the error in the cos-agent refresh and set a status instead, so a broken optional mgr module cannot block cluster formation. [Test Case] + + Deploy the microceph charm with snap-channel=tentacle/edge: + + juju deploy microceph --channel tentacle/edge --config snap- + channel=tentacle/edge -n 3 + + Watch the leader: bootstrap succeeds, then config-changed fails with failure 1 above. + On the leader, run sudo microceph.ceph mgr module enable prometheus --force, then juju resolve microceph/<leader>. The hook now fails with failure 2. + + Snap only, without the charm. We saw the same module failure in a + single-node VM on rev 1956 via tentacle/candidate. The enable command is + the same one the charm runs: + + sudo snap install microceph --channel tentacle/edge + sudo microceph cluster bootstrap + sudo microceph.ceph mgr module enable prometheus + # Error ENOENT: module 'prometheus' reports that it cannot run on the active manager daemon: No module named 'cherrypy_mg The Ceph team has validated that the inclusion of this new dependency fixes the associated problems by running the functional test suite for Microceph and Ceph Rocks. - [Regression Potential] + [ Where problems can occur ] + + Shipping the proper python modules in their correct locations is unlikely to + create any issues. If an import was being skipped previously, it will now + function correctly, and may add new functionality. In order to mitigate the regression potential, the results of the aforementioned tests can be seen here: https://github.com/canonical/ceph-containers/pull/104 + + [ Other info ] + + For more details, see: https://github.com/canonical/microceph/issues/874
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2169663 Title: ceph dashboard fails to start due to missing cherrypy_mgr dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2169663/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
