Hi Brian, Did you have created snapshots on cluster while you were upgrading to new hbase version ? Based on your logs there are some snapshots operations running on your cluster. Can you dig your logs to find what is triggering SnapshotManifestLoader there should some more info just above this lines you have pasted here.
Regards Samir On Sun, Dec 27, 2015 at 6:34 PM, Brian Jeltema <[email protected]> wrote: > I recently upgraded to Hadoop 2.7.1 and HBase 1.1.2. > > Since the upgrade, the HBase master logs have been filling and rolling > over about > every 5 minutes, filled with variations of the following (modified to > obscure internal details): > > 2015-12-27 12:24:52,481 DEBUG [SnapshotManifestLoader-pool878551-t3] > snapshot.SnapshotManifestV1: Creating references for hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8] > snapshot.SnapshotManifestV1: Storing region-info for snapshot. > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8] > snapshot.SnapshotManifestV1: Creating references for hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6] > snapshot.SnapshotManifestV1: Storing region-info for snapshot. > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6] > snapshot.SnapshotManifestV1: Creating references for hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5] > snapshot.SnapshotManifestV1: Adding snapshot references for > [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>] > hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5] > snapshot.SnapshotManifestV1: Adding reference for file (1/1): > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246> > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7] > snapshot.SnapshotManifestV1: Storing region-info for snapshot. > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7] > snapshot.SnapshotManifestV1: Creating references for hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4] > snapshot.SnapshotManifestV1: Adding snapshot references for > [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>, > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>, > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4] > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]> > hfiles > 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4] > snapshot.SnapshotManifestV1: Adding reference for file (1/3): > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472> > 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4] > snapshot.SnapshotManifestV1: Adding reference for file (2/3): > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d> > 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4] > snapshot.SnapshotManifestV1: Adding reference for file (3/3): > hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4> > 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t3] > snapshot.SnapshotManifestV1: Adding snapshot references for > [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3 > <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3> > > This is also causing my HDFS audit logs to grow at a rate of about > 1.5MB/s. Has anyone observed > this behavior or know a fix? > > Thanks > Brian
