> Hi Brian, > Did you have created snapshots on cluster while you were upgrading to new > hbase version ?
There were existing snapshots when I upgraded. It looks like a bunch of new snapshot also appeared as part of the process, with names like ‘foo-ru-yyyymmdd’ > Based on your logs there are some snapshots operations > running on your cluster. Can you dig your logs to find what is > triggering SnapshotManifestLoader there should some more info just above > this lines you have pasted here. I didn’t see anything in the logs significantly different than what I included in the previous post. Brian > > Regards > Samir > > On Sun, Dec 27, 2015 at 6:34 PM, Brian Jeltema <[email protected]> wrote: > >> I recently upgraded to Hadoop 2.7.1 and HBase 1.1.2. >> >> Since the upgrade, the HBase master logs have been filling and rolling >> over about >> every 5 minutes, filled with variations of the following (modified to >> obscure internal details): >> >> 2015-12-27 12:24:52,481 DEBUG [SnapshotManifestLoader-pool878551-t3] >> snapshot.SnapshotManifestV1: Creating references for hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8] >> snapshot.SnapshotManifestV1: Storing region-info for snapshot. >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t8] >> snapshot.SnapshotManifestV1: Creating references for hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6] >> snapshot.SnapshotManifestV1: Storing region-info for snapshot. >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t6] >> snapshot.SnapshotManifestV1: Creating references for hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5] >> snapshot.SnapshotManifestV1: Adding snapshot references for >> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246>] >> hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t5] >> snapshot.SnapshotManifestV1: Adding reference for file (1/1): >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/efeae4f976778ed5b87fd53fd82f9f8f/p/b73e5e35e6cd4699bc48b67427173246> >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7] >> snapshot.SnapshotManifestV1: Storing region-info for snapshot. >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t7] >> snapshot.SnapshotManifestV1: Creating references for hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4] >> snapshot.SnapshotManifestV1: Adding snapshot references for >> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472>, >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d>, >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4] >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4]> >> hfiles >> 2015-12-27 12:24:52,482 DEBUG [SnapshotManifestLoader-pool878551-t4] >> snapshot.SnapshotManifestV1: Adding reference for file (1/3): >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/54f2f56e456d451a9a3601bc58540472> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4] >> snapshot.SnapshotManifestV1: Adding reference for file (2/3): >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/a75caa4c2da2485db941769be491792d> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t4] >> snapshot.SnapshotManifestV1: Adding reference for file (3/3): >> hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f061f0ad27a1bb55b0f00be2c78779fb/p/ead9c7f82c554c93a8c7b3361e7550a4> >> 2015-12-27 12:24:52,483 DEBUG [SnapshotManifestLoader-pool878551-t3] >> snapshot.SnapshotManifestV1: Adding snapshot references for >> [hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3 >> <hdfs://mycluster/apps/hbase/data/.hbase-snapshot/Foo-ru-20151223/f19b1b94666c38d6c3585b3> >> >> This is also causing my HDFS audit logs to grow at a rate of about >> 1.5MB/s. Has anyone observed >> this behavior or know a fix? >> >> Thanks >> Brian
