Re: hbase CMS gc pause serious program

[Anoop John]

96 GB of heap size? !!

What kind of work load is this?

-Anoop-

On Fri, Mar 10, 2017 at 10:54 AM, gehaijiang  wrote:
> CMS GC  program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800:
> 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs]
> 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07
> sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800:
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs]
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94
> sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep:
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
>
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset:
> 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800:
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs]
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63
> sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark:
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean:
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928:
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800:
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs]
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79
> sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648:
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652:
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew:
> 2011637K->340736K(3067136K), 2.0773980 secs]
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09
> sys=0.38, real=2.07 secs]
>
>
> regionserver JVM config:
>
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
>
> attachment：hdfs-site.xml

RE: Pattern for Bulk Loading to Remote HBase Cluster

[ashish singhi]

If I understand your question you are asking, how to completebulkload files 
which are on cluster1 to cluster2 without copying them to cluster2. Answer is 
with the existing code it's not possible.

Bq. How do I choose hdfs://storefile-outputdir in a way that does not perform 
an extra copy operation when completebulkload is invoked, without assuming 
knowledge of HBase server implementation details?

You can configure the output dir to remote cluster active Namenode IP, so that 
the output of importtsv is written there and then use completebulkload in the 
remote cluster specifying this output dir path as it argument.

Bq. In essence, how does my client application know that it should write to
hdfs://cluster2 even though the application is running in a context where 
fs.defaultFs is hdfs://cluster1?

If you are talking about importtsv then it read the URI from the path and 
connect to that respective NN. If you use the nameservices name in the path 
instead of active NN IP then you may have to write your own code something 
similar to importtsv where you can construct remote cluster configuration 
object and use it to write output there. You can refer HBASE-13153 for an idea 
to understand it much better.

-Original Message-
From: Ben Roling [mailto:ben.rol...@gmail.com] 
Sent: 09 March 2017 19:53
To: user@hbase.apache.org
Subject: Re: Pattern for Bulk Loading to Remote HBase Cluster

I'm not sure you understand my question.  Or perhaps I just don't quite 
understand yours?

I'm not using importtsv.  If I was, and I was using the form that prepares 
StoreFiles for completebulkload, then my question would be, how do I 
(generically as an application acting as an HBase client, and using importtsv 
to load data) choose the path to which I write the StoreFiles?

The following is an example of importtsv from the documentation:

bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dimporttsv.columns=a,b,c
-Dimporttsv.bulk.output=hdfs://storefile-outputdir  


How do I choose hdfs://storefile-outputdir in a way that does not perform an 
extra copy operation when completebulkload is invoked, without assuming 
knowledge of HBase server implementation details?

In essence, how does my client application know that it should write to
hdfs://cluster2 even though the application is running in a context where 
fs.defaultFs is hdfs://cluster1?

How does the HBase installation share this information with client applications?

I know I can just go dig into the hdfs-site.xml on a RegionServer and figure 
this out (such as by looking at "hbase.rootdir" there), but my question is how 
to do it from the perspective of a generic HBase client application?

On Wed, Mar 8, 2017 at 11:13 PM ashish singhi 
wrote:

> Hi,
>
> Did you try giving the importtsv output path to remote HDFS ?
>
> Regards,
> Ashish
>
> -Original Message-
> From: Ben Roling [mailto:ben.rol...@gmail.com]
> Sent: 09 March 2017 03:22
> To: user@hbase.apache.org
> Subject: Pattern for Bulk Loading to Remote HBase Cluster
>
> My organization is looking at making some changes that would introduce 
> HBase bulk loads that write into a remote cluster.  Today our bulk 
> loads write to a local HBase.  By local, I mean the home directory of 
> the user preparing and executing the bulk load is on the same HDFS 
> filesystem as the HBase cluster.  In the remote cluster case, the 
> HBase being loaded to will be on a different HDFS filesystem.
>
> The thing I am wondering about is what the best pattern is for 
> determining the location to write HFiles to from the job preparing the bulk 
> load.
> Typical examples write the HFiles somewhere in the user's home directory.
> When HBase is local, that works perfectly well.  With remote HBase, it 
> can work, but results in writing the files twice: once from the 
> preparation job and a second time by the RegionServer when it reacts 
> to the bulk load by copying the HFiles into the filesystem it is running on.
>
> Ideally the preparation job would have some mechanism to know where to 
> write the files such that they are initially written on the same 
> filesystem as HBase itself.  This way the bulk load can simply move 
> them into the HBase storage directory like happens when bulk loading to a 
> local cluster.
>
> I've considered a pattern where the bulk load preparation job reads 
> the hbase.rootdir property and pulls the filesystem off of that.  
> Then, it sticks the output in some directory (e.g. /tmp) on that same 
> filesystem.
> I'm inclined to think that hbase.rootdir should only be considered a 
> server-side property and as such I shouldn't expect it to be present 
> in client configuration.  Under that assumption, this isn't really a 
> workable strategy.
>
> It feels like HBase should have a mechanism for sharing a staging 
> directory with clients doing bulk loads.  Doing some searching, I ran 
> across "hbase.bulkload.staging.dir", but my impression is that its 
> 

hbase CMS gc pause serious program

[gehaijiang]

CMS GC  program: 
2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 
4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 
79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, 
real=2.09 secs]2017-03-10T10:15:29.524+0800: 4555920.160: 
[GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 
3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 
2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 
secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 
1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 
secs]2017-03-10T10:15:32.911+0800: 4555923.547: 
[CMS-concurrent-reset-start]2017-03-10T10:15:33.126+0800: 4555923.762: 
[CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 
secs]
2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 
4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 
80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, 
real=2.41 secs]
2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 
77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 
sys=0.00, real=0.01 secs]
2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 
secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 
0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
2017-03-10T10:15:36.291+0800: 4555926.928: 
[CMS-concurrent-abortable-preclean-start]
2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 
4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 
81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, 
real=2.62 secs]
2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 
2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs]
2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 
K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 
4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 
80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, 
real=2.07 secs]
regionserver JVM config:export 
HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m 
-XX:MaxPermSize=256m -Xms96G -Xmx96G"
export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 
-XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
-XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps -verbose:gc
-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 
-XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
attachment：    hdfs-site.xml 

Re: Pattern for Bulk Loading to Remote HBase Cluster

[Ben Roling]

I'm not sure you understand my question.  Or perhaps I just don't quite
understand yours?

I'm not using importtsv.  If I was, and I was using the form that prepares
StoreFiles for completebulkload, then my question would be, how do I
(generically as an application acting as an HBase client, and using
importtsv to load data) choose the path to which I write the StoreFiles?

The following is an example of importtsv from the documentation:

bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dimporttsv.columns=a,b,c
-Dimporttsv.bulk.output=hdfs://storefile-outputdir 


How do I choose hdfs://storefile-outputdir in a way that does not perform
an extra copy operation when completebulkload is invoked, without assuming
knowledge of HBase server implementation details?

In essence, how does my client application know that it should write to
hdfs://cluster2 even though the application is running in a context where
fs.defaultFs is hdfs://cluster1?

How does the HBase installation share this information with client
applications?

I know I can just go dig into the hdfs-site.xml on a RegionServer and
figure this out (such as by looking at "hbase.rootdir" there), but my
question is how to do it from the perspective of a generic HBase client
application?

On Wed, Mar 8, 2017 at 11:13 PM ashish singhi 
wrote:

> Hi,
>
> Did you try giving the importtsv output path to remote HDFS ?
>
> Regards,
> Ashish
>
> -Original Message-
> From: Ben Roling [mailto:ben.rol...@gmail.com]
> Sent: 09 March 2017 03:22
> To: user@hbase.apache.org
> Subject: Pattern for Bulk Loading to Remote HBase Cluster
>
> My organization is looking at making some changes that would introduce
> HBase bulk loads that write into a remote cluster.  Today our bulk loads
> write to a local HBase.  By local, I mean the home directory of the user
> preparing and executing the bulk load is on the same HDFS filesystem as the
> HBase cluster.  In the remote cluster case, the HBase being loaded to will
> be on a different HDFS filesystem.
>
> The thing I am wondering about is what the best pattern is for determining
> the location to write HFiles to from the job preparing the bulk load.
> Typical examples write the HFiles somewhere in the user's home directory.
> When HBase is local, that works perfectly well.  With remote HBase, it can
> work, but results in writing the files twice: once from the preparation job
> and a second time by the RegionServer when it reacts to the bulk load by
> copying the HFiles into the filesystem it is running on.
>
> Ideally the preparation job would have some mechanism to know where to
> write the files such that they are initially written on the same filesystem
> as HBase itself.  This way the bulk load can simply move them into the
> HBase storage directory like happens when bulk loading to a local cluster.
>
> I've considered a pattern where the bulk load preparation job reads the
> hbase.rootdir property and pulls the filesystem off of that.  Then, it
> sticks the output in some directory (e.g. /tmp) on that same filesystem.
> I'm inclined to think that hbase.rootdir should only be considered a
> server-side property and as such I shouldn't expect it to be present in
> client configuration.  Under that assumption, this isn't really a workable
> strategy.
>
> It feels like HBase should have a mechanism for sharing a staging
> directory with clients doing bulk loads.  Doing some searching, I ran
> across "hbase.bulkload.staging.dir", but my impression is that its intent
> does not exactly align with mine.  I've read about it here [1].  It seems
> the idea is that users prepare HFiles in their own directory, then
> SecureBulkLoad moves them to "hbase.bulkload.staging.dir".  A move like
> that isn't really a move when dealing with a remote HBase cluster.  Instead
> it is a copy.  A question would be why doesn't the job just write the files
> to "hbase.bulkload.staging.dir" initially and skip the extra step of moving
> them?
>
> I've been inclined to invent my own application-specific Hadoop property
> to use to communicate an HBase-local staging directory with my bulk load
> preparation jobs.  I don't feel perfectly good about that idea though.  I'm
> curious to hear experiences or opinions from others.  Should I have my bulk
> load prep jobs look at "hbase.rootdir" or "hbase.bulkload.staging.dir" and
> make sure those get propagated to client configuration?  Is there some
> other mechanism that already exists for clients to discover an HBase-local
> directory to write the files?
>
> [1] http://hbase.apache.org/book.html#hbase.secure.bulkload
>

Re: Custom Retention of data based on Rowkey

[Anoop John]

>From 0.98.9 version onward, there a per cell TTL feature available.
(See HBASE-10560)..  TTL can be controlled even per cell level.

It might eat up more space as the TTL is stored with every cell.  What
u want is same TTL for one group and another for another.  The major
concern is u need "dynamically updatable" TTL..  Per cell TTL can not
do this.

So u might need a custom compaction policy plugged in.. Might be like
u need to decorate the HBase compaction policy using CPs.  Also to
remember that u will need some CP work in read (Scan/Get) to make sure
TTL expired data is not retrieved back (Compaction might be not
happened yet but data is TTL expired. Actually at HBase level u dont
give any TTL)

-Anoop-

On Thu, Mar 9, 2017 at 4:34 PM, Gaurav Agarwal  wrote:
> Hi,
>
> Looking deeper, I found that RegionObserver interface provides general hooks 
> to intercept the pre-compaction scanner. That should suffice for our purpose!
>
> In any case, if there are any suggestions/guidelines, it will be much 
> appreciated.
>
>
>
> From: Gaurav Agarwal 
> Date: Thursday, 9 March 2017 at 2:08 PM
> To: 
> Cc: Kshitij Gupta , Mukul Gupta 
> Subject: Custom Retention of data based on Rowkey
>
>
>
> Hi All,
>
>
>
> We have an application that stores information on multiple 
> users/customers/tenants in a common table. Each tenant has a unique id which 
> we encode in the row key of the records that are stored in the table.
>
>
>
> We want to apply custom (and dynamically updatable) data retention policies 
> for each tenant.  What would be a reasonable way to achieve that?
>
>
>
> Searching through forums, I came across this link that suggests to either 
> write an external process to retrieve and delete cells based on the retention 
> policy or write a custom compaction policy:
>
> https://community.hortonworks.com/questions/14883/best-way-to-achieve-custom-retention-of-some-rows.html
>
>
>
> We felt that writing an external scanner for managing retention would be 
> simpler but very inefficient as it would require getting the entire data set 
> out of the hbase server and then issuing delete calls back to it.
>
>
>
> Does any one know if there has been any recent progress on this aspect of 
> data retention in hbase?
>
>
>
> Additionally, if I go the route of writing my own custom compaction policy, 
> what would be the best place to start? Maybe I could copy/extend the 
> “default” hbase compaction policy and enhance it to look at rowkey inside 
> every Cell to make a call if the cell needs to be deleted?
>
>
>
> --
>
> cheers,
>
> gaurav
>
>
>
>
>

Re: Custom Retention of data based on Rowkey

[Gaurav Agarwal]

Hi,

Looking deeper, I found that RegionObserver interface provides general hooks to 
intercept the pre-compaction scanner. That should suffice for our purpose!

In any case, if there are any suggestions/guidelines, it will be much 
appreciated.

 

From: Gaurav Agarwal 
Date: Thursday, 9 March 2017 at 2:08 PM
To: 
Cc: Kshitij Gupta , Mukul Gupta 
Subject: Custom Retention of data based on Rowkey

 

Hi All,

 

We have an application that stores information on multiple 
users/customers/tenants in a common table. Each tenant has a unique id which we 
encode in the row key of the records that are stored in the table.

 

We want to apply custom (and dynamically updatable) data retention policies for 
each tenant.  What would be a reasonable way to achieve that?

 

Searching through forums, I came across this link that suggests to either write 
an external process to retrieve and delete cells based on the retention policy 
or write a custom compaction policy:

https://community.hortonworks.com/questions/14883/best-way-to-achieve-custom-retention-of-some-rows.html

 

We felt that writing an external scanner for managing retention would be 
simpler but very inefficient as it would require getting the entire data set 
out of the hbase server and then issuing delete calls back to it.

 

Does any one know if there has been any recent progress on this aspect of data 
retention in hbase? 

 

Additionally, if I go the route of writing my own custom compaction policy, 
what would be the best place to start? Maybe I could copy/extend the “default” 
hbase compaction policy and enhance it to look at rowkey inside every Cell to 
make a call if the cell needs to be deleted?

 

--

cheers,

gaurav

 

 

Custom Retention of data based on Rowkey

[Gaurav Agarwal]

Hi All,

 

We have an application that stores information on multiple 
users/customers/tenants in a common table. Each tenant has a unique id which we 
encode in the row key of the records that are stored in the table.

 

We want to apply custom (and dynamically updatable) data retention policies for 
each tenant.  What would be a reasonable way to achieve that?

 

Searching through forums, I came across this link that suggests to either write 
an external process to retrieve and delete cells based on the retention policy 
or write a custom compaction policy:

https://community.hortonworks.com/questions/14883/best-way-to-achieve-custom-retention-of-some-rows.html

 

We felt that writing an external scanner for managing retention would be 
simpler but very inefficient as it would require getting the entire data set 
out of the hbase server and then issuing delete calls back to it.

 

Does any one know if there has been any recent progress on this aspect of data 
retention in hbase? 

 

Additionally, if I go the route of writing my own custom compaction policy, 
what would be the best place to start? Maybe I could copy/extend the “default” 
hbase compaction policy and enhance it to look at rowkey inside every Cell to 
make a call if the cell needs to be deleted?

 

--

cheers,

gaurav