[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-23 Thread Yu Li (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346220#comment-15346220 ] Yu Li commented on HBASE-15900: --- We were running 0.98.12 with hadoop 2.6.2 for a long time and now running

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-23 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345872#comment-15345872 ] Heng Chen commented on HBASE-15900: --- After test 2.6.0 with one week, and 2.7.2 with one more week.

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-07 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319877#comment-15319877 ] Heng Chen commented on HBASE-15900: --- I have upgraded my hbase cluster hadoop version to be 2.6.0, it

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-07 Thread Mikhail Antonov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319797#comment-15319797 ] Mikhail Antonov commented on HBASE-15900: - [~chenheng] curious if you had a chance to test your

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-07 Thread Mikhail Antonov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319795#comment-15319795 ] Mikhail Antonov commented on HBASE-15900: - For 1.3 I think 2.7.2 should be the default version,

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-07 Thread Andrew Purtell (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319790#comment-15319790 ] Andrew Purtell commented on HBASE-15900: We also saw long stuck flushes or compactions, and

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-03 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315121#comment-15315121 ] Nick Dimiduk commented on HBASE-15900: -- bq. any plan to upgrade branch-1.1 hadoop version to 2.6.0

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309944#comment-15309944 ] Sergey Soldatov commented on HBASE-15900: - Well, there are a lot of possibilities to block

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309926#comment-15309926 ] Heng Chen commented on HBASE-15900: --- {code} do you happen to have any steps to reliably (or, well, with

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Mikhail Antonov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309910#comment-15309910 ] Mikhail Antonov commented on HBASE-15900: - Hm. I _think_ I've seen stuck memstores on 2.7+

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Mikhail Antonov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309900#comment-15309900 ] Mikhail Antonov commented on HBASE-15900: - As in the support / compatibility matrix we say that

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309393#comment-15309393 ] Heng Chen commented on HBASE-15900: --- branch-1.3 seems compile with 2.5.x too. [~mantonov] > RS stuck

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-06-01 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309386#comment-15309386 ] Heng Chen commented on HBASE-15900: --- Yeah , it looks like this bug block the longCompaction thread. I

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-31 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309259#comment-15309259 ] Heng Chen commented on HBASE-15900: --- Maybe. HDFS version is 2.5.1. Let me try to upgrade my hdfs

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-31 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308776#comment-15308776 ] Sergey Soldatov commented on HBASE-15900: - [~chenheng] Which HDFS version you are using? It may

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-31 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307450#comment-15307450 ] Heng Chen commented on HBASE-15900: --- We could see in 0d32a6bab354e6cc170cd59a2d485797.rs.log, the

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-31 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307438#comment-15307438 ] Heng Chen commented on HBASE-15900: --- I open DEBUG Level log in RS, and found the problem which i

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-30 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306269#comment-15306269 ] Heng Chen commented on HBASE-15900: --- 1.8.0_20 I can find threads which own the read lock, but can't

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306264#comment-15306264 ] Sergey Soldatov commented on HBASE-15900: - [~chenheng] Thank you. Quite interesting. A bunch of

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306195#comment-15306195 ] Sergey Soldatov commented on HBASE-15900: - [~chenheng] that would be great. Thank you! > RS

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306194#comment-15306194 ] Heng Chen commented on HBASE-15900: --- Sorry, [~sergey.soldatov] My RS has been restart, but it always

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306192#comment-15306192 ] Sergey Soldatov commented on HBASE-15900: - [~chenheng] yet again, could you use jstack with -l

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306187#comment-15306187 ] Heng Chen commented on HBASE-15900: --- [~stack] I have found something HStore.lock.readLock was hold by

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-29 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306119#comment-15306119 ] stack commented on HBASE-15900: --- I didn't know about the '-l'. It is not on mac os x it seems, only on

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-27 Thread Sergey Soldatov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303921#comment-15303921 ] Sergey Soldatov commented on HBASE-15900: - I would highly recommend to try get dump using

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-27 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303771#comment-15303771 ] Heng Chen commented on HBASE-15900: --- In the log, we can see, before request flush of region

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-27 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303759#comment-15303759 ] Heng Chen commented on HBASE-15900: --- Yes, something like this. I extract one region log which always

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-27 Thread Mikhail Antonov (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303751#comment-15303751 ] Mikhail Antonov commented on HBASE-15900: - I've seen that as well, for sure (w/o running MR

[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore

2016-05-27 Thread Heng Chen (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303748#comment-15303748 ] Heng Chen commented on HBASE-15900: --- Check code on HStore.java, the method which may hold write lock