[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762627#comment-16762627
]
Hudson commented on HBASE-21342:
Results for branch branch-1.2
[build #651 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/651/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/651//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/651//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/651//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762356#comment-16762356 ] Hudson commented on HBASE-21342: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1202 (See [https://builds.apache.org/job/HBase-1.2-IT/1202/]) HBASE-21374 Backport HBASE-21342 to branch-1 (busbey: [https://github.com/apache/hbase/commit/f7b51ccdf7414bc1deada02e3add556dc76e97be]) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSecureBulkLoadEndpoint.java > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740060#comment-16740060
]
Hudson commented on HBASE-21342:
Results for branch branch-1
[build #627 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/627/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/627//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/627//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/627//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740051#comment-16740051
]
Hudson commented on HBASE-21342:
Results for branch branch-1.4
[build #623 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/623/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/623//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/623//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/623//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740036#comment-16740036
]
Hudson commented on HBASE-21342:
Results for branch branch-1.3
[build #606 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/606/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/606//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/606//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/606//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739973#comment-16739973 ] Hudson commented on HBASE-21342: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #516 (See [https://builds.apache.org/job/HBase-1.3-IT/516/]) HBASE-21374 Backport HBASE-21342 to branch-1 (apurtell: [https://github.com/apache/hbase/commit/25d0c3ec7f80f51e814ed53caa9509fe60a4a8cf]) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSecureBulkLoadEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662299#comment-16662299
]
Hudson commented on HBASE-21342:
Results for branch master
[build #565 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/565/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/565//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/565//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/565//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661782#comment-16661782
]
Hudson commented on HBASE-21342:
Results for branch branch-2
[build #1435 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1435/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1435//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1435//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1435//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661744#comment-16661744
]
Hudson commented on HBASE-21342:
Results for branch branch-2.0
[build #1004 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1004/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1004//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1004//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1004//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661718#comment-16661718
]
Hudson commented on HBASE-21342:
Results for branch branch-2.1
[build #522 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661356#comment-16661356 ] Mike Drob commented on HBASE-21342: --- I'm not trying to ignore any versions or leave anything unfixed. I'm not trying to make policy. The fix versions are the set of versions that I had already done backports and pushed to. I hadn't pushed code to branch-1 or branch-2.0 yet, so they weren't included. I was using fix version in Jira as descriptive, not prescriptive. I wanted to be able to close the issue so that the RM could generate release notes if needed, and then expected to continue work in a separate backport issue. I have pushed this to branch-2.0+ There is a conflict cherry-picking to branch-1 that I don't have time to resolve today. I will open a backport Jira to not conflict with the release notes for Stack if he cuts a 2.1.1 tonight. > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661350#comment-16661350 ] Andrew Purtell commented on HBASE-21342: The affects versions are set. Shouldn’t the fix versions be the same if that code is similarly affected? Are we leaving known bugs in branch-1 unfixed by policy now? Separate back port JIRA is better than nothing but not much > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661336#comment-16661336 ] Mike Drob commented on HBASE-21342: --- I didn't drop the branch-1 versions, they were never in the fix version list. They're in affects version still. Paused my backports to consult with Stack offline about wether it's safe to commit to branch-2.1 right now, since I know he's prepping for a release. Will likely spin off separate backport issues. > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661275#comment-16661275 ] Andrew Purtell commented on HBASE-21342: Why did we drop all of the branch-1 fix versions? Is this not an issue there? > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661256#comment-16661256 ] Ted Yu commented on HBASE-21342: Mike: Please go ahead. Thanks > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661215#comment-16661215 ] Mike Drob commented on HBASE-21342: --- v7 LGTM. You want to commit this one, Ted? > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > HBASE-21342.006.patch, HBASE-21342.007.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661194#comment-16661194
]
Hadoop QA commented on HBASE-21342:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 33s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
41s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945244/HBASE-21342.007.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 937d90be826e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3b68e5393e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14825/testReport/ |
| Max. process+thread count | 4698 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14825/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> FileSystem in use may get
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660684#comment-16660684
]
Hadoop QA commented on HBASE-21342:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}137m
38s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945182/HBASE-21342.006.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0cfa10fc6521 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3b68e5393e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14820/testReport/ |
| Max. process+thread count | 4843 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14820/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> FileSystem in use
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660659#comment-16660659
]
Mike Drob commented on HBASE-21342:
---
Also, if you generate the patch using, {{git format-patch}} instead of {{git
diff}} then it will be easier for us to retain credit for your authorship in
the commit.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660651#comment-16660651
]
Mike Drob commented on HBASE-21342:
---
Thanks for the quick turnaround!
We have SecureBulkLoadManager and TestSecureBulkloadManager, capitalization
should be consistent.
{code}
final AtomicReference t1Exception = new AtomicReference();
final AtomicReference t2Exception = new AtomicReference();
{code}
This needs generics, my fault for pointing you at an example that didn't
include them.
{code}
Assert.assertEquals(null, t1Exception.get());
Assert.assertEquals(null, t2Exception.get());
{code}
Could be {{assertNull}}.
{code}
try {
h.doBulkLoad(dir, testUtil.getAdmin(), connection.getTable(TABLE),
connection.getRegionLocator(TABLE));
} catch (MyExceptionToAvoidRetry e) {
}
{code}
Please add a descriptive {{Assert.fail}} to the try block and a comment
{{//expected}} to the catch.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> HBASE-21342.006.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659108#comment-16659108
]
Mike Drob commented on HBASE-21342:
---
Thanks for the patch, [~mazhenlin]! It's looking much better now, I think the
structure is all there. Just a few more minor issues with implementation.
{code}
+ ConcurrentHashMap ugiReferenceCounter;
+ InternalObserverForFSOperation testObserver = null;
+ void incrementUgiReference(UserGroupInformation ugi) {
+ void decrementUgiReference(UserGroupInformation ugi) {
+ boolean isUserReferenced(UserGroupInformation ugi) {
{code}
should be private
bq. +ugiReferenceCounter = new ConcurrentHashMap();
Needs generics.
{code}
+ @VisibleForTesting
+ interface InternalObserverForFSOperation {
+void afterFileSystemCreated(HRegion r) throws Exception;
+ }
{code}
{code}
@Override
public void afterFileSystemCreated(HRegion r) throws Exception{
if (r.getRegionInfo().containsRow(key3)) {
ealierBulkload.join(); /// wait util the other bulkload finished
}
}
{code}
Is the exception throwing necessary? I can't tell if we actually use that to
fail the test or not. If not, then the interface could be {{Consumer}}
instead of declaring our own.
{code}
+ void incrementUgiReference(UserGroupInformation ugi) {
+ugiReferenceCounter.merge(ugi, 1, new BiFunction() {
+ @Override
+ public Integer apply(Integer integer, Integer integer2) {
+return ++integer;
+ }
+});
+ }
{code}
Personal preference would be to use a lambda here, but that's strictly personal
preference. Would be good practice to call those integer arguments something
more descriptive than "int" and "int2" for folks unfamiliar with how merge
works. Maybe "oldvalue" and "value"
{code}
+ boolean isUserReferenced(UserGroupInformation ugi) {
+return ugiReferenceCounter.containsKey(ugi) &&
ugiReferenceCounter.get(ugi) > 0;
+ }
{code}
Contains is effectively a get under the hood, so better performance would be to
do a single get and check against null and > 0.
{code}
+/// just for test
{code}
Unnecessary comment.
{code}
+/// create table
+HTableDescriptor htd2 = new HTableDescriptor(TABLE);
+htd2.addFamily(new HColumnDescriptor(FAMILY));
+testUtil.getHBaseAdmin().createTable(htd2,
Bytes.toByteArrays(SPLIT_ROWKEY));
{code}
Please don't use deprecated APIs. There's a helper method on HBaseTestUtility
for create table to make this easier. {{createTable(TableName, byte[] family,
byte[][] splits)}}
{code}
+HTableDescriptor desc = testUtil.getAdmin().getTableDescriptor(TABLE);
+HColumnDescriptor family = desc.getFamily(FAMILY);
{code}
Again, avoid deprecated APIs.
{code}
+ class MyExceptionToAvoidRetry extends DoNotRetryIOException {
{code}
Why is the original DoNotRetryIOException insufficient here? Probably best to
add a comment.
{code}
+ealierBulkload = new Thread(new Runnable() {
+ @Override
+ public void run() {
+try {
+ doBulkloadWithoutRetry(dir1);
+} catch (Exception e) {
+ LOG.error("bulk load failed .",e);
+}
+ }
+});
{code}
If this bulk load fails, do we want that to fail the whole test? I would assume
we do, you can look at
https://github.com/apache/hbase/blob/master/hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureNonce.java#L181-L182
for a pattern of how we do this in other places.
{code}
+@Category({RegionServerTests.class, SmallTests.class})
{code}
This starts a mini-cluster, so I think that would make it a medium test at a
minimum.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch,
> race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658392#comment-16658392 ] Ted Yu commented on HBASE-21342: [~mdrob]: Can you review the patch one more time ? Thanks > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, HBASE-21342.005.patch, > race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658372#comment-16658372
]
Hadoop QA commented on HBASE-21342:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 3 new + 3 unchanged
- 0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}145m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944909/HBASE-21342.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2f5b93405f3e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / dd474ef199 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14786/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14786/testReport/ |
| Max. process+thread count | 4679 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658354#comment-16658354
]
Hadoop QA commented on HBASE-21342:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 3 new + 3 unchanged
- 0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 50s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}143m 36s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944908/HBASE-21342.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5a0d9b55f3dc 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / dd474ef199 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14785/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14785/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658283#comment-16658283 ] Ted Yu commented on HBASE-21342: bq. It is in finally block. Perhaps I was not very clear in the previous comment. postBulkLoadHFile() throws IOE. When this happens, I don't think Java would run the rest of the code (decrementUgiReference in this case) in the finally block. You can either move decrementUgiReference call to the beginning of finally block (since it doesn't throw IOE) or, use nested finally in the current finally block. > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, HBASE-21342.002.patch, > HBASE-21342.003.patch, HBASE-21342.004.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658279#comment-16658279
]
mazhenlin commented on HBASE-21342:
---
{quote}Should the above be enclosed in finally block ?
{quote}
It is in finally block.
Other comments were fixed. Thank you very much for the review, I have learned a
lot.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, HBASE-21342.004.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657084#comment-16657084
]
Hadoop QA commented on HBASE-21342:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
20s{color} | {color:red} hbase-server: The patch generated 7 new + 3 unchanged
- 0 fixed = 10 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
40s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}229m 36s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}279m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944714/HBASE-21342.003.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 916f234338f2 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 05d22ed960 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14755/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656904#comment-16656904
]
Ted Yu commented on HBASE-21342:
{code}
+ interface InternalObserverForTest {
{code}
Since the interface has VisibleForTesting annotation, the interface name
doesn't have to include ForTest.
How about naming the interface InternalObserverForFSOperation or something
similar ?
{code}
+ decrementUgiReference(ugi);
{code}
Should the above be enclosed in finally block ?
{code}
+ public void testForRaceCondition() throws Exception {
{code}
There may be more test added to this class in the future. If including the race
condition in test name makes the method name too long, can you add a comment
describing the race ?
{code}
+ class MyExceptionToAvoidRetry extends DoNotRetryIOException {};
{code}
Put the right curly on separate line.
{code}
+ e.printStackTrace();
{code}
Please replace with a rethrow of the exception.
{code}
+Thread.sleep(3000); /// sleep 3s so the fs will be closed by the time
we wakeup
{code}
Is it possible to use some other condition so that the wait is not hardcoded ?
Sometime down the road, what if 3 second is not long enough ?
{code}
+ } catch (InterruptedException e) {}
{code}
Please log and rethrow InterruptedIOException.
thanks for the great work.
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: 21342.v1.txt, HBASE-21342.002.patch,
> HBASE-21342.003.patch, race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656618#comment-16656618
]
Hadoop QA commented on HBASE-21342:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
58s{color} | {color:red} hbase-server: The patch generated 21 new + 3 unchanged
- 0 fixed = 24 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
59s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}140m
27s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944670/HBASE-21342.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0e006bc17937 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 05d22ed960 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14752/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14752/testReport/ |
| Max. process+thread count | 5040 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656447#comment-16656447 ] Ted Yu commented on HBASE-21342: Can you change the name of the test table to reflect the nature of the test ? Thanks > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, HBASE-21342.002.patch, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656372#comment-16656372
]
Hadoop QA commented on HBASE-21342:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 47s{color}
| {color:red} hbase-server generated 3 new + 185 unchanged - 3 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
13s{color} | {color:red} hbase-server: The patch generated 29 new + 3 unchanged
- 0 fixed = 32 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
21s{color} | {color:red} patch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 34s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables |
| | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944661/21342.v1.txt |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a9ef520180ed 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 05d22ed960 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe;
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656262#comment-16656262 ] Ted Yu commented on HBASE-21342: Added ClassRule to TestSecureBulkloadManager. Let's see what QA bot says. Zhenlin: Please add Apache license header to TestSecureBulkloadManager in your next patch. > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: 21342.v1.txt, race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655732#comment-16655732
]
Mike Drob commented on HBASE-21342:
---
{quote}
+ private static byte[] value2 = Bytes.toBytes("t2");
+ private static byte[] value3 = Bytes.toBytes("t2");
{quote}
These are the same.
We should probably be using ConcurrentHashMap and the various concurrency
functions there instead of making our own big synchronized blocks. Looking at
putIfAbsent, computeIfAbsent, compute, and merge in particular.
When I try to run the test on master branch I get
{{java.lang.ArrayIndexOutOfBoundsException: 0}} so something is going wrong
there... which branch did you work on this for initially?
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655712#comment-16655712 ] mazhenlin commented on HBASE-21342: --- [~mdrob] Thank you. I will submit my patch tomorrow. Right now I have not figured out how to submit:( > FileSystem in use may get closed by other bulk load call in secure bulkLoad > > > Key: HBASE-21342 > URL: https://issues.apache.org/jira/browse/HBASE-21342 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7 >Reporter: mazhenlin >Assignee: mazhenlin >Priority: Major > Attachments: race.patch > > > As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If > Two secure bulkload calls from the same UGI into two different regions and > one region finishes earlier, it will close the bulk load fs, and the other > region will fail. > > Another case would be more serious. The FileSystem.close() function needs two > synchronized variables : CACHE and deleteOnExit. If one region calls > FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while > another region is trying to close srcFS ( in > SecureBulkLoadListener.closeSrcFs) , can cause deadlock here. > > I have wrote a UT for this and fixed it using reference counter. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21342) FileSystem in use may get closed by other bulk load call in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655710#comment-16655710
]
Ted Yu commented on HBASE-21342:
{code}
+import org.apache.commons.collections.map.HashedMap;
{code}
Did you intend to import HashMap instead ?
{code}
+ * Created by mazhenlin on 2018/10/18.
{code}
Please drop author.
{code}
+public class TestSecureBulkloadManager implements InternalObserverForTest {
{code}
Missing test category.
The test also misses @ClassRule e.g.
{code}
@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestCopyTable.class);
{code}
> FileSystem in use may get closed by other bulk load call in secure bulkLoad
>
>
> Key: HBASE-21342
> URL: https://issues.apache.org/jira/browse/HBASE-21342
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1, 1.2.7
>Reporter: mazhenlin
>Assignee: mazhenlin
>Priority: Major
> Attachments: race.patch
>
>
> As mentioned in [HBASE-15291|#HBASE-15291], there is a race condition. If
> Two secure bulkload calls from the same UGI into two different regions and
> one region finishes earlier, it will close the bulk load fs, and the other
> region will fail.
>
> Another case would be more serious. The FileSystem.close() function needs two
> synchronized variables : CACHE and deleteOnExit. If one region calls
> FileSystem.closeAllForUGI ( in SecureBulkLoadManager.cleanupBulkLoad) while
> another region is trying to close srcFS ( in
> SecureBulkLoadListener.closeSrcFs) , can cause deadlock here.
>
> I have wrote a UT for this and fixed it using reference counter.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
