[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844288#comment-17844288 ] Duo Zhang commented on HBASE-28459: --- Good. Thanks. Let me resolve. > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Sravi Kommineni >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844284#comment-17844284 ] Bryan Beaudreault commented on HBASE-28459: --- [~zhangduo] , The user name is [~ksravista] . Their user request had been missed. I just approved it, added as contributor, and assigned to this jira. > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Sravi Kommineni >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844154#comment-17844154 ] Hudson commented on HBASE-28459: Results for branch branch-2 [build #1049 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1049/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1049/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1049/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1049/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1049/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844138#comment-17844138 ] Hudson commented on HBASE-28459: Results for branch branch-2.4 [build #731 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/731/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/731/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/731/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/731/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/731/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844104#comment-17844104 ] Hudson commented on HBASE-28459: Results for branch branch-2.6 [build #111 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/111/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/111/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/111/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843917#comment-17843917 ] Hudson commented on HBASE-28459: Results for branch branch-3 [build #199 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/199/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/199/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/199/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/199/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843881#comment-17843881 ] Hudson commented on HBASE-28459: Results for branch master [build #1067 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1067/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1067/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1067/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1067/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843868#comment-17843868 ] Hudson commented on HBASE-28459: Results for branch branch-2.5 [build #523 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/523/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/523/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/523/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/523/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/523/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843682#comment-17843682 ] Duo Zhang commented on HBASE-28459: --- Haven't found the jira account of the PR's author... I've already replied on the jira to let the contributor register a jira account. Thanks. > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)