[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException
[
https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24021:
-
Fix Version/s: (was: 2.4.0)
> Fail fast when bulkLoadHFiles method catch some IOException
> ---
>
> Key: HBASE-24021
> URL: https://issues.apache.org/jira/browse/HBASE-24021
> Project: HBase
> Issue Type: Improvement
> Components: HFile, regionserver
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> In production environment, we usually do bulkload huge amount hfile . It
> reasonable fail fast when any IOException occur
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> {code:java}
> public Map> bulkLoadHFiles(Collection String>> familyPaths,
> boolean assignSeqId, BulkLoadListener bulkLoadListener,
> boolean copyFile, List clusterIds, boolean replicate) throws
> IOException {
> ..
> try {
> this.writeRequestsCount.increment();
> // There possibly was a split that happened between when the split keys
> // were gathered and before the HRegion's write lock was taken. We need
> // to validate the HFile region before attempting to bulk load all of them
> List ioes = new ArrayList<>();
> List> failures = new ArrayList<>();
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> HStore store = getStore(familyName);
> if (store == null) {
> IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> IOException e = MultipleIOException.createIOException(ioes);
> LOG.error("There were one or more IO errors when checking if the bulk
> load is ok.", e);
> throw e;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException
[
https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24021:
-
Summary: Fail fast when bulkLoadHFiles method catch some IOException (was:
Fail fast when bulkLoadHFiles method catch some IOException during invoke
assertBulkLoadHFileOK)
> Fail fast when bulkLoadHFiles method catch some IOException
> ---
>
> Key: HBASE-24021
> URL: https://issues.apache.org/jira/browse/HBASE-24021
> Project: HBase
> Issue Type: Improvement
> Components: HFile, regionserver
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> In production environment, we usually do bulkload huge amount hfile . It
> reasonable fail fast when any IOException occur
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> {code:java}
> public Map> bulkLoadHFiles(Collection String>> familyPaths,
> boolean assignSeqId, BulkLoadListener bulkLoadListener,
> boolean copyFile, List clusterIds, boolean replicate) throws
> IOException {
> ..
> try {
> this.writeRequestsCount.increment();
> // There possibly was a split that happened between when the split keys
> // were gathered and before the HRegion's write lock was taken. We need
> // to validate the HFile region before attempting to bulk load all of them
> List ioes = new ArrayList<>();
> List> failures = new ArrayList<>();
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> HStore store = getStore(familyName);
> if (store == null) {
> IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> IOException e = MultipleIOException.createIOException(ioes);
> LOG.error("There were one or more IO errors when checking if the bulk
> load is ok.", e);
> throw e;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException during invoke assertBulkLoadHFileOK
[
https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24021:
-
Description:
In production environment, we usually do bulkload huge amount hfile . It
reasonable fail fast when any IOException occur
hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
{code:java}
public Map> bulkLoadHFiles(Collection>
familyPaths,
boolean assignSeqId, BulkLoadListener bulkLoadListener,
boolean copyFile, List clusterIds, boolean replicate) throws
IOException {
..
try {
this.writeRequestsCount.increment();
// There possibly was a split that happened between when the split keys
// were gathered and before the HRegion's write lock was taken. We need
// to validate the HFile region before attempting to bulk load all of them
List ioes = new ArrayList<>();
List> failures = new ArrayList<>();
for (Pair p : familyPaths) {
byte[] familyName = p.getFirst();
String path = p.getSecond();
HStore store = getStore(familyName);
if (store == null) {
IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
"No such column family " + Bytes.toStringBinary(familyName));
ioes.add(ioe);
} else {
try {
store.assertBulkLoadHFileOk(new Path(path));
} catch (WrongRegionException wre) {
// recoverable (file doesn't fit in region)
failures.add(p);
} catch (IOException ioe) {
// unrecoverable (hdfs problem)
ioes.add(ioe);
}
}
}
// validation failed because of some sort of IO problem.
if (ioes.size() != 0) {
IOException e = MultipleIOException.createIOException(ioes);
LOG.error("There were one or more IO errors when checking if the bulk
load is ok.", e);
throw e;
}
{code}
was:
{code:java}
public Map> bulkLoadHFiles(Collection>
familyPaths,
boolean assignSeqId, BulkLoadListener bulkLoadListener,
boolean copyFile, List clusterIds, boolean replicate) throws
IOException {
long seqId = -1;
Map> storeFiles = new TreeMap<>(Bytes.BYTES_COMPARATOR);
Map storeFilesSizes = new HashMap<>();
Preconditions.checkNotNull(familyPaths);
// we need writeLock for multi-family bulk load
startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));
boolean isSuccessful = false;
try {
this.writeRequestsCount.increment();
// There possibly was a split that happened between when the split keys
// were gathered and before the HRegion's write lock was taken. We need
// to validate the HFile region before attempting to bulk load all of them
List ioes = new ArrayList<>();
List> failures = new ArrayList<>();
for (Pair p : familyPaths) {
byte[] familyName = p.getFirst();
String path = p.getSecond();
HStore store = getStore(familyName);
if (store == null) {
IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
"No such column family " + Bytes.toStringBinary(familyName));
ioes.add(ioe);
} else {
try {
store.assertBulkLoadHFileOk(new Path(path));
} catch (WrongRegionException wre) {
// recoverable (file doesn't fit in region)
failures.add(p);
} catch (IOException ioe) {
// unrecoverable (hdfs problem)
ioes.add(ioe);
}
}
}
// validation failed because of some sort of IO problem.
if (ioes.size() != 0) {
IOException e = MultipleIOException.createIOException(ioes);
LOG.error("There were one or more IO errors when checking if the bulk
load is ok.", e);
throw e;
}
{code}
> Fail fast when bulkLoadHFiles method catch some IOException during invoke
> assertBulkLoadHFileOK
> ---
>
> Key: HBASE-24021
> URL: https://issues.apache.org/jira/browse/HBASE-24021
> Project: HBase
> Issue Type: Improvement
> Components: HFile, regionserver
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> In production environment, we usually do bulkload huge amount hfile . It
> reasonable fail fast when any IOException occur
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> {code:java}
> public Map> bulkLoadHFiles(Collection String>> familyPaths,
> boolean assignSeqId, BulkLoadListener bulkLoadListener,
> boolean copyFile, List clusterIds, boolean replicate) throws
> IOException {
> ..
> try {
> this.writeRequestsCount.increment();
> // There possibly was a split that happened between when the split keys
> // were gathered and before the HRegion's write lock was taken. We need
>
[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException during invoke assertBulkLoadHFileOK
[
https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24021:
-
Description:
{code:java}
public Map> bulkLoadHFiles(Collection>
familyPaths,
boolean assignSeqId, BulkLoadListener bulkLoadListener,
boolean copyFile, List clusterIds, boolean replicate) throws
IOException {
long seqId = -1;
Map> storeFiles = new TreeMap<>(Bytes.BYTES_COMPARATOR);
Map storeFilesSizes = new HashMap<>();
Preconditions.checkNotNull(familyPaths);
// we need writeLock for multi-family bulk load
startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));
boolean isSuccessful = false;
try {
this.writeRequestsCount.increment();
// There possibly was a split that happened between when the split keys
// were gathered and before the HRegion's write lock was taken. We need
// to validate the HFile region before attempting to bulk load all of them
List ioes = new ArrayList<>();
List> failures = new ArrayList<>();
for (Pair p : familyPaths) {
byte[] familyName = p.getFirst();
String path = p.getSecond();
HStore store = getStore(familyName);
if (store == null) {
IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
"No such column family " + Bytes.toStringBinary(familyName));
ioes.add(ioe);
} else {
try {
store.assertBulkLoadHFileOk(new Path(path));
} catch (WrongRegionException wre) {
// recoverable (file doesn't fit in region)
failures.add(p);
} catch (IOException ioe) {
// unrecoverable (hdfs problem)
ioes.add(ioe);
}
}
}
// validation failed because of some sort of IO problem.
if (ioes.size() != 0) {
IOException e = MultipleIOException.createIOException(ioes);
LOG.error("There were one or more IO errors when checking if the bulk
load is ok.", e);
throw e;
}
{code}
> Fail fast when bulkLoadHFiles method catch some IOException during invoke
> assertBulkLoadHFileOK
> ---
>
> Key: HBASE-24021
> URL: https://issues.apache.org/jira/browse/HBASE-24021
> Project: HBase
> Issue Type: Improvement
> Components: HFile, regionserver
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> {code:java}
> public Map> bulkLoadHFiles(Collection String>> familyPaths,
> boolean assignSeqId, BulkLoadListener bulkLoadListener,
> boolean copyFile, List clusterIds, boolean replicate) throws
> IOException {
> long seqId = -1;
> Map> storeFiles = new TreeMap<>(Bytes.BYTES_COMPARATOR);
> Map storeFilesSizes = new HashMap<>();
> Preconditions.checkNotNull(familyPaths);
> // we need writeLock for multi-family bulk load
> startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));
> boolean isSuccessful = false;
> try {
> this.writeRequestsCount.increment();
> // There possibly was a split that happened between when the split keys
> // were gathered and before the HRegion's write lock was taken. We need
> // to validate the HFile region before attempting to bulk load all of them
> List ioes = new ArrayList<>();
> List> failures = new ArrayList<>();
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> HStore store = getStore(familyName);
> if (store == null) {
> IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> IOException e = MultipleIOException.createIOException(ioes);
> LOG.error("There were one or more IO errors when checking if the bulk
> load is ok.", e);
> throw e;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
