mengqi created HBASE-26693:
------------------------------
Summary: Throw exception if region has same start key when
writePartitions use HFileOutputFormat2
Key: HBASE-26693
URL: https://issues.apache.org/jira/browse/HBASE-26693
Project: HBase
Issue Type: Improvement
Reporter: mengqi
Some exception info is not friendly in MapReduce task log such as:
{code:java}
2022-01-21 06:23:39,156 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalArgumentException: Can't read
partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:169)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.IOException: Wrong number of partitions in keyset
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:90)
... 10 more
{code}
when I write hfile use HFileOutputFormat2 in a MapReduce task.
I can not understand what happend to my job. And got some wrong suggestions
such as: trucate table or add cancelDeleteOnExit code in my code. Those are
useless and stupid.
*The real reason is my table has same start key ,and I didn't find this problem
in hbck web ui overlaps list immediately.*
{*}So{*}， I thanks it's a good idea to check same start key in
writePartitions(), we can print each same start key and throw a same start key
exception if my table regions has same start key.
It is more friendly exception info than 'Can't read partitions file'.
Here is my code
[https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1|https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1],]
can I create a pull request ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
