[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13489947#comment-13489947
]
Jie Huang commented on HBASE-6416:
--
I am taking sick leave from Oct. 31 to Nov. 13. Any urgency, please call
18964958151. Sorry for any action delay during these days. Thanks.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.4
Attachments: hbase-6416.patch, hbase-6416-v1.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483002#comment-13483002
]
Jie Huang commented on HBASE-6416:
--
I am taking sick leave from Oct. 24 to Oct. 30. Any urgency, please call
18964958151. Sorry for any action delay during these days. Thanks.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.94.3, 0.96.0
Attachments: hbase-6416.patch, hbase-6416-v1.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE
[ https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474822#comment-13474822 ] Jie Huang commented on HBASE-6069: -- I am taking sick leave from Oct. 10 to Oct. 23. Any urgency, please call 18964958151. Sorry for any action delay during these days. Thanks. TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE --- Key: HBASE-6069 URL: https://issues.apache.org/jira/browse/HBASE-6069 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.94.0 Reporter: Jie Huang Assignee: Jie Huang Priority: Critical Fix For: 0.94.1, 0.96.0 Attachments: 6069-v2.txt, HBASE-6069.patch While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, there always throws a Null Pointer Exception on Scanner object. Since the TableInputFormatBase#createRecordReader() missed the initialization of TableRecordReader object. The scanner will be null in that case. This issue causes Hive query fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[
https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455507#comment-13455507
]
Jie Huang commented on HBASE-6592:
--
Thanks Stack. 'c(CLASSNAME).METHOD' is much better. What I mean is that the
user can define their own converter function in the CONVERTER definition. :)
For that Exception, it is caused by the converter itself. We also met the
similar problem while running java code.
The current Bytes.toXXX needs to check the length of the byte array in advance.
For example, if you want to encode *1*(Int) to a byte array, the array will
contain 4 bytes (something like \000\000\000\001). And then you are able to
convert that byte array to an Int object. You can find the corresponding code
in *org.apache.hadoop.hbase.util.Bytes.java*. In order to get rid of those
'\000' in the ruby shell result, here we allow the end user to provide a
converter to print out the proper output format as they like.
{code}
public static int toInt(byte[] bytes, int offset, final int length) {
if (length != SIZEOF_INT || offset + length bytes.length) { //check the
length here.
throw explainWrongLengthOrOffset(bytes, offset, length, SIZEOF_INT);
}
int n = 0;
for(int i = offset; i (offset + length); i++) {
n = 8;
n ^= bytes[i] 0xFF;
}
return n;
}
{code}
Meanwhile, we also allow the end user to provide their own converter for the
case as you pointed out. They can define a converter as 'c(CLASSNAME).METHOD'
as you mentioned.
[shell] Add means of custom formatting output by column
---
Key: HBASE-6592
URL: https://issues.apache.org/jira/browse/HBASE-6592
Project: HBase
Issue Type: New Feature
Components: shell
Reporter: stack
Priority: Minor
Labels: noob
Attachments: hbase-6592.patch, hbase-6592-v2.patch,
hbase-6952-v1.patch
See Jacques suggestion toward end of this thread for how we should allow
adding a custom formatter per column to use outputting column content in
shell:
http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455521#comment-13455521
]
Jie Huang commented on HBASE-6416:
--
Sorry for my late response.
My concern is that now we try to fix .tableinfo file during the online phase,
which only happens after the offline phase. However, the fixHdfsOrphan does
happen during the offline phase. I wonder if it is possible to recover those
missing files respectively. And then, re-run the hbck for a better status.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.3
Attachments: hbase-6416.patch, hbase-6416-v1.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[
https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455537#comment-13455537
]
Jie Huang commented on HBASE-6592:
--
here list the examples in the unit test case:
{code}
+
+define_test get should support COLUMNS with value CONVERTER information
do
+@test_table.put(1, x:c, [1024].pack('N'))
+@test_table.put(1, x:d, [98].pack('N'))
+begin
+ res = @test_table._get_internal('1', ['x:c:toInt'],
['x:d:c(org.apache.hadoop.hbase.util.Bytes).toInt'])
+ assert_not_nil(res)
+ assert_kind_of(Hash, res)
+ assert_not_nil(/value=1024/.match(res['x:c']))
+ assert_not_nil(/value=98/.match(res['x:d']))
+ensure
+ # clean up newly added columns for this test only.
+ @test_table.delete(1, x:c)
+ @test_table.delete(1, x:d)
+end
+end
+
+define_test scan should support COLUMNS with value CONVERTER information
do
+ @test_table.put(1, x:c, [1024].pack('N'))
+ @test_table.put(1, x:d, [98].pack('N'))
+ begin
+res = @test_table._scan_internal COLUMNS = ['x:c:toInt',
'x:d:c(org.apache.hadoop.hbase.util.Bytes).toInt']
+assert_not_nil(res)
+assert_kind_of(Hash, res)
+assert_not_nil(/value=1024/.match(res['1']['x:c']))
+assert_not_nil(/value=98/.match(res['1']['x:d']))
+ ensure
+# clean up newly added columns for this test only.
+@test_table.delete(1, x:c)
+@test_table.delete(1, x:d)
+end
+end
{code}
[shell] Add means of custom formatting output by column
---
Key: HBASE-6592
URL: https://issues.apache.org/jira/browse/HBASE-6592
Project: HBase
Issue Type: New Feature
Components: shell
Reporter: stack
Priority: Minor
Labels: noob
Attachments: hbase-6592.patch, hbase-6592-v2.patch,
hbase-6952-v1.patch
See Jacques suggestion toward end of this thread for how we should allow
adding a custom formatter per column to use outputting column content in
shell:
http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453779#comment-13453779 ] Jie Huang commented on HBASE-6592: -- [~stack], any comment on the latest patch file? thanks. [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6592-v2.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6500) hbck comlaining, Exception in thread main java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HBASE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13454536#comment-13454536 ] Jie Huang commented on HBASE-6500: -- I guess so. Meanwhile, I am wondering if we need to add one more unit test case to verfiy this case. Since it is mentioned again and again. I can take that. What do you think? hbck comlaining, Exception in thread main java.lang.NullPointerException --- Key: HBASE-6500 URL: https://issues.apache.org/jira/browse/HBASE-6500 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Environment: Hadoop 0.20.205.0 Zookeeper: zookeeper-3.3.5.jar Hbase: hbase-0.94.0 Reporter: liuli I met problem with starting Hbase: I have 5 machines (Ubuntu) 109.123.121.23 rsmm-master.example.com 109.123.121.24 rsmm-slave-1.example.com 109.123.121.25 rsmm-slave-2.example.com 109.123.121.26 rsmm-slave-3.example.com 109.123.121.27 rsmm-slave-4.example.com Hadoop 0.20.205.0 Zookeeper: zookeeper-3.3.5.jar Hbase: hbase-0.94.0 After starting HBase, running hbck hduser@rsmm-master:~/hbase/bin$ ./hbase hbck 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:host.name=rsmm-master.example.com 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_33 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jre1.6.0_33 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client
[jira] [Created] (HBASE-6771) To recover parts of the properties of .tableinfo file by reading HFile
Jie Huang created HBASE-6771: Summary: To recover parts of the properties of .tableinfo file by reading HFile Key: HBASE-6771 URL: https://issues.apache.org/jira/browse/HBASE-6771 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.96.0 Reporter: Jie Huang Assignee: Jie Huang Currently, we only fabricate a bare minimum .tableinfo version while it is missing in hbck. The end-user still needs to correct them later. According to the [~jmhsieh]'s proposal in HBASE-5631, we'd better to recover it by reading some properties(e.g., compression settings, encodings, etc) from exiting newest HFile under each region folder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6592: - Attachment: (was: hbase-6592-v2.patch) [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6592-v2.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6592: - Attachment: hbase-6592-v2.patch [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6592-v2.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451586#comment-13451586 ] Jie Huang commented on HBASE-5631: -- Thanks. Nice to have that FamilyDirFilter() already. bq. Jie Huang with regards to the second cut – I was suggesting that if you want tackle it, file a new separate jira after this one is handled. No need for creep scope on this jira. Yes. Sure. I will take the new issue for the second cut as well. Thanks. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: hbase-5631-v2.patch Done. Thank you again. :) hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch, hbase-5631-v2.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451588#comment-13451588
]
Jie Huang commented on HBASE-6416:
--
Yes. No problem. I will try to handle the other part too. Thanks for your
suggestion. I will upload a new patch file soon.
Meanwhile, I wonder if we need to tell the user to run -fixOrphanTable at
first, and then re-run -fixHdfsOrphan again. Since it can somehow help them
repair the system when facing the similar problems.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.3
Attachments: hbase-6416.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[
https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451721#comment-13451721
]
Jie Huang commented on HBASE-5631:
--
bq. I noticed one thing as I was committing – we don't actually test the
non-cached tableinfo fabrication path. File a follow-on issue for that?
I have tested the non-cached case as well. see following lines
{code}
+
+ // fix OrphanTable with default .tableinfo non-cached case here
+ hbck = doFsck(conf, true);
+ assertNoErrors(hbck);
+ status = null;
+ status = FSTableDescriptors.getTableInfoPath(fs, hbaseTableDir);
+ assertNotNull(status);
+
+ HTableDescriptor htd = admin.getTableDescriptor(table.getBytes());
+ htd.setValue(NOT_DEFAULT, true);
+ admin.disableTable(table);
+ admin.modifyTable(table.getBytes(), htd);
+ admin.enableTable(table);
+ fs.delete(status.getPath(), true);
+
+ // fix OrphanTable with cache cached case here
+ htd = admin.getTableDescriptor(table.getBytes());
+ hbck = doFsck(conf, true);
+ assertNoErrors(hbck);
+ status = null;
+ status = FSTableDescriptors.getTableInfoPath(fs, hbaseTableDir);
+ assertNotNull(status);
+ htd = admin.getTableDescriptor(table.getBytes());
+ assertEquals(htd.getValue(NOT_DEFAULT), true);
{code}
hbck should handle case where .tableinfo file is missing.
-
Key: HBASE-5631
URL: https://issues.apache.org/jira/browse/HBASE-5631
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jie Huang
Fix For: 0.96.0, 0.92.3, 0.94.2
Attachments: hbase-5631.patch, hbase-5631-v1.patch,
hbase-5631-v2.patch
0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck
should be able to detect and repair this properly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6416:
-
Attachment: hbase-6416-v1.patch
Done with quick fix.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.3
Attachments: hbase-6416.patch, hbase-6416-v1.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[
https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6592:
-
Attachment: hbase-6592-v2.patch
@Stack, Update the patch file with following changes.
# add test cases for both get and scan.
# add usage examples in scan/get helper.
Here is the usage info for this feature. Anything improper, please let me know.
thanks.
{noformat}
Besides the default 'toStringBinary' format, scan also supports customized
output format for each column.
User can define a CONVERTER, which follows the general column spec of
cf:qualifier, like cf:qualifier[:CONVERTER].
The CONVERTER can be formatted as
1. either converterFun alone (e.g, toInt, toString)
2. or a complete pattern of 'c(MyConverterClass).converterFun'.
While using the 1st. CONVERTER definition, it chooses
org.apache.hadoop.hbase.util.Bytes as the default class.
Example:
hbase scan 't1', {COLUMNS = ['cf:qualifier:toInt',
'cf:qualifier:c(org.apache.hadoop.hbase.util.Bytes).toInt'] }
ATTENTION!The above feature cannot be applied on the entire column family,
since it is too confusing to distinguish cf:qualifier and cf:CONVERTER
{noformat}
[shell] Add means of custom formatting output by column
---
Key: HBASE-6592
URL: https://issues.apache.org/jira/browse/HBASE-6592
Project: HBase
Issue Type: New Feature
Components: shell
Reporter: stack
Priority: Minor
Labels: noob
Attachments: hbase-6592.patch, hbase-6592-v2.patch,
hbase-6952-v1.patch
See Jacques suggestion toward end of this thread for how we should allow
adding a custom formatter per column to use outputting column content in
shell:
http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[
https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450399#comment-13450399
]
Jie Huang commented on HBASE-6592:
--
Oops. Here is a concrete example. c1 and c2 are formated as
cf:qualifier:format
{noformat}
get 't2', 'r2', [ 'f:e_word:toString','f:f_word:toString' ]
{noformat}
[shell] Add means of custom formatting output by column
---
Key: HBASE-6592
URL: https://issues.apache.org/jira/browse/HBASE-6592
Project: HBase
Issue Type: New Feature
Components: shell
Reporter: stack
Priority: Minor
Labels: noob
Attachments: hbase-6592.patch, hbase-6952-v1.patch
See Jacques suggestion toward end of this thread for how we should allow
adding a custom formatter per column to use outputting column content in
shell:
http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450406#comment-13450406 ] Jie Huang commented on HBASE-6592: -- From my understanding, the formatting converter is applied on the certain qualifier. Otherwise, it will be confusing between cf:q and cf:format. what do you think? [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450411#comment-13450411 ] Jie Huang commented on HBASE-6592: -- I should have written more concretely in the help notes. Sorry. [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450428#comment-13450428 ] Jie Huang commented on HBASE-6592: -- Thanks Stack. More information in the help and code comments will be expected in the coming version. :) [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch, hbase-6952-v1.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6416:
-
Attachment: hbase-6416.patch
Yes. The problem can be reproduced in my test environment. And here attaches
one fix for this bug to solve that *hanging around* issue.
According to the jstack result, the most suspicious threads are those
pool-1-thread-* threads (which are not daemon). I guess it is caused by the
following ExecutionException, which leads to an incomplete FutureTask.
{code}
for(int i=0; ihbiFutures.size(); i++) {
WorkItemHdfsRegionInfo work = hbis.get(i);
FutureVoid f = hbiFutures.get(i);
try {
f.get();
} catch(ExecutionException e) {
LOG.warn(Failed to read .regioninfo file for region +
work.hbi.getRegionNameAsString(), e.getCause());
}
}
{code}
The simple and quick fix is to shutdown the ExecutorService finally in exec()
function.
Now, in my test environment, the process will be terminated with that NPE after
applied this patch.
hbck dies on NPE when a region folder exists but the table does not
---
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.3
Attachments: hbase-6416.patch
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull(Table + tableName + ' not present!,
tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: hbase-5631-v1.patch Updated the patch file. 1. if get that htd from cache, create a new .tableinfo accordingly. 2. else create a default .tableinfo file with the correct table name, column family and default properties. Hint the end-user to modify that default .tableinfo file if necessary. 3. add the unit test according to Jon's comments. @Jonathon, What do you think? hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449604#comment-13449604 ] Jie Huang commented on HBASE-5631: -- Oops. Sorry to miss your latest comment. For the second cut, I haven't done in this patch version. We may try to figure out all those properties we can get from the HFile. Thanks Jonathan. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[
https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6592:
-
Attachment: hbase-6952-v1.patch
Thanks for your comment. Updated the patch file.
Add the usage examples in the shell help for both *scan* and *get* as follows
{noformat}
hbase get 't1', 'r1', {COLUMN = ['c1:toInt',
'c2:c(org.apache.hadoop.hbase.util.Bytes).toInt'] }
hbase scan 't1', {COLUMNS = ['c1:toInt',
'c2:c(org.apache.hadoop.hbase.util.Bytes).toInt'] }
{noformat}
[shell] Add means of custom formatting output by column
---
Key: HBASE-6592
URL: https://issues.apache.org/jira/browse/HBASE-6592
Project: HBase
Issue Type: New Feature
Components: shell
Reporter: stack
Priority: Minor
Labels: noob
Attachments: hbase-6592.patch, hbase-6952-v1.patch
See Jacques suggestion toward end of this thread for how we should allow
adding a custom formatter per column to use outputting column content in
shell:
http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6592: - Attachment: hbase-6592.patch [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6592: - Attachment: (was: hbase-6592.patch) [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448548#comment-13448548 ] Jie Huang commented on HBASE-6592: -- Add unit-test for this new feature. Any idea? [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: (was: hbase-5631-trunk.patch) hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: hbase-5631.patch here attaches the patch file for this feature. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6592: - Status: Patch Available (was: Open) [shell] Add means of custom formatting output by column --- Key: HBASE-6592 URL: https://issues.apache.org/jira/browse/HBASE-6592 Project: HBase Issue Type: New Feature Components: shell Reporter: stack Priority: Minor Labels: noob Attachments: hbase-6592.patch See Jacques suggestion toward end of this thread for how we should allow adding a custom formatter per column to use outputting column content in shell: http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shellsubj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449351#comment-13449351 ] Jie Huang commented on HBASE-5631: -- Thanks Jonathan. bq. Have you tried shutting down the cluster and then restarting it? I have a suspicion that this may not work if the HTD isn't cached. Yes. you are right. This proposal only fixes .tableinfo files which have been already cached. From my understanding, we have no way to figure out the entire table information without cache. Another thought is that we can try to recover parts of the information(like. table name, column-family name with some default properties). Meanwhile some warnings are printed out since part of the information(what we guessed) may not be as accurate as before. Is there any other suggestion? Please let me know. bq. I wasn't consistent with error.print vs log. I think I prefer log. Any reason you picked this vs the other? I have hesitated between log and error.print before. I will change that in the next version. thanks. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6516: - Attachment: hbase-6516-v4.patch Thanks. How about this one? hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Assignee: Jie Huang Attachments: hbase-6516.patch, hbase-6516-v2.patch, hbase-6516-v3.patch, hbase-6516-v4.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[
https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447693#comment-13447693
]
Jie Huang commented on HBASE-4364:
--
[~alexb], [~lucjb] I have checked those code related to the problem you
mentioned. According to the comments, it seems that this topic has been
discussed before. The conclusion is that
{code}
/**
* Filters should be checked before checking column trackers. If we do
* otherwise, as was previously being done, ColumnTracker may increment its
* counter for even that KV which may be discarded later on by Filter. This
* would lead to incorrect results in certain cases.
*/
if (filter != null) {
ReturnCode filterResponse = filter.filterKeyValue(kv);
if (filterResponse == ReturnCode.SKIP) {
return MatchCode.SKIP;
} else if (filterResponse == ReturnCode.NEXT_COL) {
return columns.getNextRowOrNextColumn(bytes, offset, qualLength);
} else if (filterResponse == ReturnCode.NEXT_ROW) {
stickyNextRow = true;
return MatchCode.SEEK_NEXT_ROW;
} else if (filterResponse == ReturnCode.SEEK_NEXT_USING_HINT) {
return MatchCode.SEEK_NEXT_USING_HINT;
}
}
MatchCode colChecker = columns.checkColumn(bytes, offset, qualLength,
timestamp, type, kv.getMemstoreTS() maxReadPointToTrackVersions);
{code}
If both of you are still interested in this problem, we may try to figure out
some potential solution. Any comment?
Filters applied to columns not in the selected column list are ignored
--
Key: HBASE-4364
URL: https://issues.apache.org/jira/browse/HBASE-4364
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Todd Lipcon
Priority: Critical
Attachments:
HBASE-4364-failing-test-with-simplest-custom-filter.patch,
hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch
For a scan, if you select some set of columns using addColumns(), and then
apply a SingleColumnValueFilter that restricts the results based on some
other columns which aren't selected, then those filter conditions are ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: hbase-5631-trunk.patch I have tried to implement this feature against to trunk. Any suggestion? A simple test has been done over a 4-node small cluster. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Attachments: hbase-5631-trunk.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6516: - Attachment: hbase-6516-v3.patch Update the patch file according to Jon's comments. One more unit-test case is added as well. Any idea, please feel free to let me know. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516.patch, hbase-6516-v2.patch, hbase-6516-v3.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439982#comment-13439982 ] Jie Huang commented on HBASE-6516: -- Thanks. I will upload a new patch soon. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516.patch, hbase-6516-v2.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437846#comment-13437846 ] Jie Huang commented on HBASE-6516: -- Agree with you to some extent. However I still have one question. How about .META. table? Since for that kind of table, it is quite acceptable not to have the .tableinfo file there. Actually, it is not an IOException or TableInfoMissingException for that case. Shall we handle this case in *getTableDescriptor()* ? OR let its final caller to handle it? Besides, I found the HBaseIOException was introduced in HBASE-6586, so will provide the new patch later. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516.patch, hbase-6516-v2.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435984#comment-13435984 ] Jie Huang commented on HBASE-4364: -- @Alex, totally understand your point. Yes. What you have stated in the unit-test is a problem. Not only for the functionality, but also for the better performance as Lucas mentioned. We'd better to feed all those *selected columns* to the filter. However, to make the filter only work on those selected columns doesn't help to solve the original problem here. The problem is sth. like *select A from table where B 'b'*. If we only applies the filtering criteria on column A as what we expected in the unit-test case, we will get empty result here. So basically, the solution proposed here is # To add filterred column into the selected columns, and then do the filter. The sanityCheck() function gives the chance to combine projected and where if necessary. # To remove that newly added column before returning back to the end-user. Since that newly added column is not expected by the end-user. After making the filter only work on those required columns, using suck kind of solution, we still can extend the projected by adding those where columns. And the filter may apply on required + criteria column only, instead of going through the whole column list. How about to create a new issue about the only feeds those selected columns to the filter. The initial thought is to do some comparison work in the matcher. Any comment? Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[
https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436485#comment-13436485
]
Jie Huang commented on HBASE-4364:
--
Actually, I know we can use SingleColumnValueExcludeFilter to solve the problem
somehow. That is why I said this is not a bug before. The document already
hints to include both required and criteria columns altogether. Otherwise,
the criteria column will be regarded as missing part in the filter. The correct
way is:
{noformat}
scan 't2', { COLUMNS = ['f:e_word', 'f:f_word'], FILTER =
SingleColumnValueExcludeFilter('f', 'f_word', , 'binary:b') }
{noformat}
But, SingleColumnValueExcludeFilter only helps to not return that filter's
criteria column. We still need to specify both e_word and f_word columns
in the required list. Otherwise, we still get the wrong result like this:
{noformat}
scan 't2', { COLUMNS = ['f:e_word'], FILTER =
SingleColumnValueExcludeFilter('f', 'f_word', , 'binary:b') }
ROWCOLUMN+CELL
r1column=f:e_word, timestamp=1345084564838,
value=hello
r2column=f:e_word, timestamp=1345084564895,
value=goodbye
{noformat}
The points of that proposal in the patch file are :
# *select A from table where B'b'* is more acceptable by SQL user rather than
*select A exclusively (B) from table where B'b'*
# To add one chance to combine required and criteria columns before doing
the filtering work, which may be used by other filter later. It is not only for
SingleColumnValueFilter.
For the other problem in the unit-test, shall we create a new issue and have
further discussion there?
Filters applied to columns not in the selected column list are ignored
--
Key: HBASE-4364
URL: https://issues.apache.org/jira/browse/HBASE-4364
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Todd Lipcon
Priority: Critical
Attachments:
HBASE-4364-failing-test-with-simplest-custom-filter.patch,
hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch
For a scan, if you select some set of columns using addColumns(), and then
apply a SingleColumnValueFilter that restricts the results based on some
other columns which aren't selected, then those filter conditions are ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6066) some low hanging read path improvement ideas
[
https://issues.apache.org/jira/browse/HBASE-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436506#comment-13436506
]
Jie Huang commented on HBASE-6066:
--
Regarding Part #5, We noticed that nested synchronized phenomenon
as well. But unfortunately, after eliminating that nested *synchronize* method
for RegionScanner.next(), it doesn't improve any. Just like the description
mentioned, the nested synchronized methods only adds small overhead.
In my opinion, that overhead is far less than the synchronized method itself.
But I have a question here, why we need to synchronize that .next() function
here? Is there any particular concern? thanks.
some low hanging read path improvement ideas
-
Key: HBASE-6066
URL: https://issues.apache.org/jira/browse/HBASE-6066
Project: HBase
Issue Type: Improvement
Reporter: Kannan Muthukkaruppan
Assignee: Michal Gregorczyk
Priority: Critical
Labels: noob
Attachments: metric-stringbuilder-fix.patch
I was running some single threaded scan performance tests for a table with
small sized rows that is fully cached. Some observations...
We seem to be doing several wasteful iterations over and/or building of
temporary lists.
1) One such is the following code in HRegionServer.next():
{code}
boolean moreRows = s.next(values, HRegion.METRIC_NEXTSIZE);
if (!values.isEmpty()) {
for (KeyValue kv : values) { -- wasteful in most
cases
currentScanResultSize += kv.heapSize();
}
results.add(new Result(values));
{code}
By default the maxScannerResultSize is Long.MAX_VALUE. In those cases,
we can avoid the unnecessary iteration to compute currentScanResultSize.
2) An example of a wasteful temporary array, is results in
RegionScanner.next().
{code}
results.clear();
boolean returnResult = nextInternal(limit, metric);
outResults.addAll(results);
{code}
results then gets copied over to outResults via an addAll(). Not sure why we
can not directly collect the results in outResults.
3) Another almost similar exmaple of a wasteful array is results in
StoreScanner.next(), which eventually also copies its results into
outResults.
4) Reduce overhead of size metric maintained in StoreScanner.next().
{code}
if (metric != null) {
HRegion.incrNumericMetric(this.metricNamePrefix + metric,
copyKv.getLength());
}
results.add(copyKv);
{code}
A single call to next() might fetch a lot of KVs. We can first add up the
size of those KVs in a local variable and then in a finally clause increment
the metric one shot, rather than updating AtomicLongs for each KV.
5) RegionScanner.next() calls a helper RegionScanner.next() on the same
object. Both are synchronized methods. Synchronized methods calling nested
synchronized methods on the same object are probably adding some small
overhead. The inner next() calls isFilterDone() which is a also a
synchronized method. We should factor the code to avoid these nested
synchronized methods.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435159#comment-13435159 ] Jie Huang commented on HBASE-6516: -- @Jonathan, Any idea about the comment? thanks. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516.patch, hbase-6516-v2.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435163#comment-13435163 ] Jie Huang commented on HBASE-6245: -- Any comment on that patch file? Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6500) hbck comlaining, Exception in thread main java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HBASE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435176#comment-13435176 ] Jie Huang commented on HBASE-6500: -- I found the similar problem in HBASE-6464. hbck comlaining, Exception in thread main java.lang.NullPointerException --- Key: HBASE-6500 URL: https://issues.apache.org/jira/browse/HBASE-6500 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Environment: Hadoop 0.20.205.0 Zookeeper: zookeeper-3.3.5.jar Hbase: hbase-0.94.0 Reporter: liuli I met problem with starting Hbase: I have 5 machines (Ubuntu) 109.123.121.23 rsmm-master.example.com 109.123.121.24 rsmm-slave-1.example.com 109.123.121.25 rsmm-slave-2.example.com 109.123.121.26 rsmm-slave-3.example.com 109.123.121.27 rsmm-slave-4.example.com Hadoop 0.20.205.0 Zookeeper: zookeeper-3.3.5.jar Hbase: hbase-0.94.0 After starting HBase, running hbck hduser@rsmm-master:~/hbase/bin$ ./hbase hbck 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:host.name=rsmm-master.example.com 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_33 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jre1.6.0_33 28/12/12 17:13:29 INFO zookeeper.ZooKeeper: Client
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435659#comment-13435659 ] Jie Huang commented on HBASE-4364: -- Yes. I will refine my patch against the trunk. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6245: - Attachment: hbase-6245-v2.patch Thank you very much for your suggestions. Here attaches the updated version. More idea, please let me know. Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6245: - Attachment: (was: hbase-6245-v2.patch) Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6245: - Attachment: hbase-6245-v2.patch Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch, hbase-6245-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-4364: - Attachment: hbase-4364_trunk-v2.patch Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[
https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435775#comment-13435775
]
Jie Huang commented on HBASE-4364:
--
I have updated the patch file against the latest trunk version. This patch only
aims to solve the problem which is stated in this bug entry. So that the user
can do sth. like
{noformat}
create 't2', 'f'
put 't2', 'r1', 'f:e_word', 'hello'
put 't2', 'r1', 'f:f_word', 'bonjour'
put 't2', 'r2', 'f:e_word', 'goodbye'
put 't2', 'r2', 'f:f_word', 'au revoir'
# scan with a predicate of the french word 'b', selecting only the english
word
# it only returns r1 column=f:e_word, value=hello instead of
# r1 column=f:e_word, value=hello; r2 column=f:e_word, value=goodbye
scan 't2', { COLUMNS = ['f:e_word'], FILTER = SingleColumnValueFilter('f',
'f_word', , 'binary:b') }
{noformat}
From my perspective, this case is different with that case stated in the
unit-test file. Here is a simple solution to make that unit-test case passed.
{code}
private static class ValueHasMoreThan2BytesFilter extends FilterBase {
@Override
public ReturnCode filterKeyValue(KeyValue kv) {
byte[] val = kv.getValue();
if (val.length 2) {
return ReturnCode.INCLUDE;
} else {
return ReturnCode.NEXT_ROW; // it should be return ReturnCode.SKIP;
}
}
{code}
Then we can get the expected results here.
Filters applied to columns not in the selected column list are ignored
--
Key: HBASE-4364
URL: https://issues.apache.org/jira/browse/HBASE-4364
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Todd Lipcon
Priority: Critical
Attachments:
HBASE-4364-failing-test-with-simplest-custom-filter.patch,
hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch
For a scan, if you select some set of columns using addColumns(), and then
apply a SingleColumnValueFilter that restricts the results based on some
other columns which aren't selected, then those filter conditions are ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[
https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432958#comment-13432958
]
Jie Huang commented on HBASE-6516:
--
Furthermore, while .tableinfo file is missing, there won't be any IOException
thrown in *getTableDescriptor(FileSystem fs. Path tableDir)*. More details can
be found in the following code.
{code}
public static HTableDescriptor getTableDescriptor(FileSystem fs, Path
tableDir)
throws IOException, NullPointerException {
if (tableDir == null) throw new NullPointerException();
FileStatus status = getTableInfoPath(fs, tableDir);
if (status == null) return null; // return W/O any IOException
...
private static FileStatus getTableInfoPath(final FileSystem fs,
final Path tabledir)
throws IOException {
FileStatus [] status = FSUtils.listStatus(fs, tabledir, new PathFilter() {
@Override
public boolean accept(Path p) {
// Accept any file that starts with TABLEINFO_NAME
return p.getName().startsWith(TABLEINFO_NAME);
}
});
if (status == null || status.length 1) return null; // cannot list
.tableinfo file under tableDir directory.
{code}
For such kind of case, to modify *getTableDescriptor* still cannot solve the
problem here as well. Any idea?
hbck cannot detect any IOException while .tableinfo file is missing
-
Key: HBASE-6516
URL: https://issues.apache.org/jira/browse/HBASE-6516
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Attachments: hbase-6516.patch, hbase-6516-v2.patch
HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos()
function. However, no IoException will be catched while .tableinfo is
missing, since FSTableDescriptors.getTableDescriptor doesn't throw any
IoException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6529) With HFile v2, the region server will always perform an extra copy of source files
[
https://issues.apache.org/jira/browse/HBASE-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432959#comment-13432959
]
Jie Huang commented on HBASE-6529:
--
Hi all, since I cannot find the *submit patch* button here, do we need to
create another entry for this issue? And then, I can re-upload the patch file.
With HFile v2, the region server will always perform an extra copy of source
files
--
Key: HBASE-6529
URL: https://issues.apache.org/jira/browse/HBASE-6529
Project: HBase
Issue Type: Bug
Components: performance, regionserver
Affects Versions: 0.94.0, 0.96.0
Reporter: Jason Dai
Attachments: hbase-6529.diff
With HFile v2 implementation in HBase 0.94 0.96, the region server will use
HFileSystem as its {color:blue}fs{color}. When it performs bulk load in
Store.bulkLoadHFile(), it checks if its {color:blue}fs{color} is the same as
{color:blue}srcFs{color}, which however will be DistributedFileSystem.
Consequently, it will always perform an extra copy of source files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6529) With HFile v2, the region server will always perform an extra copy of source files
[
https://issues.apache.org/jira/browse/HBASE-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6529:
-
Attachment: hbase-6529.diff
Oh~~. That is why bulkload becomes quite slow since 0.94 version, even for
those data storing over the same HDFS cluster. Here attaches the quick fix.
With HFile v2, the region server will always perform an extra copy of source
files
--
Key: HBASE-6529
URL: https://issues.apache.org/jira/browse/HBASE-6529
Project: HBase
Issue Type: Bug
Components: performance, regionserver
Affects Versions: 0.94.0, 0.96.0
Reporter: Jason Dai
Attachments: hbase-6529.diff
With HFile v2 implementation in HBase 0.94 0.96, the region server will use
HFileSystem as its {color:blue}fs{color}. When it performs bulk load in
Store.bulkLoadHFile(), it checks if its {color:blue}fs{color} is the same as
{color:blue}srcFs{color}, which however will be DistributedFileSystem.
Consequently, it will always perform an extra copy of source files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6529) With HFile v2, the region server will always perform an extra copy of source files
[
https://issues.apache.org/jira/browse/HBASE-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431697#comment-13431697
]
Jie Huang commented on HBASE-6529:
--
BTW, all the tests are passed with this patch file. thanks.
With HFile v2, the region server will always perform an extra copy of source
files
--
Key: HBASE-6529
URL: https://issues.apache.org/jira/browse/HBASE-6529
Project: HBase
Issue Type: Bug
Components: performance, regionserver
Affects Versions: 0.94.0, 0.96.0
Reporter: Jason Dai
Attachments: hbase-6529.diff
With HFile v2 implementation in HBase 0.94 0.96, the region server will use
HFileSystem as its {color:blue}fs{color}. When it performs bulk load in
Store.bulkLoadHFile(), it checks if its {color:blue}fs{color} is the same as
{color:blue}srcFs{color}, which however will be DistributedFileSystem.
Consequently, it will always perform an extra copy of source files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
Jie Huang created HBASE-6516: Summary: hbck cannot detect any IOException while .tableinfo file is missing Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6516: - Attachment: hbase-6516.patch Here proposed a possible fix. The basic idea is to check if the HTableDescriptor *htd* is null. If it is null, we'd better to print out the error message and throw an IOException accordingly. Any comment? Thanks. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429654#comment-13429654 ] Jie Huang commented on HBASE-6516: -- Thanks Andrew. I have modified the patch file accordingly. bq. Have you tried running the unit test suite with your patch applied? What is the result? Yes, I have verified all unit tests before uploading the patch file. hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516-v2.patch, hbase-6516.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6516) hbck cannot detect any IOException while .tableinfo file is missing
[ https://issues.apache.org/jira/browse/HBASE-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6516: - Attachment: hbase-6516-v2.patch hbck cannot detect any IOException while .tableinfo file is missing - Key: HBASE-6516 URL: https://issues.apache.org/jira/browse/HBASE-6516 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Attachments: hbase-6516-v2.patch, hbase-6516.patch HBaseFsck checks those missing .tableinfo files in loadHdfsRegionInfos() function. However, no IoException will be catched while .tableinfo is missing, since FSTableDescriptors.getTableDescriptor doesn't throw any IoException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426596#comment-13426596
]
Jie Huang commented on HBASE-6460:
--
bq. When you say ignore, do you mean treat it as a warning as opposed to a
error?
Without that *-fixHdfsOrphans* option, the hbck won't try to fix the problem as
you like.
Regarding the shortcut, from my perspective, we can let -repairHoles to fix
both items at the same time, which is quite acceptable literally. And I wonder
if it is OK to have another shortcut for -fixAssignments -fixMeta, which aim
to solve the meta related parts. And then we can combine any single option
(like -fixHdfsHole) along with those options. What do you think?
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6460:
-
Attachment: (was: hbase-6460.patch)
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6460:
-
Attachment: hbase-6460.patch
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427077#comment-13427077
]
Jie Huang commented on HBASE-6460:
--
@Jimmy @Jonathan, Thanks for your comments. I have fixed the usage info
instead of changing the code. Any more comment?
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427079#comment-13427079 ] Jie Huang commented on HBASE-6464: -- It seems that the problem has already been solved in 0.94 branch. We can close this bug entry. Thanks. reportTableInFlux() throws NPE when no table is returned. - Key: HBASE-6464 URL: https://issues.apache.org/jira/browse/HBASE-6464 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Reporter: Jie Huang Priority: Critical Fix For: 0.94.2 Attachments: hbase-6464.patch reportTableInFlux() gets all tables not in flux. However, when no table is found in getTable(numSkipped) function, it will return null. Then, errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk-v4.patch Thanks. Quite reasonable. Done in the updated version (v4). Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch, hbase-6429-trunk-v4.patch, hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425746#comment-13425746
]
Jie Huang commented on HBASE-6460:
--
OK. I see your point. We can fix this issue to make the help info and the code
implementation consistent here. Regarding that feature, I wonder if we can run
hbck without -fixHdfsOrphans to ignore those orphan regions. Any comment?
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6460) hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425460#comment-13425460
]
Jie Huang commented on HBASE-6460:
--
From my perspective, to change the code is preferable, since both of
-fixHdfsOrphans and -fixHdfsHole try to restore the missing region(s).
It is quite reasonable to enable both of them while using -repairHoles.
hbck -repairHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6245: - Attachment: hbase-6245.patch It is quite convenient to refresh the page and return that previously selected tab. Here attaches a js solution to keep the previously selected tab while refreshing the page. If it is necessary, we can create a new .js file under hbase-webapps/static/js folder. Any idea, please let me know. Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk-v3.patch Thanks Ted. Here attaches the updated version. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch, hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424670#comment-13424670
]
Jie Huang commented on HBASE-6429:
--
TestFromClientSide is passed on my local server.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 170.132 sec
Results :
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3
{noformat}
Filter with filterRow() returning true is incompatible with scan with limit
---
Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch,
hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6464: - Affects Version/s: (was: 0.96.0) reportTableInFlux() throws NPE when no table is returned. - Key: HBASE-6464 URL: https://issues.apache.org/jira/browse/HBASE-6464 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Reporter: Jie Huang Priority: Critical Attachments: hbase-6464.patch reportTableInFlux() gets all tables not in flux. However, when no table is found in getTable(numSkipped) function, it will return null. Then, errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424696#comment-13424696 ] Jie Huang commented on HBASE-6464: -- Yes. you are right. It is the issue for 0.94.0 only. thanks. reportTableInFlux() throws NPE when no table is returned. - Key: HBASE-6464 URL: https://issues.apache.org/jira/browse/HBASE-6464 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Reporter: Jie Huang Priority: Critical Attachments: hbase-6464.patch reportTableInFlux() gets all tables not in flux. However, when no table is found in getTable(numSkipped) function, it will return null. Then, errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6460:
-
Attachment: hbase-6460.patch
A quick and simple fix in the patch file
hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6460) hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
Jie Huang created HBASE-6460:
Summary: hbck -repareHoles shortcut doesn't enable
-fixHdfsOrphans
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
Jie Huang created HBASE-6464: Summary: reportTableInFlux() throws NPE when no table is returned. Key: HBASE-6464 URL: https://issues.apache.org/jira/browse/HBASE-6464 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Critical reportTableInFlux() gets all tables not in flux. However, when no table is found in getTable(numSkipped) function, it will return null. Then, errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[
https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423606#comment-13423606
]
Jie Huang commented on HBASE-6464:
--
See the code implementation
{code}
private void reportTablesInFlux() {
AtomicInteger numSkipped = new AtomicInteger(0);
HTableDescriptor[] allTables = getTables(numSkipped); // allTables ==
null
errors.print(Number of Tables: + allTables.length); // Null pointer
exception here
{code}
We'd better to check if (allTables == null) to avoid that NPE.
reportTableInFlux() throws NPE when no table is returned.
-
Key: HBASE-6464
URL: https://issues.apache.org/jira/browse/HBASE-6464
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Critical
reportTableInFlux() gets all tables not in flux. However, when no table is
found in getTable(numSkipped) function, it will return null. Then,
errorReporter attempts to print the table number, which causes a NPE here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6464: - Attachment: hbase-6464.patch reportTableInFlux() throws NPE when no table is returned. - Key: HBASE-6464 URL: https://issues.apache.org/jira/browse/HBASE-6464 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Critical Attachments: hbase-6464.patch reportTableInFlux() gets all tables not in flux. However, when no table is found in getTable(numSkipped) function, it will return null. Then, errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421303#comment-13421303
]
Jie Huang commented on HBASE-6429:
--
Here is another example using existing filter.
{code}
private static void prepareData() {
try {
HTable table = new HTable(TestFilter.conf, name);
assertTrue(Fail to create the table, admin.tableExists(name));
ListPut puts = new ArrayListPut();
// row1 = f1:c1 , 1_c1, ts=1 , f1:c2 , 1_c2,ts=2, f1:c3 ,
1_c3,ts=3, f1:c4 ,
// 1_c4,ts=4, f1:c5 , 1_c5,ts=5
// row2 = f1:c1 , 2_c1, ts=2 , f1:c2 , 2_c2, ts=2, f1:c3 , 2_c3,
ts=2, f1:c4 ,
// 2_c4, ts=2, f1:c5 , 2_c5, ts=2
// row3 = f1:c1 , 3_c1, ts=3 , f1:c2 , 3_c2, ts=3, f1:c3 ,
3_c3, ts=3, f1:c4 ,
// 3_c4, ts=3, f1:c5 , 3_c5, ts=3
for (int i = 1; i 4; i++) {
Put put = new Put(Bytes.toBytes(row + i));
for (int j = 1; j 6; j++) {
long timestamp = j;
if(i!=1) timestamp = i;
put.add(Bytes.toBytes(f1),
Bytes.toBytes(c + j),
timestamp,
Bytes.toBytes(i + _c + j));
}
puts.add(put);
}
table.put(puts);
table.close();
} catch (IOException e) {
e.printStackTrace();
assertNull(Exception found while putting data into table, e);
}
}
public testFilter() {
int kv_number = 0;
int row_number = 0;
try {
Scan scan = new Scan();
ListFilter fs = new ArrayListFilter();
DependentColumnFilter f1 = new DependentColumnFilter(Bytes.toBytes(f1),
Bytes.toBytes(c5), true, CompareFilter.CompareOp.EQUAL,
new SubstringComparator(c5));
PageFilter f2 = new PageFilter(2);
fs.add(f1);
fs.add(f2);
FilterList filter = new FilterList(fs);
scan.setFilter(filter);
HTable table = new HTable(conf, name);
ResultScanner scanner = table.getScanner(scan);
// row2 (c1-c4) and row3(c1-c4) are returned
for (Result result : scanner) {
row_number++;
for (KeyValue kv : result.list()) {
LOG.debug(kv_number + . kv: + kv);
kv_number++;
assertEquals(Returned row is not correct, new String(kv.getRow()),
row + (row_number+1));
}
}
scanner.close();
table.close();
} catch (Exception e) {
assertNull(Exception happens in scan, e);
}
LOG.debug(check the fetched kv number);
assertEquals(We should get 8 results returned., 8, kv_number);
assertEquals(We should get 2 rows returned, 2, row_number);
}
{code}
Filter with filterRow() returning true is incompatible with scan with limit
---
Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421304#comment-13421304 ] Jie Huang commented on HBASE-6429: -- The above case can show the difference. If it is necessary, we can add the test case later. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk-v2.patch Thanks Ted. I have added that new test case in TestFilterWrapper file, which checks if the FilterWrapper keeps the right semantics. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421107#comment-13421107
]
Jie Huang commented on HBASE-6429:
--
I am afraid we cannot do that. See the following case :
{code}
// pseudocode
void filterRow(ListKeyValue kvs){
if kvs.contains(target_kv) {
kvs.remove(target_kv)
find_target_kv_count++
}
}
// According to the javadoc, it is the Last chance to veto row based on
previous filterKeyValue(KeyValue) calls.
boolean filterRow() {
return true; // kvs is empty finally
}
Case A
void filterRowWrapper(ListKeyValue kvs) {
this.filter.filterRow(kvs) // find_target_kv_count++ has been called here
if ( !kvs.isEmpty() this.filter.filterRow() ) {
kvs.clear(); // kvs is empty here
}
// find_target_kv_count has been changed
}
Case B
void filterRowWrapper(ListKeyValue kvs) {
if ( !kvs.isEmpty() this.filter.filterRow() ) {
kvs.clear(); // kvs is empty here
}
if( !this.filter.filterRow() ) {
this.filter.filterRow(kvs);
}
// find_target_kv_count hasn't been changed
}
{code}
Any question , please let me know.
Filter with filterRow() returning true is incompatible with scan with limit
---
Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421117#comment-13421117 ] Jie Huang commented on HBASE-6429: -- This is just a hypothesis. The find_target_kv_count is a user defined variable in real FilterImpl. The user may need this value to decide if filterRow() returns true or false , like if (find_target_kv_count xxx) return true;. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: (was: hbase-6429-trunk.patch) Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk.patch Thank you very much for your time Ted. Really sorry for my mistakes. All tests are passed with this version according the local results. More comments, please let me know. thanks. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420446#comment-13420446
]
Jie Huang commented on HBASE-6429:
--
Re-run TestMetaReaderEditor locally, PASSED.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.53 sec
Results :
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
{noformat}
Filter with filterRow() returning true is incompatible with scan with limit
---
Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: (was: hbase-6429-trunk.patch) Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk.patch Thank you very much, Ted. Here updates that trunk patch file. More comment, please let me know. Filter with filterRow() returning true is incompatible with scan with limit --- Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is also incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418260#comment-13418260
]
Jie Huang commented on HBASE-6429:
--
Checking the 0.94.0 version, it has the similar problem as well. See the
nextInternal() function in HRegion.java file.
{noformat}
do {
this.storeHeap.next(results, limit - results.size(), metric);
if (limit 0 results.size() == limit) {
if (this.filter != null filter.hasFilterRow()) { check the
compatibility between filterRow(kvs) and scan with limit only
throw new IncompatibleFilterException(
Filter with filterRow(ListKeyValue) incompatible with scan
with limit!);
}
return true; // we are expecting more yes, but also limited to
how many we can return.
}
} while (Bytes.equals(currentRow, nextRow = peekRow()));
{noformat}
According to current implementation, hasFilterRow() function is the only way
to invoke filterRow(kvs), which means that filter has some extra filtering
work on the entire row. Furthermore, the user has another chance to clear that
entire row only if boolean filterRow() returns true. The semantics of those
2 functions , hasFilterRow() and boolean filterRow(), are quite similar. From
my perspective, it would be better to merge two criterion into one.
a) We can simply make hasFilterRow() returning true, if either filterRow() or
filterRow(kvs) has been implemented.
b) To do the result.clear() in the filterRow(kvs) function if filterRow()
returns true.
It not only fixes this issue, but also somehow eliminates some confusion. If
someone wants to do something on the entire row (like clear all or parts of the
row content) in filter, just to return true in hasFilterRow() in their Filter
implementation. And at them same time, the cleaning job will be called in the
filterRow(kvs) function by checking the returned value of boolean
filterRow(), which means all those changings on a row happen in
filterRow(kvs).
Any comment? OR I can try to upload one patch later.
Filter with filterRow() returning true is also incompatible with scan with
limit
Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is also incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418286#comment-13418286 ] Jie Huang commented on HBASE-6429: -- Yes. Filter is an interface to define those necessary APIs in FilterImpl. And FilteBase just implements those APIs with default or blank function block. If the actual FilterImpl (like SingleColumnValueFiler) overrides or really needs those 2 functions, it means some modifications are expected on the entire row. Consequently, we should make hasFilterRow() to be ture. Filter with filterRow() returning true is also incompatible with scan with limit Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is also incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429_0_94_0.patch 1. Add one FilterWrapper to call if (filterRow()) kvs.clear() at the end of filterRow(kvs) function. 2. Make hasFilterRow() return true while some modifications are expected in Filter, such as clearing or updating the row content. Filter with filterRow() returning true is also incompatible with scan with limit Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is also incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418830#comment-13418830 ] Jie Huang commented on HBASE-6429: -- Oops.I will fix those 2 failures and regenerate the patch soon. Thanks Ted. Filter with filterRow() returning true is also incompatible with scan with limit Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is also incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk.patch 1. Prepare a patch against trunk 2. Add one more unit test case (TestFilterWithScanLimits) 3. Fix 2 unit test failures in the previous version. Filter with filterRow() returning true is also incompatible with scan with limit Key: HBASE-6429 URL: https://issues.apache.org/jira/browse/HBASE-6429 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.96.0 Reporter: Jason Dai Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch Currently if we scan with bot limit and a Filter with filterRow(ListKeyValue) implemented, an IncompatibleFilterException will be thrown. The same exception should also be thrown if the filer has its filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6312) Make BlockCache eviction thresholds configurable
[ https://issues.apache.org/jira/browse/HBASE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6312: - Attachment: (was: hbase-6312_v2.patch) Make BlockCache eviction thresholds configurable Key: HBASE-6312 URL: https://issues.apache.org/jira/browse/HBASE-6312 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.94.0 Reporter: Jie Huang Assignee: Jie Huang Priority: Minor Fix For: 0.96.0 Attachments: hbase-6312.patch, hbase-6312_v2.patch Some of our customers found that tuning the BlockCache eviction thresholds made test results different in their test environment. However, those thresholds are not configurable in the current implementation. The only way to change those values is to re-compile the HBase source code. We wonder if it is possible to make them configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6312) Make BlockCache eviction thresholds configurable
[ https://issues.apache.org/jira/browse/HBASE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6312: - Attachment: hbase-6312_v3.patch Yes. You are right. Thanks JD. Update the patch file. Make BlockCache eviction thresholds configurable Key: HBASE-6312 URL: https://issues.apache.org/jira/browse/HBASE-6312 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.94.0 Reporter: Jie Huang Assignee: Jie Huang Priority: Minor Fix For: 0.96.0 Attachments: hbase-6312.patch, hbase-6312_v2.patch, hbase-6312_v3.patch Some of our customers found that tuning the BlockCache eviction thresholds made test results different in their test environment. However, those thresholds are not configurable in the current implementation. The only way to change those values is to re-compile the HBase source code. We wonder if it is possible to make them configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-4364: - Attachment: (was: hbase-4364-0_94_0.patch) Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-4364: - Attachment: hbase-4364_trunk.patch Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: hbase-4364_trunk.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6380) bulkload should update the store.storeSize
[
https://issues.apache.org/jira/browse/HBASE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413592#comment-13413592
]
Jie Huang commented on HBASE-6380:
--
Re-run those 2 test cases locally (on a 64-bit Linux server), Passed.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 172.105 sec
Results :
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3
---
T E S T S
---
Running org.apache.hadoop.hbase.master.TestSplitLogManager
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.548 sec
Results :
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
{noformat}
bulkload should update the store.storeSize
--
Key: HBASE-6380
URL: https://issues.apache.org/jira/browse/HBASE-6380
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
Attachments: 6380-trunk.txt, 6380-trunk.txt, hbase-6380_0_94_0.patch
After bulkloading some HFiles into the Table, we found the force-split didn't
work because of the MidKey == NULL. Only if we re-booted the HBase service,
the force-split can work normally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6380) bulkload should update the store.storeSize
Jie Huang created HBASE-6380: Summary: bulkload should update the store.storeSize Key: HBASE-6380 URL: https://issues.apache.org/jira/browse/HBASE-6380 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Critical After bulkloading some HFiles into the Table, we found the force-split didn't work because of the MidKey == NULL. Only if we re-booted the HBase service, the force-split can work normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6380) bulkload should update the store.storeSize
[
https://issues.apache.org/jira/browse/HBASE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412725#comment-13412725
]
Jie Huang commented on HBASE-6380:
--
We found that action of force-split on certain table (after bulkloading)
failed, and we can see following debug log
{noformat}
DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Region
t1,,1342088772367.767829f873354d78a742084c22f47895. not splittable because
midkey=null
{noformat}
But actually we can get the non-empty midkey value in getSplitPoint() function.
The only reason is that the storeSize and largestStoreSize are zero, which
makes the splitPointFromLargetStore returned with NULL (its initial value).
{noformat}
byte[] splitPointFromLargestStore = null;
long largestStoreSize = 0;
for (Store s : stores.values()) {
byte[] splitPoint = s.getSplitPoint();
long storeSize = s.getSize();
if (splitPoint != null largestStoreSize storeSize) {
splitPointFromLargestStore = splitPoint;
largestStoreSize = storeSize;
}
}
return splitPointFromLargestStore;
{noformat}
The simple solution is to update the store's size in the bulkload() function.
After applied attached patch, the problem is solved.
bulkload should update the store.storeSize
--
Key: HBASE-6380
URL: https://issues.apache.org/jira/browse/HBASE-6380
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Critical
Attachments: hbase-6380_0_94_0.patch
After bulkloading some HFiles into the Table, we found the force-split didn't
work because of the MidKey == NULL. Only if we re-booted the HBase service,
the force-split can work normally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6380) bulkload should update the store.storeSize
[ https://issues.apache.org/jira/browse/HBASE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6380: - Attachment: hbase-6380_0_94_0.patch bulkload should update the store.storeSize -- Key: HBASE-6380 URL: https://issues.apache.org/jira/browse/HBASE-6380 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Critical Attachments: hbase-6380_0_94_0.patch After bulkloading some HFiles into the Table, we found the force-split didn't work because of the MidKey == NULL. Only if we re-booted the HBase service, the force-split can work normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6380) bulkload should update the store.storeSize
[ https://issues.apache.org/jira/browse/HBASE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412735#comment-13412735 ] Jie Huang commented on HBASE-6380: -- Thank you very much Ted. I am running the unittest locally, it seems no failure is introduced by this patch. bulkload should update the store.storeSize -- Key: HBASE-6380 URL: https://issues.apache.org/jira/browse/HBASE-6380 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Critical Attachments: 6380-trunk.txt, hbase-6380_0_94_0.patch After bulkloading some HFiles into the Table, we found the force-split didn't work because of the MidKey == NULL. Only if we re-booted the HBase service, the force-split can work normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
