date:20120722

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesse Yates updated HBASE-5547:
---

Attachment: java_HBASE-5547.addendum

Attaching addendum that updates the test (removes the TTL cleaner to avoid
timing issues) and fixes a couple of comments. Looks like a flapper given that
build #3159 worked.

Couldn't get the NPE to reproduce (didn't seem fatal though and was from a log
statement).

Ran the updated test 20x without failure.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch,
java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch,
java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

This came up in a discussion I had with Stack.
It would be nice if HBase could be notified that a backup is in progress (via
a znode for example) and in that case either:
1. rename HFiles to be delete to file.bck
2. rename the HFiles into a special directory
3. rename them to a general trash directory (which would not need to be tied
to backup mode).
That way it should be able to get a consistent backup based on HFiles (HDFS
snapshots or hard links would be better options here, but we do not have
those).
#1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420114#comment-13420114
 ] 

Hadoop QA commented on HBASE-5659:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12537477/5659.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The applied patch generated 5 javac compiler warnings (more than 
the trunk's current 4 warnings).

-1 findbugs.  The patch appears to introduce 14 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestSplitLogManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//console

This message is automatically generated.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420118#comment-13420118
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537478/java_HBASE-5547.addendum
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently


[ 
https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420158#comment-13420158
 ] 

Hudson commented on HBASE-6406:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #103 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/103/])
HBASE-6406 Disable TestZooKeeper#testClientSessionExpired (Revision 1364204)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java


 TestReplicationPeer.testResetZooKeeperSession and 
 TestZooKeeper.testClientSessionExpired fail frequently
 

 Key: HBASE-6406
 URL: https://issues.apache.org/jira/browse/HBASE-6406
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.1
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.96.0, 0.94.2

 Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack


 Looking back through the 0.94 test runs these two tests accounted for 11 of 
 34 failed tests.
 They should be fixed or (temporarily) disabled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420159#comment-13420159
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #103 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/103/])
HBASE-5547 Don't delete HFiles when in backup mode (Jesse Yates) 
(Revision 1364203)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
*

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420187#comment-13420187
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

For the NPE, I think the null reference should have come from below code 
snippet:
{code}
(sf == null ? none : sf.getPath().getName()) +
, size= + (sf == null ? none :
  StringUtils.humanReadableInt(sf.getReader().length()))
{code}
Meaning either sf.getPath() or sf.getReader() was null.

For the addendum, if TimeToLiveHFileCleaner is removed from 
TestZooKeeperTableArchiveClient, we should add test for it since it is a new 
class introduced by this feature.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420226#comment-13420226
]

Jesse Yates commented on HBASE-5547:

{quote}
Meaning either sf.getPath() or sf.getReader() was null.
{quote}

Looked through the code and can't say I see that was possible (though it seems
the most likely given the setting/checking done on the store file). Looking
through the test of the code, it doesn't look like an issue if the reader is
null since it either gets closed immediately or is re-created when needed.

The getPath() call could be bad, but its not doing anything funky and we would
have seen a warning/error earlier in the run if that had actually been a
problem.

{quote}
For the addendum, if TimeToLiveHFileCleaner is removed from
TestZooKeeperTableArchiveClient, we should add test for it since it is a new
class introduced by this feature.
{quote}

It is tested in o.a.h.h.master.cleaner.TestHFileCleaner. This change just
cleans up the test so we don't have to worry about interaction with the
TTLCleaner.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Created] (HBASE-6439) Ignore .archive directory as a table

Jesse Yates created HBASE-6439:
--

 Summary: Ignore .archive directory as a table
 Key: HBASE-6439
 URL: https://issues.apache.org/jira/browse/HBASE-6439
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.96.0
Reporter: Jesse Yates


From a recent test run:
{quote}
2012-07-22 02:27:30,699 WARN  [IPC Server handler 0 on 47087] 
util.FSTableDescriptors(168): The following folder is in HBase's root directory 
and doesn't contain a table descriptor, do consider deleting it: .archive
{quote}

With the addition of HBASE-5547, table-level folders are no-longer all table 
folders. FSTableDescriptors needs to then have a 'gold-list' that we can update 
with directories that aren't tables so we don't have this kind of thing showing 
up in the logs.

Currently, we have the following block:
{quote}
invocations++;
if (HTableDescriptor.ROOT_TABLEDESC.getNameAsString().equals(tablename)) {
  cachehits++;
  return HTableDescriptor.ROOT_TABLEDESC;
}
if (HTableDescriptor.META_TABLEDESC.getNameAsString().equals(tablename)) {
  cachehits++;
  return HTableDescriptor.META_TABLEDESC;
}
{quote}

to handle special cases, but that's a bit clunky and not clean in terms of 
table-level directories that need to be ignored.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420233#comment-13420233
]

Zhihong Ted Yu commented on HBASE-5547:
---

bq. we don't have to worry about interaction with the TTLCleaner.
Users may choose to enable the combination of cleaners shown in the current
TestZooKeeperTableArchiveClient.
We should verify that this combination works.

From
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-2.0.0/lastCompletedBuild/testReport/org.apache.hadoop.hbase.backup.example/TestZooKeeperTableArchiveClient/testMultipleTables/,
there is no NPE.

I used the following command to run the test:

mvn clean -Dhadoop.profile=2.0 test
-Dtest=TestZooKeeperTableArchiveClient#testMultipleTables
{code}
Failed tests:
testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient):
Archived HFiles should have gotten deleted, but didn't
{code}
The failure was due to IllegalArgumentException shown above.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420235#comment-13420235
]

Jesse Yates commented on HBASE-5547:

bq. The failure was due to IllegalArgumentException shown above.

Yeah, it seems to be breaking all the tests. Its really odd that HDFS goes to
the DistributedFileSystem to do the listStatus when its a local filesystem -
seems wrong.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420237#comment-13420237
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

Could this be related to HBASE-6403 ?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420250#comment-13420250
 ] 

Jesse Yates commented on HBASE-5547:


bq. Could this be related to HBASE-6403 ?

Similar, but not exactly the same. In short, the FS has a configuration, but 
that configuration isn't correctly specifying the filesystem (from the 
MasterFileSystem). Working on a fix now.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420266#comment-13420266
]

Jesse Yates commented on HBASE-5547:

{quote}
Users may choose to enable the combination of cleaners shown in the current
TestZooKeeperTableArchiveClient.
We should verify that this combination works.
{quote}

I'd argue that we have verified by inspection. Using the time-based waiting on
the cluster tends to lead to flapping tests (e.g. the original error), which is
bad. Because the cleaner chain works, the TTL cleaner works (both of which are
tested) and the zk archiver cleaner works, we can then say that they work
together.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesse Yates updated HBASE-5547:
---

Attachment: java_HBASE-5547.addendum-v1

Attaching updated version. Tested all the failing tests from the hadoop-2.0
(which is a proper superset of the failures from the hadoop-1.0 run) run on
both versions of hadoop and passed locally.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch,
java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
java_HBASE-5547_v7.patch

[jira] [Created] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.

Jimmy Xiang created HBASE-6440:
--

 Summary: SplitLogManager - log the exception when failed to move 
recovered edits.
 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang


We should log the exception itself too:

{noformat}
try {
  HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
} catch (IOException e) {
  LOG.warn(Could not finish splitting of log file  + logfile);
  return Status.ERR;
}
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420269#comment-13420269
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537517/java_HBASE-5547.addendum-v1
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestCatalogJanitor

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420271#comment-13420271
]

Lars Hofhansl commented on HBASE-5547:
--

Addendum v1 looks good to me. I also agree that we should remove the TTL
cleaner when the ZK cleaner is tested.
Might be prudent to add an integration test that uses both in some manner.

One question looking at the addendum: When would the passed configuration to
archiveRegion be different from fs.getConfiguration()? It seems that should not
happen; if it did other confusion might arise.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420272#comment-13420272
]

Jesse Yates commented on HBASE-5547:

{quote}
When would the passed configuration to archiveRegion be different from
fs.getConfiguration()? It seems that should not happen; if it did other
confusion might arise.
{quote}

The reason the error was happening was the conf in fs.getConf() didn't have the
fs set correctly, so it was going to the local filesystem, but the FS we were
deleting from was a DistributedFileSystem. The passed in conf is now from an
'authorative' source (the HMaster).

Didn't run the catalogjanitor test...working on fix.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness

[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420274#comment-13420274
]

Lars Hofhansl commented on HBASE-5121:
--

@chunhiu: Could have a look at the suggested change in HBASE-5659?
The observation is that:
# after the scanner is reset the top KV will in the vast majority of all case
be changed
# that is only a problem when the top KV's row has changed, in many case
StoreScanner.next indicates more work than necessary to a caller.

MajorCompaction may affect scan's correctness
-

Key: HBASE-5121
URL: https://issues.apache.org/jira/browse/HBASE-5121
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.1, 0.94.0

Attachments: 5121-0.92.txt, 5121-suggest.txt,
5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch,
hbase-5121.patch, hbase-5121v2.patch

In our test, there are two families' keyvalue for one row.
But we could find a infrequent problem when doing scan's next if
majorCompaction happens concurrently.
In the client's two continuous doing scan.next():
1.First time, scan's next returns the result where family A is null.
2.Second time, scan's next returns the result where family B is null.
The two next()'s result have the same row.
If there are more families, I think the scenario will be more strange...
We find the reason is that storescanner.peek() is changed after
majorCompaction if there are delete type KeyValue.
This change causes the PriorityQueueKeyValueScanner of RegionScanner's heap
is not sure to be sorted.

[jira] [Comment Edited] (HBASE-5121) MajorCompaction may affect scan's correctness

[
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420274#comment-13420274
]

Lars Hofhansl edited comment on HBASE-5121 at 7/22/12 7:07 PM:
---

@chunhui: Could have a look at the suggested change in HBASE-5659?
The observation is that:
# after the scanner is reset the top KV will in the vast majority of all case
be changed
# that is only a problem when the top KV's row has changed, in many case
StoreScanner.next indicates more work than necessary to a caller.

Edit: Misspelled chunhui's name...

was (Author: lhofhansl):
@chunhiu: Could have a look at the suggested change in HBASE-5659?
The observation is that:
# after the scanner is reset the top KV will in the vast majority of all case
be changed
# that is only a problem when the top KV's row has changed, in many case
StoreScanner.next indicates more work than necessary to a caller.

MajorCompaction may affect scan's correctness
-

Attachments: 5121-0.92.txt, 5121-suggest.txt,
5121-trunk-combined.txt, 5121.90, hbase-5121-testcase.patch,
hbase-5121.patch, hbase-5121v2.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420275#comment-13420275
 ] 

Lars Hofhansl commented on HBASE-5547:
--

I see... Seems something else is wrong then, I think. Don't have to fix it 
here, just something to keep in mind.


 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6433) Improve HBaseServer#getRemoteAddress by utilizing HBaseServer.Connection.hostAddress


 [ 
https://issues.apache.org/jira/browse/HBASE-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6433:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to 0.94.

 Improve HBaseServer#getRemoteAddress by utilizing 
 HBaseServer.Connection.hostAddress
 

 Key: HBASE-6433
 URL: https://issues.apache.org/jira/browse/HBASE-6433
 Project: HBase
  Issue Type: Improvement
Reporter: binlijin
Assignee: binlijin
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 6433-getRemoteAddress-trunk.txt, HBASE-6433-90.patch, 
 HBASE-6433-92.patch, HBASE-6433-94.patch, HBASE-6433-trunk.patch


 Currently, HBaseServer#getRemoteAddress would call getRemoteIp(), leading to 
 call.connection.socket.getInetAddress().
 The host address is actually stored in HBaseServer.Connection.hostAddress 
 field. We don't need to go through Socket to get this information.
 Without this patch it costs 4000ns, with this patch it costs 1600ns

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420279#comment-13420279
]

Jesse Yates commented on HBASE-5547:

I'd rather have the fix in, then revert as necessary when we find the root
cause (just in case this is 'expected' behavior. Particularly given the
occasional oddness between hdfs-1.X and 2.x., hack it to fix the build and then
fix it 'right' later seems ok.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Updated] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently


 [ 
https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6406:
-

Fix Version/s: (was: 0.94.2)
   0.94.1

 TestReplicationPeer.testResetZooKeeperSession and 
 TestZooKeeper.testClientSessionExpired fail frequently
 

 Key: HBASE-6406
 URL: https://issues.apache.org/jira/browse/HBASE-6406
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.1
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.96.0, 0.94.1

 Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack


 Looking back through the 0.94 test runs these two tests accounted for 11 of 
 34 failed tests.
 They should be fixed or (temporarily) disabled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420280#comment-13420280
 ] 

Jesse Yates commented on HBASE-5547:


oh, wait, spent the two minutes to RTFS and realized that, when calling 
path#getFileSystem(conf), if there is no scheme for the path (an unqualified 
path), it tries to make a filesystem object from the configuration using the 
default URI, which if it isn't set, uses file:/// (see FileSystem.java:131, 
hadoop-1.0.3).

Guess we need to make sure that the fileSystem sets the default URI for its 
configuration. 

Should I file another ticket or just roll the fix in here?



 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420281#comment-13420281
 ] 

Lars Hofhansl commented on HBASE-5659:
--

The patch is also a slight performance improvement. HBASE-5121 is too eager I 
think. The tests introduced with HBASE-5121 still pass with this change.


 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420284#comment-13420284
]

Lars Hofhansl commented on HBASE-5547:
--

Yeah, separate ticket.
Should we commit v1 as is now to get most of the tests working again and then
one last addendum to fix the catalog janitor test?

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420285#comment-13420285
 ] 

Jesse Yates commented on HBASE-5547:


The catalog janitor is just one more line. I'll post a new addendum momentarily.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420293#comment-13420293
]

Zhihong Ted Yu commented on HBASE-5547:
---

bq. Because the cleaner chain works, the TTL cleaner works (both of which are
tested) and the zk archiver cleaner works, we can then say that they work
together.
If some user encounters the scenario shown in existing
TestZooKeeperTableArchiveClient#testMultipleTables and asked us on mailing
list, can we go with the above answer ?

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420296#comment-13420296
]

Jesse Yates commented on HBASE-5547:

@Ted: yeah. In the worst case, the data is kept around for an extra ttl
(because that's how the cleaner works), so at worst they have the reference
file for another 5-10 mins (depending on defaults).

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420305#comment-13420305
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

If I keep the cleaner in addendum v1, testMultipleTables failed at the 
following assertion:
{code}
assertFalse(fs.exists(otherStoreArchive));
{code}

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420310#comment-13420310
 ] 

Jesse Yates commented on HBASE-5547:


@Ted yes, the sleeps for the ttl cleaner were also removed, which probably 
caused the error. Give me a little bit and I'll post an addendum that:

* removes the TLL cleaner from the primary ZKTableArchiver test
* adds a test that just tests the interaction between the ZKTableArchiver and 
the TTLCleaner
* fixes the fs scheme for all tests (general things like avro server and the 
catalogjanitor)

And I'll file another ticket to fix the fs configuration/rootdir stuff from the 
master.

Did I miss anything?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6433) Improve HBaseServer#getRemoteAddress by utilizing HBaseServer.Connection.hostAddress


[ 
https://issues.apache.org/jira/browse/HBASE-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420311#comment-13420311
 ] 

Hudson commented on HBASE-6433:
---

Integrated in HBase-0.94 #352 (See 
[https://builds.apache.org/job/HBase-0.94/352/])
HBASE-6433 Improve HBaseServer#getRemoteAddress by utilizing 
HBaseServer.Connection.hostAddress n(binlijin) (Revision 1364397)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java


 Improve HBaseServer#getRemoteAddress by utilizing 
 HBaseServer.Connection.hostAddress
 

 Key: HBASE-6433
 URL: https://issues.apache.org/jira/browse/HBASE-6433
 Project: HBase
  Issue Type: Improvement
Reporter: binlijin
Assignee: binlijin
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 6433-getRemoteAddress-trunk.txt, HBASE-6433-90.patch, 
 HBASE-6433-92.patch, HBASE-6433-94.patch, HBASE-6433-trunk.patch


 Currently, HBaseServer#getRemoteAddress would call getRemoteIp(), leading to 
 call.connection.socket.getInetAddress().
 The host address is actually stored in HBaseServer.Connection.hostAddress 
 field. We don't need to go through Socket to get this information.
 Without this patch it costs 4000ns, with this patch it costs 1600ns

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420312#comment-13420312
]

Lars Hofhansl commented on HBASE-5547:
--

@Ted: The two cleaners should be tested in isolation. That's what unit tests
are all about.

We can have an additional test that verifies that the cleaners work together
(which just means that if one cleaner decides the file needs to be kept it will
be kept around). Let's do that in a separate jira.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Updated] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.


 [ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6440:
---

Attachment: hbase-6440.patch

 SplitLogManager - log the exception when failed to move recovered edits.
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.


 [ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang reassigned HBASE-6440:
--

Assignee: Jimmy Xiang

 SplitLogManager - log the exception when failed to move recovered edits.
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.


 [ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6440:
---

Status: Patch Available  (was: Open)

 SplitLogManager - log the exception when failed to move recovered edits.
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420314#comment-13420314
 ] 

Lars Hofhansl commented on HBASE-5659:
--

In the latest failed runs I see this:
{quote}
org.apache.hadoop.hbase.DroppedSnapshotException: region: 
testtable,,1342893920897.74c4711d8d09b8d84618388ce125cedd.at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1474)
at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1353)
at 
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1294)
at 
org.apache.hadoop.hbase.regionserver.TestAtomicOperation$1.run(TestAtomicOperation.java:278)
Caused by: java.io.FileNotFoundException: 
/home/hudson/hudson-slave/workspace/HBase-0.94/trunk/target/test-data/bca87d6a-50fb-4cdc-9e8c-72912928f4ed/TestAtomicOperationtestRowMutationMultiThreads/testtable/74c4711d8d09b8d84618388ce125cedd/.tmp/62b35712575041f28c04f6f953830b80
 (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at 
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.init(RawLocalFileSystem.java:188)
at 
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.init(RawLocalFileSystem.java:184)
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:255)
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:236)
at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:335)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:151)
at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:126)
at 
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.createOutputStream(AbstractHFileWriter.java:271)
at 
org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.init(StoreFile.java:956)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.init(StoreFile.java:901)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:815)
at 
org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:836)
at 
org.apache.hadoop.hbase.regionserver.Store.createWriterInTmp(Store.java:816)
at 
org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:717)
at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:674)
at org.apache.hadoop.hbase.regionserver.Store.access$400(Store.java:108)
at 
org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:2247)
at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1449)
... 3 more
{quote}


 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420316#comment-13420316
 ] 

Lars Hofhansl commented on HBASE-5659:
--

In fact all recent runs with this test failed show this exception.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
  entries=12, sequenceid=7934, filesize=1.3k
 2012-03-27 04:36:12,642 DEBUG

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420320#comment-13420320
 ] 

Lars Hofhansl commented on HBASE-5659:
--

So then here's a new theory: The produces  1000 flushes. Compactions can't 
keep up, hence a scan will need to merge all these files together. The limit 
for the test is 1024 open files.
I would still like to include the first patch.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Comment Edited] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420320#comment-13420320
 ] 

Lars Hofhansl edited comment on HBASE-5659 at 7/22/12 8:29 PM:
---

So then here's a new theory: The test produces  1000 flushes. Compactions 
can't keep up, hence a scan will need to merge all these files together. The 
limit for the test is 1024 open files.
I would still like to include the first patch.

  was (Author: lhofhansl):
So then here's a new theory: The produces  1000 flushes. Compactions can't 
keep up, hence a scan will need to merge all these files together. The limit 
for the test is 1024 open files.
I would still like to include the first patch.
  
 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to

[jira] [Updated] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


 [ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5659:
-

Attachment: 5659-v2.txt

Patch that only compares rows when top KV is changed and also reduces the 
number of ops/thread to 500 in testMultiRowMutationMultiThreads (same value now 
as in testRowMutationMultiThreads).

I will commit this soon, if no objections.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Created] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem

Jesse Yates created HBASE-6441:
--

 Summary: MasterFS doesn't set scheme for internal FileSystem
 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0


FSUtils.getRootDir() just takes a configuration object, which is used to:
1) Get the name of the root directory
2) Create a filesystem (based on the configured scheme)
3) Qualify the root onto the filesystem

However, the FileSystem from the master filesystem won't generate the correctly 
qualified root directory under hadoop-2.0 (though it works fine on hadoop-1.0). 
Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


 [ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-6441:
---

Attachment: java_HBASE-6441_v0.patch

Solves problem in HBASE-5547, passes unit test for hadoop-1.0 and 2.0

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


 [ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-6441:
---

Status: Patch Available  (was: Open)

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420329#comment-13420329
]

Jesse Yates commented on HBASE-5547:

Attached fix for the fs scheme to HBASE-6441. Seems to fix the hadoop-2.0 test
problem. Working on simpler version of addendum.v1 that just handles the
multipleTests issue and formatting.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.

[
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420332#comment-13420332
]

Hadoop QA commented on HBASE-6440:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537518/hbase-6440.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2426//console

This message is automatically generated.

SplitLogManager - log the exception when failed to move recovered edits.

Key: HBASE-6440
URL: https://issues.apache.org/jira/browse/HBASE-6440
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-6440.patch

We should log the exception itself too:
{noformat}
try {
HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
} catch (IOException e) {
LOG.warn(Could not finish splitting of log file + logfile);
return Status.ERR;
}
{noformat}

[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesse Yates updated HBASE-5547:
---

Attachment: java_HBASE-5547.addendum-v2

Simpler version of v2 that just removes the TTL cleaner from the
ZKARchiveClient test. Requires HBASE-6441 to fix all the broken tests on this
ticket (and might not apply cleanly without 6441, though I haven't tried).

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547.addendum-v1, java_HBASE-5547.addendum-v2,
java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch,
java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch,
java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-6440) SplitLogManager - log the exception when failed to move recovered edits.


[ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420334#comment-13420334
 ] 

stack commented on HBASE-6440:
--

+1 on patch.  This diff could not have caused the above fails.

 SplitLogManager - log the exception when failed to move recovered edits.
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420335#comment-13420335
 ] 

stack commented on HBASE-6441:
--

+1 on patch.  Anyone up on hadoop 2.0 want to take a look too?

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420336#comment-13420336
 ] 

stack commented on HBASE-6441:
--

Else will commit in next day or so unless I'm beaten to it.

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420337#comment-13420337
 ] 

stack commented on HBASE-5659:
--

Patch seems fine to me.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
  entries=12, sequenceid=7934, filesize=1.3k
 2012-03-27 04:36:12,642 DEBUG [Thread-118]

[jira] [Commented] (HBASE-6439) Ignore .archive directory as a table


[ 
https://issues.apache.org/jira/browse/HBASE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420340#comment-13420340
 ] 

stack commented on HBASE-6439:
--

Add it to HConstants#HBASE_NON_USER_TABLE_DIRS


 Ignore .archive directory as a table
 

 Key: HBASE-6439
 URL: https://issues.apache.org/jira/browse/HBASE-6439
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.96.0
Reporter: Jesse Yates
  Labels: newbie

 From a recent test run:
 {quote}
 2012-07-22 02:27:30,699 WARN  [IPC Server handler 0 on 47087] 
 util.FSTableDescriptors(168): The following folder is in HBase's root 
 directory and doesn't contain a table descriptor, do consider deleting it: 
 .archive
 {quote}
 With the addition of HBASE-5547, table-level folders are no-longer all table 
 folders. FSTableDescriptors needs to then have a 'gold-list' that we can 
 update with directories that aren't tables so we don't have this kind of 
 thing showing up in the logs.
 Currently, we have the following block:
 {quote}
 invocations++;
 if (HTableDescriptor.ROOT_TABLEDESC.getNameAsString().equals(tablename)) {
   cachehits++;
   return HTableDescriptor.ROOT_TABLEDESC;
 }
 if (HTableDescriptor.META_TABLEDESC.getNameAsString().equals(tablename)) {
   cachehits++;
   return HTableDescriptor.META_TABLEDESC;
 }
 {quote}
 to handle special cases, but that's a bit clunky and not clean in terms of 
 table-level directories that need to be ignored.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


 [ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6440:
---

Summary: SplitLogManager - log the exception when failed to finish split 
log file  (was: SplitLogManager - log the exception when failed to move 
recovered edits.)

Change the title to match the code better.

 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6439) Ignore .archive directory as a table


[ 
https://issues.apache.org/jira/browse/HBASE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420342#comment-13420342
 ] 

Jesse Yates commented on HBASE-6439:


Yeah, but its technically configurable, so that's not so nice...

 Ignore .archive directory as a table
 

 Key: HBASE-6439
 URL: https://issues.apache.org/jira/browse/HBASE-6439
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.96.0
Reporter: Jesse Yates
  Labels: newbie

 From a recent test run:
 {quote}
 2012-07-22 02:27:30,699 WARN  [IPC Server handler 0 on 47087] 
 util.FSTableDescriptors(168): The following folder is in HBase's root 
 directory and doesn't contain a table descriptor, do consider deleting it: 
 .archive
 {quote}
 With the addition of HBASE-5547, table-level folders are no-longer all table 
 folders. FSTableDescriptors needs to then have a 'gold-list' that we can 
 update with directories that aren't tables so we don't have this kind of 
 thing showing up in the logs.
 Currently, we have the following block:
 {quote}
 invocations++;
 if (HTableDescriptor.ROOT_TABLEDESC.getNameAsString().equals(tablename)) {
   cachehits++;
   return HTableDescriptor.ROOT_TABLEDESC;
 }
 if (HTableDescriptor.META_TABLEDESC.getNameAsString().equals(tablename)) {
   cachehits++;
   return HTableDescriptor.META_TABLEDESC;
 }
 {quote}
 to handle special cases, but that's a bit clunky and not clean in terms of 
 table-level directories that need to be ignored.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420343#comment-13420343
 ] 

Hadoop QA commented on HBASE-5659:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12537521/5659-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The applied patch generated 5 javac compiler warnings (more than 
the trunk's current 4 warnings).

-1 findbugs.  The patch appears to introduce 14 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2427//console

This message is automatically generated.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126]

[jira] [Comment Edited] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


[ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420344#comment-13420344
 ] 

Jimmy Xiang edited comment on HBASE-6440 at 7/22/12 9:41 PM:
-

Thank Stack for review.  Integrated into trunk and 0.94.1

  was (Author: jxiang):
Thank Stack for review.  Integrate into trunk and 0.94
  
 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


 [ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6440:
---

   Resolution: Fixed
Fix Version/s: 0.94.1
   0.96.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thank Stack for review.  Integrate into trunk and 0.94

 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes

2012-07-22 Thread Todd Lipcon (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420346#comment-13420346
 ] 

Todd Lipcon commented on HBASE-6435:


Good points. We should probably move this discussion over to an HDFS JIRA. 
Having a global DFSClient-wide ability to mark nodes un-preferred is probably 
advantageous.

 Reading WAL files after a recovery leads to time lost in HDFS timeouts when 
 using dead datanodes
 

 Key: HBASE-6435
 URL: https://issues.apache.org/jira/browse/HBASE-6435
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 6435.unfinished.patch


 HBase writes a Write-Ahead-Log to revover from hardware failure.
 This log is written with 'append' on hdfs.
 Through ZooKeeper, HBase gets informed usually in 30s that it should start 
 the recovery process. 
 This means reading the Write-Ahead-Log to replay the edits on the other 
 servers.
 In standards deployments, HBase process (regionserver) are deployed on the 
 same box as the datanodes.
 It means that when the box stops, we've actually lost one of the edits, as we 
 lost both the regionserver and the datanode.
 As HDFS marks a node as dead after ~10 minutes, it appears as available when 
 we try to read the blocks to recover. As such, we are delaying the recovery 
 process by 60 seconds as the read will usually fail with a socket timeout. If 
 the file is still opened for writing, it adds an extra 20s + a risk of losing 
 edits if we connect with ipc to the dead DN.
 Possible solutions are:
 - shorter dead datanodes detection by the NN. Requires a NN code change.
 - better dead datanodes management in DFSClient. Requires a DFS code change.
 - NN customisation to write the WAL files on another DN instead of the local 
 one.
 - reordering the blocks returned by the NN on the client side to put the 
 blocks on the same DN as the dead RS at the end of the priority queue. 
 Requires a DFS code change or a kind of workaround.
 The solution retained is the last one. Compared to what was discussed on the 
 mailing list, the proposed patch will not modify HDFS source code but adds a 
 proxy. This for two reasons:
 - Some HDFS functions managing block orders are static 
 (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would 
 require to implement partially the fix, change the DFS interface to make this 
 function non static, or put the hook static. None of these solution is very 
 clean. 
 - Adding a proxy allows to put all the code in HBase, simplifying dependency 
 management.
 Nevertheless, it would be better to have this in HDFS. But this solution 
 allows to target the last version only, and this could allow minimal 
 interface changes such as non static methods.
 Moreover, writing the blocks to the non local DN would be an even better 
 solution long term.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6442) Add test for archiving/cleaning across multiple tables where some tables are retained and others aren't

Zhihong Ted Yu created HBASE-6442:
-

 Summary: Add test for archiving/cleaning across multiple tables 
where some tables are retained and others aren't
 Key: HBASE-6442
 URL: https://issues.apache.org/jira/browse/HBASE-6442
 Project: HBase
  Issue Type: Sub-task
Reporter: Zhihong Ted Yu


This task is continuation of previous work in 
TestZooKeeperTableArchiveClient#testMultipleTables

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase


[ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420348#comment-13420348
 ] 

stack commented on HBASE-5954:
--

bq. Would be nice to be able to enable this (when using Hadoop-2).

Can we do it in the hadoop2 compatibility module?

I think its fine its optionally off in 0.94 and that in 0.96 its on by default.

I like your list.  Would think a CF/Table option and the HTable#hsync the more 
important options to offer (though on the latter, perhaps a Put+hsync would be 
better given extra rpc).



 Allow proper fsync support for HBase
 

 Key: HBASE-5954
 URL: https://issues.apache.org/jira/browse/HBASE-5954
 Project: HBase
  Issue Type: Improvement
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.96.0, 0.94.2

 Attachments: 5954-trunk-hdfs-trunk-v2.txt, 
 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 
 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 
 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420349#comment-13420349
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537526/java_HBASE-5547.addendum-v2
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547.addendum-v1, java_HBASE-5547.addendum-v2,
java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch,
java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch,
java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420350#comment-13420350
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

From https://builds.apache.org/job/PreCommit-HBASE-Build/2429//testReport/:
{code}
Running org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 238.611 sec
{code}
I think TestZooKeeperTableArchiveClient should be categorized as large test.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547.addendum-v2, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem

[
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420351#comment-13420351
]

Hadoop QA commented on HBASE-6441:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537523/java_HBASE-6441_v0.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient

org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2428//console

This message is automatically generated.

MasterFS doesn't set scheme for internal FileSystem
---

Key: HBASE-6441
URL: https://issues.apache.org/jira/browse/HBASE-6441
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0

Attachments: java_HBASE-6441_v0.patch

FSUtils.getRootDir() just takes a configuration object, which is used to:
1) Get the name of the root directory
2) Create a filesystem (based on the configured scheme)
3) Qualify the root onto the filesystem
However, the FileSystem from the master filesystem won't generate the
correctly qualified root directory under hadoop-2.0 (though it works fine on
hadoop-1.0). Seems to be an issue with the configuration parameters.

[jira] [Commented] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


[ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420355#comment-13420355
 ] 

Hudson commented on HBASE-6440:
---

Integrated in HBase-TRUNK #3161 (See 
[https://builds.apache.org/job/HBase-TRUNK/3161/])
HBASE-6440 SplitLogManager - log the exception when failed to finish split 
log file (Revision 1364435)

 Result = SUCCESS
jxiang : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java


 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


[ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420360#comment-13420360
 ] 

Hudson commented on HBASE-6440:
---

Integrated in HBase-0.94 #354 (See 
[https://builds.apache.org/job/HBase-0.94/354/])
HBASE-6440 SplitLogManager - log the exception when failed to finish split 
log file (Revision 1364436)

 Result = FAILURE
jxiang : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java


 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6440) SplitLogManager - log the exception when failed to finish split log file


[ 
https://issues.apache.org/jira/browse/HBASE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420378#comment-13420378
 ] 

Hudson commented on HBASE-6440:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #105 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/105/])
HBASE-6440 SplitLogManager - log the exception when failed to finish split 
log file (Revision 1364435)

 Result = FAILURE
jxiang : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java


 SplitLogManager - log the exception when failed to finish split log file
 

 Key: HBASE-6440
 URL: https://issues.apache.org/jira/browse/HBASE-6440
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: hbase-6440.patch


 We should log the exception itself too:
 {noformat}
 try {
   HLogSplitter.moveRecoveredEditsFromTemp(tmpname, logfile, conf);
 } catch (IOException e) {
   LOG.warn(Could not finish splitting of log file  + logfile);
   return Status.ERR;
 }
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420393#comment-13420393
 ] 

chunhui shen commented on HBASE-5659:
-

@Lars
Could you add the failed test result except the exception caused by too many 
open files.
From the description, I don't see the following debug log
{code}
if (r.size() != 1) {
LOG.debug(r);
failures.incrementAndGet();
fail();
  }
{code}


 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420395#comment-13420395
 ] 

chunhui shen commented on HBASE-5659:
-

Sorry, I find the debug log...Let me think about why...

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
  entries=12, sequenceid=7934, filesize=1.3k
 2012-03-27 04:36:12,642 DEBUG

[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit

2012-07-22 Thread Jie Huang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jie Huang updated HBASE-6429:
-

Attachment: (was: hbase-6429-trunk.patch)

 Filter with filterRow() returning true is incompatible with scan with limit
 ---

 Key: HBASE-6429
 URL: https://issues.apache.org/jira/browse/HBASE-6429
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
 Attachments: hbase-6429_0_94_0.patch


 Currently if we scan with bot limit and a Filter with 
 filterRow(ListKeyValue) implemented, an  IncompatibleFilterException will 
 be thrown. The same exception should also be thrown if the filer has its 
 filterRow() implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit

2012-07-22 Thread Jie Huang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jie Huang updated HBASE-6429:
-

Attachment: hbase-6429-trunk.patch

Thank you very much for your time Ted. Really sorry for my mistakes. All tests 
are passed with this version according the local results. More comments, please 
let me know. thanks.

 Filter with filterRow() returning true is incompatible with scan with limit
 ---

 Key: HBASE-6429
 URL: https://issues.apache.org/jira/browse/HBASE-6429
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
 Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch


 Currently if we scan with bot limit and a Filter with 
 filterRow(ListKeyValue) implemented, an  IncompatibleFilterException will 
 be thrown. The same exception should also be thrown if the filer has its 
 filterRow() implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420401#comment-13420401
 ] 

chunhui shen commented on HBASE-5659:
-

The description log is 2012-03-27,  from the log 
{quote}2012-03-27 04:36:12,627 DEBUG [Thread-118] 
regionserver.StoreScanner(499): Storescanner.peek() is changed where before = 
rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
{quote} 
Major compaction deleted the two 
KVs(rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922 
rowB/colfamily11:qual1/7202/DeleteColumn), but the scanner is not closed, we 
shouldn't do the deleting.

However, it is previous log, is there any recent log?

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420403#comment-13420403
 ] 

chunhui shen commented on HBASE-5659:
-

About the patch, I don't think so, it will affect the scan's correctness again
Why Storescanner.peek() will be changed when checkReseek()(Only do this action 
after Flush or Compaction), in current logic it shouldn't happen as per MVCC

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420414#comment-13420414
 ] 

chunhui shen commented on HBASE-5659:
-

@Lars
The above is something wrong, I think patch is OK, sorry to make mistake.

I run the test many times but can't reproduce,  so it's hard to find the 
reason, waiting for your recent logs, thanks!

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit


 [ 
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-6429:
--

Status: Patch Available  (was: Open)

 Filter with filterRow() returning true is incompatible with scan with limit
 ---

 Key: HBASE-6429
 URL: https://issues.apache.org/jira/browse/HBASE-6429
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
 Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch


 Currently if we scan with bot limit and a Filter with 
 filterRow(ListKeyValue) implemented, an  IncompatibleFilterException will 
 be thrown. The same exception should also be thrown if the filer has its 
 filterRow() implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode


 [ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-5547:
--

Attachment: 5547.addendum-v3

Addendum v3 is based on v2 and changes TestZooKeeperTableArchiveClient to 
LargeTest.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420419#comment-13420419
 ] 

Jesse Yates commented on HBASE-5547:


Sounds good to me.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420423#comment-13420423
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Addendum v3 looks good to me. I'm happy to commit, unless you would like to, 
Ted.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420425#comment-13420425
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

@Lars:
Go ahead.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420427#comment-13420427
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Done. Should we file an extra jira and a cleaner chain test?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420429#comment-13420429
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

HBASE-6442 has been filed as a sub-task of this JIRA.
Feel free to either put detail there or add new task(s).

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420428#comment-13420428
 ] 

Lars Hofhansl commented on HBASE-5659:
--

Thanks for looking at this Chunhui. This is a tricky area.
Let me check in this change and see whether it happens again.

A failed run is here: 
https://builds.apache.org/job/HBase-0.94/354/testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testRowMutationMultiThreads/
and here: 
https://builds.apache.org/job/HBase-0.94/353/testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testRowMutationMultiThreads/

Although, now I am beginning to think this is something new.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to

[jira] [Updated] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.

2012-07-22 Thread Jonathan Hsieh (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-6089:
--

Fix Version/s: (was: 0.90.7)

Was no in 0.90.7

 SSH and AM.joinCluster causes Concurrent Modification exception.
 

 Key: HBASE-6089
 URL: https://issues.apache.org/jira/browse/HBASE-6089
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
 Fix For: 0.92.2, 0.96.0, 0.94.1

 Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch, 
 HBASE-6089_trunk.patch


 AM.regions map is parallely accessed in SSH and Master initialization leading 
 to ConcurrentModificationException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.

2012-07-22 Thread Jonathan Hsieh (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420431#comment-13420431
 ] 

Jonathan Hsieh edited comment on HBASE-6089 at 7/23/12 4:52 AM:


Was not committed to 0.90

  was (Author: jmhsieh):
Was no in 0.90.7
  
 SSH and AM.joinCluster causes Concurrent Modification exception.
 

 Key: HBASE-6089
 URL: https://issues.apache.org/jira/browse/HBASE-6089
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
 Fix For: 0.92.2, 0.96.0, 0.94.1

 Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch, 
 HBASE-6089_trunk.patch


 AM.regions map is parallely accessed in SSH and Master initialization leading 
 to ConcurrentModificationException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit

[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420433#comment-13420433
]

Hadoop QA commented on HBASE-6429:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537537/hbase-6429-trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2430//console

This message is automatically generated.

Filter with filterRow() returning true is incompatible with scan with limit
---

Key: HBASE-6429
URL: https://issues.apache.org/jira/browse/HBASE-6429
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch

Currently if we scan with bot limit and a Filter with
filterRow(ListKeyValue) implemented, an IncompatibleFilterException will
be thrown. The same exception should also be thrown if the filer has its
filterRow() implemented.

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420434#comment-13420434
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537542/5547.addendum-v3
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3,
hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch,
java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1,
java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch,
java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420435#comment-13420435
 ] 

chunhui shen commented on HBASE-5659:
-

I think a case to fail the case， maybe it is quite rare,
For the testRowMutationMultiThreads, assume there are 3 threads

1.Thread 1: rowA/colfamily11:qual1/1/Put/vlen=6/
rowA/colfamily11:qual2/1/DeleteColumn/vlen=6/

2.Thread 3: rowA/colfamily11:qual1/3/DeleteColumn/vlen=6/
rowA/colfamily11:qual2/3/Put/vlen=6/

3.Make major compaction

4.Thread 2:rowA/colfamily11:qual1/2/Put/vlen=6/
rowA/colfamily11:qual2/2/DeleteColumn/vlen=6/

5.Thread 2: Get rowA, and the result is 
rowA/colfamily11:qual1/2/Put/vlen=6/ + rowA/colfamily11:qual2/3/Put/vlen=6/



 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.2

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to

[jira] [Commented] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420437#comment-13420437
 ] 

Zhihong Ted Yu commented on HBASE-6441:
---

I don't think the following exception was caused by Jesse's patch:
{code}
org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
CatalogTracker$$EnhancerByMockitoWithCGLIB$$69988539 cannot be returned by 
isStopped()
isStopped() should return boolean
at 
org.apache.hadoop.hbase.master.TestAssignmentManager.processServerShutdownHandler(TestAssignmentManager.java:574)
at 
org.apache.hadoop.hbase.master.TestAssignmentManager.testCaseWithPartiallyDisabledState(TestAssignmentManager.java:521)
at 
org.apache.hadoop.hbase.master.TestAssignmentManager.testSSHWhenDisableTableInProgress(TestAssignmentManager.java:424)
{code}

Integrated to trunk.

Thanks for the patch, Jesse.

Thanks for the review, Stack.

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6441) MasterFS doesn't set scheme for internal FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420438#comment-13420438
 ] 

Zhihong Ted Yu commented on HBASE-6441:
---

Before integration, I ran the failed tests manually and they passed.

 MasterFS doesn't set scheme for internal FileSystem
 ---

 Key: HBASE-6441
 URL: https://issues.apache.org/jira/browse/HBASE-6441
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0, 0.94.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: java_HBASE-6441_v0.patch


 FSUtils.getRootDir() just takes a configuration object, which is used to:
 1) Get the name of the root directory
 2) Create a filesystem (based on the configured scheme)
 3) Qualify the root onto the filesystem
 However, the FileSystem from the master filesystem won't generate the 
 correctly qualified root directory under hadoop-2.0 (though it works fine on 
 hadoop-1.0). Seems to be an issue with the configuration parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


 [ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5659:
-

   Resolution: Fixed
Fix Version/s: (was: 0.94.2)
   0.94.1
   Status: Resolved  (was: Patch Available)

Committed to 0.94 and 0.96. I'll watch the test results.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420442#comment-13420442
 ] 

chunhui shen commented on HBASE-5659:
-

For the above failed tests( https://builds.apache.org/job/HBase-0.94/353/ 
https://builds.apache.org/job/HBase-0.94/354/), I took a look and find they 
were both caused by too many open files

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420443#comment-13420443
 ] 

Lars Hofhansl commented on HBASE-5659:
--

Oops. Didn't see the comment.
The problem in HBASE-5121 was that the scanner.next(...) would see a mix of KV 
from different rows, which would violate the scanner contract.

Is this a different problem? Is the a problem with my change here?
Lastly, could you make a runnable test? That'd be extremely helpful.

If it fails, even if rarely, it is a bug. :)


 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420444#comment-13420444
 ] 

Lars Hofhansl commented on HBASE-5659:
--

Re: Failed tests. Yes, I cannot find a single run that was not caused by 
TooManyOpenFiles.

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
  entries=12, sequenceid=7934, filesize=1.3k

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420445#comment-13420445
 ] 

Lars Hofhansl commented on HBASE-5659:
--

So you think the change is bad (comparing just the row of the old and new top 
KV)?

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
  entries=12, sequenceid=7934, filesize=1.3k

[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit

2012-07-22 Thread Jie Huang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420446#comment-13420446
 ] 

Jie Huang commented on HBASE-6429:
--

Re-run TestMetaReaderEditor locally, PASSED.
{noformat}
---
 T E S T S
---
Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.53 sec

Results :

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
{noformat}

 Filter with filterRow() returning true is incompatible with scan with limit
 ---

 Key: HBASE-6429
 URL: https://issues.apache.org/jira/browse/HBASE-6429
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.96.0
Reporter: Jason Dai
 Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch


 Currently if we scan with bot limit and a Filter with 
 filterRow(ListKeyValue) implemented, an  IncompatibleFilterException will 
 be thrown. The same exception should also be thrown if the filer has its 
 filterRow() implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420447#comment-13420447
 ] 

chunhui shen commented on HBASE-5659:
-

@Lars
I think patch won't make some problem, but I don't think it's the reason of the 
failed tests.

If all failed test were caused by TooManyOpenFiles, maybe there's no problem in 
current logic.

Correct me if wrong...Thanks

 TestAtomicOperation.testMultiRowMutationMultiThreads is still failing 
 occasionally
 --

 Key: HBASE-5659
 URL: https://issues.apache.org/jira/browse/HBASE-5659
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5659-v2.txt, 5659.txt


 See run here: 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
 {quote}
 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499): 
 Storescanner.peek() is changed where before = 
 rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after = 
 rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
 2012-03-27 04:36:12,629 INFO  [Thread-121] regionserver.HRegion(1558): 
 Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms, 
 sequenceid=7927, compaction requested=true
 2012-03-27 04:36:12,629 DEBUG [Thread-126] 
 regionserver.TestAtomicOperation$2(362): flushing
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426): 
 Started memstore flush for 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region 
 memstore size 1.9k
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474): 
 Finished snapshotting 
 testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait 
 for mvcc, flushsize=1968
 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484): 
 Finished snapshotting, commencing flushing stores
 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating 
 file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  with permission=rwxrwxrwx
 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143): 
 Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
 2012-03-27 04:36:12,631 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
  CompoundBloomFilterWriter
 2012-03-27 04:36:12,632 INFO  [Thread-126] 
 regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was 
 added to HFile 
 (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
  
 2012-03-27 04:36:12,632 INFO  [Thread-126] regionserver.Store(770): Flushed , 
 sequenceid=7934, memsize=1.9k, into tmp file 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming 
 flushed file at 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
  to 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
 2012-03-27 04:36:12,634 INFO  [Thread-126] regionserver.Store(818): Added

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode