[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852258#comment-13852258 ] stack commented on HBASE-10001: --- +1 > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850290#comment-13850290 ] Nicolas Liochon commented on HBASE-10001: - Yes, it was the same patch for 0.96 & trunk. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849919#comment-13849919 ] stack commented on HBASE-10001: --- So, was there an addendum on this Nicolas? Or was the trunk patch what went into 0.96? Thanks. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828644#comment-13828644 ] Hudson commented on HBASE-10001: FAILURE: Integrated in HBase-0.94 #1209 (See [https://builds.apache.org/job/HBase-0.94/1209/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543936) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828623#comment-13828623 ] Hudson commented on HBASE-10001: SUCCESS: Integrated in HBase-0.94-security #343 (See [https://builds.apache.org/job/HBase-0.94-security/343/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543936) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828334#comment-13828334 ] Hudson commented on HBASE-10001: SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #845 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/845/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543779) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828327#comment-13828327 ] Hudson commented on HBASE-10001: SUCCESS: Integrated in hbase-0.96-hadoop2 #126 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/126/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543778) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.094.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828028#comment-13828028 ] Lars Hofhansl commented on HBASE-10001: --- +1 on that change. And +1 on a different jira :) > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[
https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827974#comment-13827974
]
Nicolas Liochon commented on HBASE-10001:
-
I was thinking about something like this:
{noformat}
@@ -2024,7 +2024,7 @@ public class HRegion implements HeapSize { // , Writable{
WALEdit walEdit = new WALEdit(isInReplay);
MultiVersionConsistencyControl.WriteEntry w = null;
long txid = 0;
-boolean walSyncSuccessful = false;
+boolean doRollBackMemstore = false;
boolean locked = false;
/** Keep track of the locks we hold so we can release them in finally
clause */
@@ -2185,6 +2185,7 @@ public class HRegion implements HeapSize { // , Writable{
!= OperationStatusCode.NOT_RUN) {
continue;
}
+doRollBackMemstore = true; // If we have a failure, we need to clean
what we wrote
addedSize += applyFamilyMapToMemstore(familyMaps[i], w);
}
@@ -2244,7 +2245,7 @@ public class HRegion implements HeapSize { // , Writable{
if (walEdit.size() > 0) {
syncOrDefer(txid, durability);
}
- walSyncSuccessful = true;
+ doRollBackMemstore = false;
// calling the post CP hook for batch mutation
if (!isInReplay && coprocessorHost != null) {
MiniBatchOperationInProgress miniBatchOp =
@@ -2286,7 +2287,7 @@ public class HRegion implements HeapSize { // , Writable{
} finally {
// if the wal sync was unsuccessful, remove keys from memstore
- if (!walSyncSuccessful) {
+ if (doRollBackMemstore) {
rollbackMemstore(batchOp, familyMaps, firstIndex, lastIndexExclusive);
}
if (w != null) mvcc.completeMemstoreInsert(w);
{noformat}
What do you think?
In any case, since we agree that it should be on HRegion, let me create another
jira (as I need to commit the coprocessor for 0.94 here :-) )
> Add a coprocessor to help testing the performances without taking into
> account the i/o
> --
>
> Key: HBASE-10001
> URL: https://issues.apache.org/jira/browse/HBASE-10001
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.98.0, 0.96.0, 0.94.13
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 10001.v1.patch, 10001.v2.patch
>
>
> We have a mockup to test only the client. If we want to include the network,
> without beeing limited by the i/o, we don't have much tools.
> This coprocessor helps to test this.
> I put it in the main code as to make it usable without adding a jar...
> I don't think it's possible avoid the WAL writes in the coprocessors. It
> would be great to have it to simplify the test with any kind of client (i.e.
> w/o changing the durability).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827960#comment-13827960 ] Lars Hofhansl commented on HBASE-10001: --- We should fix it in HRegion, I think. Are you planning to change it into the Boolean (capital B) to have a way to indicate that we do not care, yet? Or have another flag (skippedByCoProc or something)? Or we could set walSyncSuccessful right before we return due to a coproc bypass. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827933#comment-13827933 ] Nicolas Liochon commented on HBASE-10001: - There is a bug linked to this: in HRegion#doMiniBatchMutation: we start with: ??boolean walSyncSuccessful = false;?? if the preBatch hook call byPass, we then return immediately, because of this: ?? if (coprocessorHost.preBatchMutate(miniBatchOp)) return 0L;?? then the finally clause is executed with ??walSyncSuccessful?? set to false, and then we try to rollback the memstore. In my load test, we're spending 15% of our time in the rollback. I can fix this in the coprocessor (by not setting bypass), but I think we should set the value of walSyncSuccessful later, just before we try to write something in the memstore. Any opinion? I can do the fix in HRegion#doMiniBatchMutation or in the coprocessor. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827862#comment-13827862 ] Hudson commented on HBASE-10001: SUCCESS: Integrated in hbase-0.96 #197 (See [https://builds.apache.org/job/hbase-0.96/197/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543778) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827855#comment-13827855 ] Nicolas Liochon commented on HBASE-10001: - Yes, I'm going to do it for 0.94 as well (today) > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827851#comment-13827851 ] Lars Hofhansl commented on HBASE-10001: --- This is targeted to 0.94, 0.96, and 0.98, but I see only the 0.98 commit. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827797#comment-13827797 ] Hudson commented on HBASE-10001: SUCCESS: Integrated in HBase-TRUNK #4689 (See [https://builds.apache.org/job/HBase-TRUNK/4689/]) HBASE-10001 Add a coprocessor to help testing the performances without taking into account the disk i/o (nkeywal: rev 1543779) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827434#comment-13827434 ] Anoop Sam John commented on HBASE-10001: +1 Pls add description (release notes) on how this can be used. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827426#comment-13827426 ] Nicolas Liochon commented on HBASE-10001: - I forgot the license. I can add it on commit. The test failure is unrelated. Any feedback on this? If it's ok, I would like to include it in 0.94 as well to help comparing the performances w/o being impacted by the disk i/o. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch, 10001.v2.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[
https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826867#comment-13826867
]
Hadoop QA commented on HBASE-10001:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614646/10001.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7934//console
This message is automatically generated.
> Add a coprocessor to help testing the performances without taking into
> account the i/o
> --
>
> Key: HBASE-10001
> URL: https://issues.apache.org/jira/browse/HBASE-10001
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.98.0, 0.96.0, 0.94.13
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 10001.v1.patch, 10001.v2.patch
>
>
> We have a mockup to test only the client. If we want to include the network,
> without beeing limited by the i/o, we don't have much tools.
> This coprocessor helps to test this.
> I put it in the main code as to make it usable without adding a jar...
> I don't think it's possible avoid the WAL writes in the coprocessors. It
> would be great to have it to simplify the test with any kind of client (i.e.
> w/o changing the durability).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826654#comment-13826654 ] Nicolas Liochon commented on HBASE-10001: - Thanks for the confirmation, I'm going to update the comments & upload the v2 then. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826652#comment-13826652 ] Anoop Sam John commented on HBASE-10001: Oh yes.. preBatchMutation is called before WAL write. So this is just fine. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[
https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826643#comment-13826643
]
Nicolas Liochon commented on HBASE-10001:
-
Actually, if I read correctly the code in HRegion, just marking the operations
as success as I do in the v1 is enough:
{code}
//
// STEP 4. Build WAL edit
// --
boolean hasWalAppends = false;
Durability durability = Durability.USE_DEFAULT;
for (int i = firstIndex; i < lastIndexExclusive; i++) {
// Skip puts that were determined to be invalid during preprocessing
if (batchOp.retCodeDetails[i].getOperationStatusCode()
!= OperationStatusCode.NOT_RUN) { //
<<<= if the copro marked it as SUCCESS we don't write it to
the WAL.
continue;
}
{code}
> Add a coprocessor to help testing the performances without taking into
> account the i/o
> --
>
> Key: HBASE-10001
> URL: https://issues.apache.org/jira/browse/HBASE-10001
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.98.0, 0.96.0, 0.94.13
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 10001.v1.patch
>
>
> We have a mockup to test only the client. If we want to include the network,
> without beeing limited by the i/o, we don't have much tools.
> This coprocessor helps to test this.
> I put it in the main code as to make it usable without adding a jar...
> I don't think it's possible avoid the WAL writes in the coprocessors. It
> would be great to have it to simplify the test with any kind of client (i.e.
> w/o changing the durability).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826618#comment-13826618 ] Andrew Purtell commented on HBASE-10001: IIRC you can also modify the put in the pre hook: put.writeToWAL(false) > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[ https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826609#comment-13826609 ] Nicolas Liochon commented on HBASE-10001: - bq. You want LogFactory.getLog(WriteSinkCoprocessor.class); ? Yes :-) bq. You can use prePut() hook and do a bypass from there so that even WAL write also can be avoided? Great, let me update the patch. > Add a coprocessor to help testing the performances without taking into > account the i/o > -- > > Key: HBASE-10001 > URL: https://issues.apache.org/jira/browse/HBASE-10001 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.94.13 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: 10001.v1.patch > > > We have a mockup to test only the client. If we want to include the network, > without beeing limited by the i/o, we don't have much tools. > This coprocessor helps to test this. > I put it in the main code as to make it usable without adding a jar... > I don't think it's possible avoid the WAL writes in the coprocessors. It > would be great to have it to simplify the test with any kind of client (i.e. > w/o changing the durability). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10001) Add a coprocessor to help testing the performances without taking into account the i/o
[
https://issues.apache.org/jira/browse/HBASE-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826602#comment-13826602
]
Anoop Sam John commented on HBASE-10001:
bq.private static final Log LOG = LogFactory.getLog(BaseRegionObserver.class);
You want LogFactory.getLog(WriteSinkCoprocessor.class); ?
{quote}
if (coprocessorHost.prePut((Put) m, walEdit, m.getDurability())) {
// pre hook says skip this Put
// mark as success and skip in doMiniBatchMutation
batchOp.retCodeDetails[i] = OperationStatus.SUCCESS;
}
{quote}
You can use prePut() hook and do a bypass from there so that even WAL write
also can be avoided?
> Add a coprocessor to help testing the performances without taking into
> account the i/o
> --
>
> Key: HBASE-10001
> URL: https://issues.apache.org/jira/browse/HBASE-10001
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.98.0, 0.96.0, 0.94.13
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 10001.v1.patch
>
>
> We have a mockup to test only the client. If we want to include the network,
> without beeing limited by the i/o, we don't have much tools.
> This coprocessor helps to test this.
> I put it in the main code as to make it usable without adding a jar...
> I don't think it's possible avoid the WAL writes in the coprocessors. It
> would be great to have it to simplify the test with any kind of client (i.e.
> w/o changing the durability).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
