[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354136#comment-14354136 ] hongyu bi commented on HBASE-13170: --- Cool ! Looking forward for your work:) Allow block cache to be external Key: HBASE-13170 URL: https://issues.apache.org/jira/browse/HBASE-13170 Project: HBase Issue Type: New Feature Components: io Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.1.0 Allow an external service to provide the block cache. This has the nice property of allowing failover/upgrades to happen without causing a fully cold cache. Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Calling Increment#addColumn on the same column multiple times produces wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327686#comment-14327686
]
hongyu bi commented on HBASE-12948:
---
thanks ted,andrew and stack :)
I opened HBASE-13073 for API issues.
Calling Increment#addColumn on the same column multiple times produces wrong
result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: 12948-0.98.txt, 12948-v2.patch, 12948-v2.patch,
HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13073) refactor mutation's familyMap in case of multi mutation on same column
hongyu bi created HBASE-13073: - Summary: refactor mutation's familyMap in case of multi mutation on same column Key: HBASE-13073 URL: https://issues.apache.org/jira/browse/HBASE-13073 Project: HBase Issue Type: Bug Components: Client Reporter: hongyu bi Assignee: hongyu bi per HBASE-12948 it's found that we can do multi mutations on the same column for mutation object ,which will make no sense(even produce wrong results before HBASE-12948) but put more traffic to RS.So we want to refactor mutation's familyMap in case of multi mutation on same column . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317341#comment-14317341
]
hongyu bi commented on HBASE-12948:
---
Thanks [~stack] for review:)
as for open another issue to match the get's way ,i have a concern that rolling
upgrade support may be the tricky part.I just thought two options:
option#1: create a treemap to match the get's behaviour from client side but
sending a List finally which may need a copy?
option#2: prohibit multi mutations on the same column in one mutation call.
both options seems not that elegant ,just a quick thought
any suggestion are welcome
thanks
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12988) [Replication]Parallel apply edits on table-level
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313581#comment-14313581 ] hongyu bi commented on HBASE-12988: --- agreed, table-level may hit hot-spot easier than row-level. Title changed, thanks [Replication]Parallel apply edits on table-level Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Components: Replication Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12988) [Replication]Parallel apply edits on row-level
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12988: -- Summary: [Replication]Parallel apply edits on row-level (was: [Replication]Parallel apply edits on table-level) [Replication]Parallel apply edits on row-level -- Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Components: Replication Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12988) [Replication]Parallel apply edits on row-level
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12988: -- Description: we can apply edits to slave cluster in parallel on table-level to speed up replication . update : per conversion blow , it's better to apply edits on row-level in parallel was:we can apply edits to slave cluster in parallel on table-level to speed up replication . [Replication]Parallel apply edits on row-level -- Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Components: Replication Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . update : per conversion blow , it's better to apply edits on row-level in parallel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12988) [Replication]Parallel apply edits on row-level
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12988: -- Description: we can apply edits to slave cluster in parallel on table-level to speed up replication . update : per conversation blow , it's better to apply edits on row-level in parallel was: we can apply edits to slave cluster in parallel on table-level to speed up replication . update : per conversion blow , it's better to apply edits on row-level in parallel [Replication]Parallel apply edits on row-level -- Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Components: Replication Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . update : per conversation blow , it's better to apply edits on row-level in parallel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-3473) Add a configuration property for socket receive buffer size for replication endpoints
[ https://issues.apache.org/jira/browse/HBASE-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311887#comment-14311887 ] hongyu bi commented on HBASE-3473: -- Hi [~lhofhansl]， we may try parallel replication by hash rowkey to thread# , so we can preserve ordering on row-level which i think is fine to HBase. thanks Add a configuration property for socket receive buffer size for replication endpoints - Key: HBASE-3473 URL: https://issues.apache.org/jira/browse/HBASE-3473 Project: HBase Issue Type: Improvement Components: Replication Reporter: Gary Helmling Priority: Minor Looking at this blog post about optimizing replication throughput for LinkedIn's Kafka: http://sna-projects.com/blog/2011/01/optimizing-tcp-socket-across-data-centers/ It seems worth testing out if HBase replication connections can also benefit from increasing the socket receive buffer size on (expected to be) high-latency connections. To this end, we would add a new configuration property for receive buffer size for replication connection and do some benchmarking to evaluate throughput with different values, verifying that making this configurable would have significant impact. For the moment, it seems best to scope the configuration setting to replication connections only, in order to avoid also impacting (negatively) intra-cluster communications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-3473) Add a configuration property for socket receive buffer size for replication endpoints
[ https://issues.apache.org/jira/browse/HBASE-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311895#comment-14311895 ] hongyu bi commented on HBASE-3473: -- just revisit replication code, in ReplicationSink#replicateEntries, we can do a quick minor improvement to send batchRPC parallely on table-level. Add a configuration property for socket receive buffer size for replication endpoints - Key: HBASE-3473 URL: https://issues.apache.org/jira/browse/HBASE-3473 Project: HBase Issue Type: Improvement Components: Replication Reporter: Gary Helmling Priority: Minor Looking at this blog post about optimizing replication throughput for LinkedIn's Kafka: http://sna-projects.com/blog/2011/01/optimizing-tcp-socket-across-data-centers/ It seems worth testing out if HBase replication connections can also benefit from increasing the socket receive buffer size on (expected to be) high-latency connections. To this end, we would add a new configuration property for receive buffer size for replication connection and do some benchmarking to evaluate throughput with different values, verifying that making this configurable would have significant impact. For the moment, it seems best to scope the configuration setting to replication connections only, in order to avoid also impacting (negatively) intra-cluster communications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12988) [Replication]Parallel apply edits on table-level
hongyu bi created HBASE-12988: - Summary: [Replication]Parallel apply edits on table-level Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-3473) Add a configuration property for socket receive buffer size for replication endpoints
[ https://issues.apache.org/jira/browse/HBASE-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311903#comment-14311903 ] hongyu bi commented on HBASE-3473: -- filed HBASE-12988 Add a configuration property for socket receive buffer size for replication endpoints - Key: HBASE-3473 URL: https://issues.apache.org/jira/browse/HBASE-3473 Project: HBase Issue Type: Improvement Components: Replication Reporter: Gary Helmling Priority: Minor Looking at this blog post about optimizing replication throughput for LinkedIn's Kafka: http://sna-projects.com/blog/2011/01/optimizing-tcp-socket-across-data-centers/ It seems worth testing out if HBase replication connections can also benefit from increasing the socket receive buffer size on (expected to be) high-latency connections. To this end, we would add a new configuration property for receive buffer size for replication connection and do some benchmarking to evaluate throughput with different values, verifying that making this configurable would have significant impact. For the moment, it seems best to scope the configuration setting to replication connections only, in order to avoid also impacting (negatively) intra-cluster communications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12988) [Replication]Parallel apply edits on table-level
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12988: -- Component/s: Replication [Replication]Parallel apply edits on table-level Key: HBASE-12988 URL: https://issues.apache.org/jira/browse/HBASE-12988 Project: HBase Issue Type: Improvement Components: Replication Reporter: hongyu bi Assignee: hongyu bi we can apply edits to slave cluster in parallel on table-level to speed up replication . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311839#comment-14311839
]
hongyu bi commented on HBASE-12948:
---
just wonder why mutation's familyMap is type of NavigableMapbyte [],
ListCell not Mapbyte [], NavigableSetbyte [] like *Get* .
thanks
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12957) region_mover#isSuccessfulScan may be extremely slow on region with lots of expired data
[ https://issues.apache.org/jira/browse/HBASE-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308417#comment-14308417 ] hongyu bi commented on HBASE-12957: --- Thanks [~lhofhansl] , [~apurtell] . In fact the original change is to use isSuccessfulGet instead. Just as andrew said, a get-like scan is to keep a minor change . thanks. region_mover#isSuccessfulScan may be extremely slow on region with lots of expired data --- Key: HBASE-12957 URL: https://issues.apache.org/jira/browse/HBASE-12957 Project: HBase Issue Type: Improvement Components: scripts Reporter: hongyu bi Assignee: hongyu bi Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12957-v0.patch region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. workaround:before graceful_stop/rolling_restart ，call major_compact on the table with small TTL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12977) normalize handlerCount to keep handlers distributed evenly among callQueues
hongyu bi created HBASE-12977: - Summary: normalize handlerCount to keep handlers distributed evenly among callQueues Key: HBASE-12977 URL: https://issues.apache.org/jira/browse/HBASE-12977 Project: HBase Issue Type: Improvement Reporter: hongyu bi Assignee: hongyu bi Priority: Minor If enable multi callQueues , handlers may not be distributed evenly among multi queues, which mean the queue's capacity is not the same. Should we make handler's distribution even? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12977) normalize handlerCount to keep handlers distributed evenly among callQueues
[ https://issues.apache.org/jira/browse/HBASE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307072#comment-14307072 ] hongyu bi commented on HBASE-12977: --- thanks [~mbertozzi] for review. what i mean is that if handlerCount is a prime number or else with handler factor enabled, rpc handlers won't be distributed evenly among callQueues. let's say handlerCount =10 ,handler factor=0.4 which produce numCallQueues to 4, so we distribute 10 handlers to 4 callQueues. with this patch we'll normalize handlerCount to 12. IMO, the handler factor option is not that obvious to tune for administrator . Thanks. normalize handlerCount to keep handlers distributed evenly among callQueues Key: HBASE-12977 URL: https://issues.apache.org/jira/browse/HBASE-12977 Project: HBase Issue Type: Improvement Reporter: hongyu bi Assignee: hongyu bi Priority: Minor Attachments: HBASE-12977-v0.patch If enable multi callQueues , handlers may not be distributed evenly among multi queues, which mean the queue's capacity is not the same. Should we make handler's distribution even? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12977) normalize handlerCount to keep handlers distributed evenly among callQueues
[ https://issues.apache.org/jira/browse/HBASE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12977: -- Attachment: HBASE-12977-v0.patch normalize handlerCount to keep handlers distributed evenly among callQueues Key: HBASE-12977 URL: https://issues.apache.org/jira/browse/HBASE-12977 Project: HBase Issue Type: Improvement Reporter: hongyu bi Assignee: hongyu bi Priority: Minor Attachments: HBASE-12977-v0.patch If enable multi callQueues , handlers may not be distributed evenly among multi queues, which mean the queue's capacity is not the same. Should we make handler's distribution even? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12977) normalize handlerCount to keep handlers distributed evenly among callQueues
[ https://issues.apache.org/jira/browse/HBASE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307134#comment-14307134 ] hongyu bi commented on HBASE-12977: --- oops，sorry for file this jira, my bad. the deviation is not that large among callQueues. normalize handlerCount to keep handlers distributed evenly among callQueues Key: HBASE-12977 URL: https://issues.apache.org/jira/browse/HBASE-12977 Project: HBase Issue Type: Improvement Reporter: hongyu bi Assignee: hongyu bi Priority: Minor Attachments: HBASE-12977-v0.patch If enable multi callQueues , handlers may not be distributed evenly among multi queues, which mean the queue's capacity is not the same. Should we make handler's distribution even? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12968) SecureServer should not ignore CallQueueSize
[ https://issues.apache.org/jira/browse/HBASE-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12968: -- Fix Version/s: 0.94.27 SecureServer should not ignore CallQueueSize Key: HBASE-12968 URL: https://issues.apache.org/jira/browse/HBASE-12968 Project: HBase Issue Type: Bug Affects Versions: 0.94.15 Reporter: hongyu bi Assignee: hongyu bi Fix For: 0.94.27 Attachments: HBASE-12968-v0.patch Per HBASE-5190 HBaseServer will reject the request If callQueueSize exceed ipc.server.max.callqueue.length, but SecureServer ignore this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12968) SecureServer should not ignore CallQueueSize
[ https://issues.apache.org/jira/browse/HBASE-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12968: -- Attachment: HBASE-12968-v0.patch SecureServer should not ignore CallQueueSize Key: HBASE-12968 URL: https://issues.apache.org/jira/browse/HBASE-12968 Project: HBase Issue Type: Bug Affects Versions: 0.94.15 Reporter: hongyu bi Assignee: hongyu bi Attachments: HBASE-12968-v0.patch Per HBASE-5190 HBaseServer will reject the request If callQueueSize exceed ipc.server.max.callqueue.length, but SecureServer ignore this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12968) SecureServer should not ignore CallQueueSize
hongyu bi created HBASE-12968: - Summary: SecureServer should not ignore CallQueueSize Key: HBASE-12968 URL: https://issues.apache.org/jira/browse/HBASE-12968 Project: HBase Issue Type: Bug Affects Versions: 0.94.15 Reporter: hongyu bi Assignee: hongyu bi Per HBASE-5190 HBaseServer will reject the request If callQueueSize exceed ipc.server.max.callqueue.length, but SecureServer ignore this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306598#comment-14306598
]
hongyu bi commented on HBASE-12948:
---
ping [~apurtell]] [~enis]]
Thanks
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12968) [0.94]SecureServer should not ignore CallQueueSize
[ https://issues.apache.org/jira/browse/HBASE-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306712#comment-14306712 ] hongyu bi commented on HBASE-12968: --- Yes , we've fixed this in our internal branch thanks. [0.94]SecureServer should not ignore CallQueueSize -- Key: HBASE-12968 URL: https://issues.apache.org/jira/browse/HBASE-12968 Project: HBase Issue Type: Bug Affects Versions: 0.94.15 Reporter: hongyu bi Assignee: hongyu bi Fix For: 0.94.27 Attachments: HBASE-12968-v0.patch Per HBASE-5190 HBaseServer will reject the request If callQueueSize exceed ipc.server.max.callqueue.length, but SecureServer ignore this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12968) [0.94]SecureServer should not ignore CallQueueSize
[ https://issues.apache.org/jira/browse/HBASE-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12968: -- Summary: [0.94]SecureServer should not ignore CallQueueSize (was: SecureServer should not ignore CallQueueSize) [0.94]SecureServer should not ignore CallQueueSize -- Key: HBASE-12968 URL: https://issues.apache.org/jira/browse/HBASE-12968 Project: HBase Issue Type: Bug Affects Versions: 0.94.15 Reporter: hongyu bi Assignee: hongyu bi Fix For: 0.94.27 Attachments: HBASE-12968-v0.patch Per HBASE-5190 HBaseServer will reject the request If callQueueSize exceed ipc.server.max.callqueue.length, but SecureServer ignore this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi reassigned HBASE-12948:
-
Assignee: hongyu bi
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, 12948-v3.patch,
HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi updated HBASE-12948:
--
Attachment: (was: 12948-v3.patch)
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Assignee: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12957) region_mover#isSuccessfulScan may extremely slow on region with lots of expired data
hongyu bi created HBASE-12957: - Summary: region_mover#isSuccessfulScan may extremely slow on region with lots of expired data Key: HBASE-12957 URL: https://issues.apache.org/jira/browse/HBASE-12957 Project: HBase Issue Type: Improvement Components: scripts Reporter: hongyu bi Priority: Minor region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12957) region_mover#isSuccessfulScan may extremely slow on region with lots of expired data
[ https://issues.apache.org/jira/browse/HBASE-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12957: -- Description: region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. workaround:before graceful_stop/rolling_restart ，call major_compact on the table with small TTL was:region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. region_mover#isSuccessfulScan may extremely slow on region with lots of expired data Key: HBASE-12957 URL: https://issues.apache.org/jira/browse/HBASE-12957 Project: HBase Issue Type: Improvement Components: scripts Reporter: hongyu bi Priority: Minor Attachments: HBASE-12957-v0.patch region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. workaround:before graceful_stop/rolling_restart ，call major_compact on the table with small TTL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12957) region_mover#isSuccessfulScan may extremely slow on region with lots of expired data
[ https://issues.apache.org/jira/browse/HBASE-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12957: -- Attachment: HBASE-12957-v0.patch region_mover#isSuccessfulScan may extremely slow on region with lots of expired data Key: HBASE-12957 URL: https://issues.apache.org/jira/browse/HBASE-12957 Project: HBase Issue Type: Improvement Components: scripts Reporter: hongyu bi Priority: Minor Attachments: HBASE-12957-v0.patch region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12957) region_mover#isSuccessfulScan may extremely slow on region with lots of expired data
[ https://issues.apache.org/jira/browse/HBASE-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302862#comment-14302862 ] hongyu bi commented on HBASE-12957: --- IMO, isSuccessfulScan is called to make sure the target region is online and serving ,or it'll throw TableNotFoundException/TableNotEnabledException/NotServingRegionException.It's not have to make sure we find the first item. please correct me if wrong. thanks stack:) region_mover#isSuccessfulScan may extremely slow on region with lots of expired data Key: HBASE-12957 URL: https://issues.apache.org/jira/browse/HBASE-12957 Project: HBase Issue Type: Improvement Components: scripts Reporter: hongyu bi Priority: Minor Attachments: HBASE-12957-v0.patch region_mover will call isSuccessfulScan when region has moved to make sure it's healthy, however , if the moved region has lots of expired data region_mover#isSuccessfulScan will take a long time to finish ,that may even exceed lease timeout.So I made isSuccessfulScan a get-like scan to achieve faster response in that case. workaround:before graceful_stop/rolling_restart ，call major_compact on the table with small TTL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302714#comment-14302714
]
hongyu bi commented on HBASE-12948:
---
Thanks Andrew，so patch-v2 should be fine here?
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, 12948-v3.patch,
HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300782#comment-14300782
]
hongyu bi commented on HBASE-12948:
---
Hi Ted,
i just found that according transaction isolation, since multi
Increment#addColumn on the same column are inside one transaction,current edit
should see previous edits ,which means we should merge the edits instead if
last edit wins on the same column . if this is true,it'll produce totally
different results to 0.94 and before.
please correct me if i'm wrong
thanks
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi updated HBASE-12948:
--
Attachment: 12948-v3.patch
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, 12948-v3.patch,
HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300921#comment-14300921
]
hongyu bi commented on HBASE-12948:
---
Thanks for ur suggestion,setColumn here just avoid the misunderstand of
transaction isolation mentioned above ,we can change addColumn to setColumn and
make addColumn refer to it.
As for fix this client side, do you mean put last edit win logic inside
client? If so, what about open another jira to fix all the subclass of mutation
or under this one?
Thanks
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299521#comment-14299521
]
hongyu bi commented on HBASE-12948:
---
ut added;
I need to compare the cq between current and the next edit to decide whether
push idx or not, so i used index loop
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch,
HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi updated HBASE-12948:
--
Attachment: HBASE-12948-0.99.2-v1.patch
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: HBASE-12948-0.99.2-v1.patch, HBASE-12948-v0.patch,
HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299685#comment-14299685
]
hongyu bi commented on HBASE-12948:
---
thanks ted:)
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: 12948-v2.patch, HBASE-12948-0.99.2-v1.patch,
HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi updated HBASE-12948:
--
Attachment: HBASE-12948.patch
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu bi updated HBASE-12948:
--
Attachment: HBASE-12948-v0.patch
resubmit
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: HBASE-12948-v0.patch, HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
hongyu bi created HBASE-12948:
-
Summary: Increment#addColumn on the same column multi times
produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent mutation
which change familyMap from NavigableMap to List, so from client side, we can
buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same column
won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12948) Increment#addColumn on the same column multi times produce wrong result
[
https://issues.apache.org/jira/browse/HBASE-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298684#comment-14298684
]
hongyu bi commented on HBASE-12948:
---
oops, there's logic mistake, resubmit
Increment#addColumn on the same column multi times produce wrong result
Key: HBASE-12948
URL: https://issues.apache.org/jira/browse/HBASE-12948
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Reporter: hongyu bi
Priority: Critical
Attachments: HBASE-12948.patch
Case:
Initially get('row1'):
rowkey=row1 value=1
run:
Increment increment = new Increment(Bytes.toBytes(row1));
for (int i = 0; i N; i++) {
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes(c), 1)
}
hobi.increment(increment);
get('row1'):
if N=1 then result is 2 else if N1 the result will always be 1
Cause:
https://issues.apache.org/jira/browse/HBASE-7114 let increment extent
mutation which change familyMap from NavigableMap to List, so from client
side, we can buffer many edits on the same column；
However, HRegion#increment use idx to iterate the get's results, here
results.sizefamily.value().size if N1，so the latter edits on the same
column won't match the condition {idx results.size()
CellUtil.matchingQualifier(results.get(idx), kv) }, meantime the edits share
the same mvccVersion ,so this case happen.
Fix:
according to the put/delete#add on the same column behaviour ,
fix from server side: process last edit wins on the same column inside
HRegion#increment to maintenance HBASE-7114's extension and keep the same
result from 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11586) HFile's HDFS op latency sampling code is not used
[
https://issues.apache.org/jira/browse/HBASE-11586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296593#comment-14296593
]
hongyu bi commented on HBASE-11586:
---
we just upgrade 0.98.6 from 0.94.15 , and I can't find the
fs{read|write}ops/latency metrics, what about the new jira? many thanks
HFile's HDFS op latency sampling code is not used
-
Key: HBASE-11586
URL: https://issues.apache.org/jira/browse/HBASE-11586
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.4
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.99.0, 0.98.5, 2.0.0
Attachments: HBASE-11586.patch, HBASE-11586.patch
HFileReaderV2 calls HFile#offerReadLatency and HFileWriterV2 calls
HFile#offerWriteLatency but the samples are never retrieved. There are no
callers of HFile#getReadLatenciesNanos, HFile#getWriteLatenciesNanos, and
related. The three ArrayBlockingQueues we are using as sample buffers in
HFile will fill quickly and are never drained.
There are also no callers of HFile#getReadTimeMs or HFile#getWriteTimeMs, and
related, so we are incrementing a set of AtomicLong counters that will never
be read nor reset.
We are calling System.nanoTime in block read and write paths twice but not
utilizing the measurements.
We should hook this code back up to metrics or remove it.
We are also not using HFile#getChecksumFailuresCount anywhere but in some
unit test code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12649) [0.94] port HBASE-11705 callQueueSize should be decremented in a fail-fast scenario
hongyu bi created HBASE-12649: - Summary: [0.94] port HBASE-11705 callQueueSize should be decremented in a fail-fast scenario Key: HBASE-12649 URL: https://issues.apache.org/jira/browse/HBASE-12649 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: hongyu bi Refer to https://issues.apache.org/jira/browse/HBASE-11705 which may lead to callQueueSize leak -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12649) [0.94] port HBASE-11705 callQueueSize should be decremented in a fail-fast scenario
[ https://issues.apache.org/jira/browse/HBASE-12649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu bi updated HBASE-12649: -- Description: Refer to HBASE-11705 which may lead to callQueueSize leak (was: Refer to https://issues.apache.org/jira/browse/HBASE-11705 which may lead to callQueueSize leak) [0.94] port HBASE-11705 callQueueSize should be decremented in a fail-fast scenario --- Key: HBASE-12649 URL: https://issues.apache.org/jira/browse/HBASE-12649 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: hongyu bi Refer to HBASE-11705 which may lead to callQueueSize leak -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10506) Fail-fast if client connection is lost before the real call be executed in RPC layer
[
https://issues.apache.org/jira/browse/HBASE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236729#comment-14236729
]
hongyu bi commented on HBASE-10506:
---
Sorry, just found HBASE-11705，I will open a new JIRA HBASE-12649 to backport it
to 0.94
Fail-fast if client connection is lost before the real call be executed in
RPC layer
Key: HBASE-10506
URL: https://issues.apache.org/jira/browse/HBASE-10506
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Affects Versions: 0.94.3
Reporter: Liang Xie
Assignee: Liang Xie
Fix For: 0.98.0, 0.96.2, 0.99.0, 0.94.17
Attachments: HBASE-10506-0.94.txt, HBASE-10506-trunk.txt
In current HBase rpc impletement, there is no any connection double-checking
just before the call be invoked, considing there's a gc or other OS
scheduling or the call queue is full enough(e.g. the server side is slow/hang
due to some issues), and if the client side has a small rpc timeout value, it
could be possible when this request be taken from call queue, the client
connection is lost in that moment. we'd better has some fail-fast code before
the reall call be invoked, it just waste the server side resource.
Here is a strace trace from our production env:
2014-02-11,18:16:19,525 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call get([B@3eae6c77,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:0741031-m8997060}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43252: output error
2014-02-11,18:16:19,526 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 151 on 12600 caught a ClosedChannelException, this means that the
server was processing a request but the client went away. The error message
was: null
2014-02-11,18:16:19,797 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
get([B@3f10ffd2,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:4245978-m7281526}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43259 after 0 ms, since caller disconnected
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:450)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3633)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3590)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3615)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4414)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4387)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2075)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:460)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1457)
2014-02-11,18:16:19,802 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call get([B@3f10ffd2,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:4245978-m7281526}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43259: output error
2014-02-11,18:16:19,802 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 46 on 12600 caught a ClosedChannelException, this means that the
server was processing a request but the client went away. The error message
was: null
With this fix, we can reduce this hit probability at least:) the upstream
hadoop has this checking already, see:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L2034-L2036
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10506) Fail-fast if client connection is lost before the real call be executed in RPC layer
[
https://issues.apache.org/jira/browse/HBASE-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236599#comment-14236599
]
hongyu bi commented on HBASE-10506:
---
fail-fast doesn't decrease the callQueueSize which may lead to Call queue is
full
Fail-fast if client connection is lost before the real call be executed in
RPC layer
Key: HBASE-10506
URL: https://issues.apache.org/jira/browse/HBASE-10506
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Affects Versions: 0.94.3
Reporter: Liang Xie
Assignee: Liang Xie
Fix For: 0.98.0, 0.96.2, 0.99.0, 0.94.17
Attachments: HBASE-10506-0.94.txt, HBASE-10506-trunk.txt
In current HBase rpc impletement, there is no any connection double-checking
just before the call be invoked, considing there's a gc or other OS
scheduling or the call queue is full enough(e.g. the server side is slow/hang
due to some issues), and if the client side has a small rpc timeout value, it
could be possible when this request be taken from call queue, the client
connection is lost in that moment. we'd better has some fail-fast code before
the reall call be invoked, it just waste the server side resource.
Here is a strace trace from our production env:
2014-02-11,18:16:19,525 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call get([B@3eae6c77,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:0741031-m8997060}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43252: output error
2014-02-11,18:16:19,526 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 151 on 12600 caught a ClosedChannelException, this means that the
server was processing a request but the client went away. The error message
was: null
2014-02-11,18:16:19,797 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
get([B@3f10ffd2,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:4245978-m7281526}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43259 after 0 ms, since caller disconnected
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:450)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3633)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3590)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3615)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4414)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4387)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2075)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:460)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1457)
2014-02-11,18:16:19,802 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call get([B@3f10ffd2,
{timeRange:[0,9223372036854775807],totalColumns:1,cacheBlocks:true,families:{X:[T]},maxVersions:1,row:4245978-m7281526}),
rpc version=1, client version=29, methodsFingerPrint=-241105381 from
10.101.10.181:43259: output error
2014-02-11,18:16:19,802 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 46 on 12600 caught a ClosedChannelException, this means that the
server was processing a request but the client went away. The error message
was: null
With this fix, we can reduce this hit probability at least:) the upstream
hadoop has this checking already, see:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L2034-L2036
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11769) Truncate table shouldn't revoke user privileges
hongyu bi created HBASE-11769: - Summary: Truncate table shouldn't revoke user privileges Key: HBASE-11769 URL: https://issues.apache.org/jira/browse/HBASE-11769 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.94.15 Reporter: hongyu bi hbase(main):002:0 create 'a','cf' 0 row(s) in 0.2500 seconds = Hbase::Table - a hbase(main):003:0 grant 'usera','R','a' 0 row(s) in 0.2080 seconds hbase(main):007:0 user_permission 'a' User Table,Family,Qualifier:Permission usera a,,: [Permission: actions=READ] hbase(main):004:0 truncate 'a' Truncating 'a' table (it may take a while): - Disabling table... - Dropping table... - Creating table... 0 row(s) in 1.5320 seconds hbase(main):005:0 user_permission 'a' User Table,Family,Qualifier:Permission -- This message was sent by Atlassian JIRA (v6.2#6252)
