[
https://issues.apache.org/jira/browse/OMID-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782091#comment-17782091
]
Rajeshbabu Chintaguntla edited comment on OMID-240 at 11/2/23 11:29 AM:
------------------------------------------------------------------------
Here is the metadata of the hfile gives the timeranges of cells in the post
commit.
{noformat}
reader=/apps/hbase/data/data/default/TEST/054c7bae986e6b6f6625dd7d898b1839/0/aa83662a8d2740b4a94fb23f514b9f03,
compression=none,
cacheConf=cacheDataOnRead=false,
cacheDataOnWrite=false,
cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false,
cacheEvictOnClose=false,
cacheDataCompressed=false,
prefetchOnOpen=false,
firstKey=Optional[\xB5E\xDDZ/0:\x00\xC2\x80Y\xC2\x80/1698917242286000014/Put/seqid=0],
lastKey=Optional[\xBF\x7F\xFF\xE9/0:_0/1698917242286000016/Put/seqid=0],
avgKeyLen=21,
avgValueLen=5,
entries=1043920,
length=15539025
Trailer:
fileinfoOffset=15534307,
loadOnOpenDataOffset=15515573,
dataIndexCount=593,
metaIndexCount=0,
totalUncomressedBytes=15515484,
entryCount=1043920,
compressionCodec=NONE,
uncompressedDataIndexSize=18656,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=15447293,
comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl,
encryptionKey=NONE,
majorVersion=3,
minorVersion=3
Fileinfo:
BLOOM_FILTER_TYPE = ROW
COMPACTION_EVENT_KEY = PBUF
DATA_BLOCK_ENCODING = FAST_DIFF
DELETE_FAMILY_COUNT = 0
EARLIEST_PUT_TS = 1698917242286000000
KEY_VALUE_VERSION = 1
LAST_BLOOM_KEY = \xBF\x7F\xFF\xE9
MAJOR_COMPACTION_KEY = false
MAX_MEMSTORE_TS_KEY = 3152
MAX_SEQ_ID_KEY = 3153
TIMERANGE = 1698917242286000000....1698917242286000018
hfile.AVG_KEY_LEN = 21
hfile.AVG_VALUE_LEN = 5
hfile.CREATE_TIME_TS = 1698917556166
hfile.LASTKEY = \xBF\x7F\xFF\xE9/0:_0/1698917242286000016/Put/vlen=0/mvcc=0
Mid-key: Optional[\xBF\x00-k/0:\x01/LATEST_TIMESTAMP/Maximum/seqid=0]
{noformat}
{noformat} TIMERANGE = 1698917242286000000....1698917242286000018 {noformat}
First scan info giving wrong result
{noformat}
2023-11-02 09:29:16,469 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 37 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917407892 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\200\211\374\346\271\212\361\311\027\020\200\211\374\346\271\212\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917347842000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302081 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 2711, receiveTime: 1698917347892, queueTime: 1,
processingTime: 8576, totalTime: 8577
{noformat}
Further scans info giving correct results.
{noformat}
2023-11-02 09:29:26,658 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 41 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917418631 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\200\356\207\353\341\212\361\311\027\020\200\356\207\353\341\212\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917358588000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302083 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 3149, receiveTime: 1698917358632, queueTime: 0,
processingTime: 8026, totalTime: 8026
2023-11-02 09:29:37,715 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 45 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917429297 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\300\260\347\302\211\213\361\311\027\020\300\260\347\302\211\213\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917369241000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302086 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 3152, receiveTime: 1698917369298, queueTime: 0,
processingTime: 8417, totalTime: 8417
{noformat}
As for the timeranges of scans and KVs the scan result giving wrong result also
fine only.
Need to check how those rows are visible while commit going on.
was (Author: rajeshbabu):
Here is the metadata of the hfile gives the timeranges of cells in the post
commit.
{noformat}
reader=/apps/hbase/data/data/default/TEST/054c7bae986e6b6f6625dd7d898b1839/0/aa83662a8d2740b4a94fb23f514b9f03,
compression=none,
cacheConf=cacheDataOnRead=false,
cacheDataOnWrite=false,
cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false,
cacheEvictOnClose=false,
cacheDataCompressed=false,
prefetchOnOpen=false,
firstKey=Optional[\xB5E\xDDZ/0:\x00\xC2\x80Y\xC2\x80/1698917242286000014/Put/seqid=0],
lastKey=Optional[\xBF\x7F\xFF\xE9/0:_0/1698917242286000016/Put/seqid=0],
avgKeyLen=21,
avgValueLen=5,
entries=1043920,
length=15539025
Trailer:
fileinfoOffset=15534307,
loadOnOpenDataOffset=15515573,
dataIndexCount=593,
metaIndexCount=0,
totalUncomressedBytes=15515484,
entryCount=1043920,
compressionCodec=NONE,
uncompressedDataIndexSize=18656,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=15447293,
comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl,
encryptionKey=NONE,
majorVersion=3,
minorVersion=3
Fileinfo:
BLOOM_FILTER_TYPE = ROW
COMPACTION_EVENT_KEY = PBUF
DATA_BLOCK_ENCODING = FAST_DIFF
DELETE_FAMILY_COUNT = 0
EARLIEST_PUT_TS = 1698917242286000000
KEY_VALUE_VERSION = 1
LAST_BLOOM_KEY = \xBF\x7F\xFF\xE9
MAJOR_COMPACTION_KEY = false
MAX_MEMSTORE_TS_KEY = 3152
MAX_SEQ_ID_KEY = 3153
TIMERANGE = 1698917242286000000....1698917242286000018
hfile.AVG_KEY_LEN = 21
hfile.AVG_VALUE_LEN = 5
hfile.CREATE_TIME_TS = 1698917556166
hfile.LASTKEY = \xBF\x7F\xFF\xE9/0:_0/1698917242286000016/Put/vlen=0/mvcc=0
Mid-key: Optional[\xBF\x00-k/0:\x01/LATEST_TIMESTAMP/Maximum/seqid=0]
{noformat}
{noformat} TIMERANGE = 1698917242286000000....1698917242286000018 {noformat}
First scan info giving wrong result
{noformat}
2023-11-02 09:29:16,469 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 37 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917407892 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\200\211\374\346\271\212\361\311\027\020\200\211\374\346\271\212\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917347842000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302081 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 2711, receiveTime: 1698917347892, queueTime: 1,
processingTime: 8576, totalTime: 8577
{noformat}
Further scans info giving correct results.
{noformat}
2023-11-02 09:29:26,658 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 41 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917418631 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\200\356\207\353\341\212\361\311\027\020\200\356\207\353\341\212\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917358588000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302083 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 3149, receiveTime: 1698917358632, queueTime: 0,
processingTime: 8026, totalTime: 8026
2023-11-02 09:29:37,715 TRACE
[RpcServer.default.FPBQ.Fifo.handler=49,queue=1,port=16020] ipc.RpcServer:
callId: 45 service: ClientService methodName: Scan size: 581 connection:
<host_ip>:51462 deadline: 1698917429297 param: region \{ type: REGION_NAME
value: "TEST,,1698917228121.054c7bae986e6b6f6625dd7d898b1839." } scan \{ column
{ family: "0" } attribute \{ name: "_ClientVersion" value: "\000\005\001\000" }
attribute \{ name: "__OMID_LL__" value: "\000" } attribute \{ name:
"__OMID_TRANSACTION__" value:
"\b\300\260\347\302\211\213\361\311\027\020\300\260\347\302\211\213\361\311\027\030\000
\377\377\377\377\377\377\377\377\377\001" } attribute \{ name: "_UngroupedAgg"
value: "\001" } attribute \{ name: "_ServerPageSizeMs" value:
"\000\000\000\000\000\000\257\310" } attribute \{ name:
"_UseNewValueColumnQualifier" value: "\377" } attribute \{ name: "_Aggs" value:
"\001\001\030\001\003\005\200\000\000\001\006\003" } attribute \{ name:
"_QualifierEncodingScheme" value: "\000" } attribute \{ name:
"_ImmutableStorageEncodingScheme" value: "\001" } attribute \{ name:
"_ScanActualStartRow" value: "" } attribute \{ name: "_isolationlevel_" value:
"\000" } attribute \{ name: "_SCAN_REGION_SERVER" value:
"\000\000host1,16020,1698917094253" } filter \{ name:
"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter" serialized_filter: "" }
time_range \{ from: 0 to: 1698917369241000001 } max_versions: 1 cache_blocks:
true max_result_size: 2097152 load_column_families_on_demand: true caching: 100
include_stop_row: false } number_of_rows: 100 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false connection: 10.211.61.94:51462, response:
cells_per_result: 1 scanner_id: 8745977501257302086 more_results: true ttl:
60000 stale: false partial_flag_per_result: false more_results_in_region: false
mvcc_read_point: 3152, receiveTime: 1698917369298, queueTime: 0,
processingTime: 8417, totalTime: 8417
{noformat}
As for the timeranges of scans and KVs the scan result giving wrong result also
fine only.
Need to check how the commit command is functioning.
> Transactional visibility is broken
> ----------------------------------
>
> Key: OMID-240
> URL: https://issues.apache.org/jira/browse/OMID-240
> Project: Phoenix Omid
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Lars Hofhansl
> Assignee: Rajeshbabu Chintaguntla
> Priority: Critical
> Attachments: hbase-omid-client-config.yml,
> omid-server-configuration.yml
>
>
> Client I:
> {code:java}
> > create table test(x float primary key, y float) DISABLE_WAL=true,
> TRANSACTIONAL=true;
> No rows affected (1.872 seconds)
> > !autocommit off
> Autocommit status: false
> > upsert into test values(rand(), rand());
> 1 row affected (0.018 seconds)
> > upsert into test select rand(), rand() from test;
> -- 18-20x
> > !commit{code}
>
> Client II:
> {code:java}
> -- repeat quickly after the commit on client I
> > select count(*) from test;
> +----------+
> | COUNT(1) |
> +----------+
> | 0 |
> +----------+
> 1 row selected (1.408 seconds)
> > select count(*) from test;
> +----------+
> | COUNT(1) |
> +----------+
> | 259884 |
> +----------+
> 1 row selected (2.959 seconds)
> > select count(*) from test;
> +----------+
> | COUNT(1) |
> +----------+
> | 260145 |
> +----------+
> 1 row selected (4.274 seconds)
> > select count(*) from test;
> +----------+
> | COUNT(1) |
> +----------+
> | 260148 |
> +----------+
> 1 row selected (5.563 seconds)
> > select count(*) from test;
> +----------+
> | COUNT(1) |
> +----------+
> | 260148 |
> +----------+
> 1 row selected (5.573 seconds){code}
> The second client should either show 0 or 260148. But no other value!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
