RE: Inspecting SST state of rocksdb

[Schwalbe Matthias]

Good morning Kai,

Did you consider using the ‘State Processor API’ yet? : 
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/state_processor_api/

This one allows you to load any state primitive as a batch source, and …

Best greetings

Thias


From: Kai Fu 
Sent: Montag, 9. August 2021 07:01
To: user 
Subject: Inspecting SST state of rocksdb

Hi team,

I'm trying to inspect SST files of flink's state with sst related tools like 
sst_dump, ldb in 
wiki<https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool>.
 But it seems I'm getting meaningless results as shown below. The tools I'm 
using are from RocksDB's trunk and built from source. Am I doing it the right 
way, or is there any alternative to inspect the state? We're aware of Flink's 
queryable state, while it seems not well supported for SQL generated operators.

$ ./sst_dump --file=../db/30.sst --command=scan --read_num=50
options.env is 0xba33e0
Process ../db/30.sst
Sst file format: block-based
from [] to []
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>

--
Best wishes,
- Kai
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet 
unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von 
e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine 
Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser 
Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per 
e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche 
unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng 
verboten.

This message is intended only for the named recipient and may contain 
confidential or privileged information. As the confidentiality of email 
communication cannot be guaranteed, we do not accept any responsibility for the 
confidentiality and the intactness of this message. If you have received it in 
error, please advise the sender by return e-mail and delete this message and 
any attachments. Any unauthorised use or dissemination of this information is 
strictly prohibited.

Re: Inspecting SST state of rocksdb

[Yun Tang]

Hi Kai,

Since the stored key and value in RocksDB are serialized bytes, it's not easy 
to read directly. You could consider to use State Processor API [1]. However, 
this needs you to know what the state name of the SQL operator. You could 
analysis the checkpoing  _metadata [2] to know state names.


[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/state_processor_api/#state-processor-api
[2] 
https://github.com/apache/flink/blob/0d2b945729df8f0a0149d02ca24633ae52a1ef21/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99

Best
Yun Tang

From: Kai Fu 
Sent: Monday, August 9, 2021 13:01
To: user 
Subject: Inspecting SST state of rocksdb

Hi team,

I'm trying to inspect SST files of flink's state with sst related tools like 
sst_dump, ldb in 
wiki<https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool>.
 But it seems I'm getting meaningless results as shown below. The tools I'm 
using are from RocksDB's trunk and built from source. Am I doing it the right 
way, or is there any alternative to inspect the state? We're aware of Flink's 
queryable state, while it seems not well supported for SQL generated operators.

$ ./sst_dump --file=../db/30.sst --command=scan --read_num=50
options.env is 0xba33e0
Process ../db/30.sst
Sst file format: block-based
from [] to []
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>
'� =>

--
Best wishes,
- Kai

Inspecting SST state of rocksdb

[Kai Fu]

Hi team,

I'm trying to inspect SST files of flink's state with sst related tools
like sst_dump, ldb in wiki
.
But it seems I'm getting meaningless results as shown below. The tools I'm
using are from RocksDB's trunk and built from source. Am I doing it the
right way, or is there any alternative to inspect the state? We're aware of
Flink's queryable state, while it seems not well supported for SQL
generated operators.
















*$ ./sst_dump --file=../db/30.sst --command=scan
--read_num=50options.env is 0xba33e0Process ../db/30.sstSst file
format: block-basedfrom [] to []'� =>'� =>'� =>'� =>'� =>'� =>'� =>'� =>'�
=>'� =>'� =>*

-- 
*Best wishes,*
*- Kai*