[
https://issues.apache.org/jira/browse/IGNITE-23496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891744#comment-17891744
]
Roman Puchkovskiy edited comment on IGNITE-23496 at 10/22/24 7:00 AM:
----------------------------------------------------------------------
The following was done in the attached PR:
# Don't use Java Serialization to serialize replication group anymore. It was
taking 102 bytes per lease, even though TablePartitionId only contains 2 ints.
# Use varints extensively
# Collect node names and IDs in a dictionary and use int indices (which are
small as there are usually much less nodes cluster than leases) to represent
lease holders and proposed candidates
# Use delta-coding to represent physical parts of timestamps
# Calculate expirationTime instead of storing it directly in most cases
# Don't store expirationTime logical part in most cases (as it's usually 0)
# Compress boolean flags to a flags field, also using that field to store
additional flags (like 'store expiration logical part')
# Don't store partition IDs, instead use their positions to represent
partition IDs (additionally using a special flag to encode possible holes in
partition ID sequences)
# Separate leases to groups by table, so only write table ID per table, not
per lease
# Use delta-coding to represent table IDs.
For the case of 1000 tables (25 partitions each) and 3 nodes in cluster, the
gain in the number of bytes is approximately 30 times:
{quote}Old 4712658 bytes, new 126503 bytes
{quote}
Performance has also improved (serializing and deserializing same batch of
25000 leases):
{{Benchmark Mode Cnt Score
Error Units
LeaseBatchSerializationBenchmark.deserializeNew avgt 15 1321,232 ±
73,867 us/op
LeaseBatchSerializationBenchmark.deserializeOld avgt 15 68862,028 ±
2273,113 us/op
LeaseBatchSerializationBenchmark.serializeNew avgt 15 4019,344 ±
201,223 us/op
LeaseBatchSerializationBenchmark.serializeOld avgt 15 21362,380 ±
335,240 us/op}}
was (Author: rpuch):
The following was done in the attached PR:
# Don't use Java Serialization to serialize replication group anymore. It was
taking 102 bytes per lease, even though TablePartitionId only contains 2 ints.
# Use varints extensively
# Collect node names and IDs in a dictionary and use int indices (which are
small as there are usually much less nodes cluster than leases) to represent
lease holders and proposed candidates
# Use delta-coding to represent physical parts of timestamps
# Calculate expirationTime instead of storing it directly in most cases
# Don't store expirationTime logical part in most cases (as it's usually 0)
# Compress boolean flags to a flags field, also using that field to store
additional flags (like 'store expiration logical part')
# Don't store partition IDs, instead use their positions to represent
partition IDs (additionally using a special flag to encode possible holes in
partition ID sequences)
# Separate leases to groups by table, so only write table ID per table, not
per lease
# Use delta-coding to represent table IDs.
> Optimize lease batch serialization
> ----------------------------------
>
> Key: IGNITE-23496
> URL: https://issues.apache.org/jira/browse/IGNITE-23496
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Ignite nodes update leases each 2.5 seconds, in batches. A batch might
> contain dozens of thousands (or more) leases. The batch is serialized to
> bytes and stored in the Metastorage. Current seriailzation format is not too
> compact; for example, having 25000 partitions (and one lease per partition)
> in a 3-node cluster, we get approximately 4.7Mb per batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
