[
https://issues.apache.org/jira/browse/IGNITE-23496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891744#comment-17891744
]
Roman Puchkovskiy commented on IGNITE-23496:
--------------------------------------------
The following was done in the attached PR:
# Don't use Java Serialization to serialize replication group anymore. It was
taking 102 bytes per lease, even though TablePartitionId only contains 2 ints.
# Use varints extensively
# Collect node names and IDs in a dictionary and use int indices (which are
small as there are usually much less nodes cluster than leases) to represent
lease holders and proposed candidates
# Use delta-coding to represent physical parts of timestamps
# Calculate expirationTime instead of storing it directly in most cases
# Don't store expirationTime logical part in most cases (as it's usually 0)
# Compress boolean flags to a flags field, also using that field to store
additional flags (like 'store expiration logical part')
# Don't store partition IDs, instead use their positions to represent
partition IDs (additionally using a special flag to encode possible holes in
partition ID sequences)
# Separate leases to groups by table, so only write table ID per table, not
per lease
# Use delta-coding to represent table IDs.
> Optimize lease batch serialization
> ----------------------------------
>
> Key: IGNITE-23496
> URL: https://issues.apache.org/jira/browse/IGNITE-23496
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Ignite nodes update leases each 2.5 seconds, in batches. A batch might
> contain dozens of thousands (or more) leases. The batch is serialized to
> bytes and stored in the Metastorage. Current seriailzation format is not too
> compact; for example, having 25000 partitions (and one lease per partition)
> in a 3-node cluster, we get approximately 4.7Mb per batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
