We know Ignite has the function of automatic memory defragmentation:

https://ignite.apache.org/docs/latest/memory-architecture#memory-defragmentation

I want to ask what is the threshold for triggering this function?

在 2022/9/19 22:36, Alexander Polovtcev 写道:
Hm, I don't know, honestly. We need to look at the code or somebody else can give us a hint.

On Mon, Sep 19, 2022 at 2:55 PM 38797715 <[email protected]> wrote:

    if native persistence enabled, the result is the same.

    It seems that there is a bug in the calculation of the
    "PagesFillFactor" metrics?

    在 2022/9/19 17:30, Alexander Polovtcev 写道:
    Do you have native persistence enabled?

    On Mon, Sep 19, 2022 at 11:57 AM 38797715 <[email protected]> wrote:

        2.10 and 2.13, the same result.

        在 2022/9/19 16:54, Alexander Polovtcev 写道:
        Which version of Ignite do you use?

        On Mon, Sep 19, 2022 at 11:38 AM 38797715 <[email protected]>
        wrote:

            for example:

            start a node by ignite.sh.

            then:

            CREATE TABLE City (
              ID INT,
              Name VARCHAR,
              CountryCode CHAR(3),
              District VARCHAR,
              Population INT,
              PRIMARY KEY (ID, CountryCode)
            ) WITH "template=partitioned, backups=1,
            affinityKey=CountryCode, CACHE_NAME=City,
            KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City";


            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES (1,'Kabul','AFG','Kabol',1780000);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES (2,'Qandahar','AFG','Qandahar',237500);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES (3,'Herat','AFG','Herat',186800);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (4,'Mazar-e-Sharif','AFG','Balkh',127800);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (5,'Amsterdam','NLD','Noord-Holland',731200);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (6,'Rotterdam','NLD','Zuid-Holland',593321);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES (8,'Utrecht','NLD','Utrecht',234323);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (9,'Eindhoven','NLD','Noord-Brabant',201843);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (10,'Tilburg','NLD','Noord-Brabant',193238);
            INSERT INTO City(ID, Name, CountryCode, District,
            Population) VALUES
            (11,'Groningen','NLD','Groningen',172701);

            SELECT * FROM sys.metrics WHERE name LIKE
            'io.dataregion.default%';

            io.dataregion.default.OffHeapSize 104857600
            io.dataregion.default.PhysicalMemoryPages    2066
            io.dataregion.default.EmptyDataPages 0
            io.dataregion.default.UsedCheckpointBufferSize    0
            io.dataregion.default.TotalThrottlingTime    0
            io.dataregion.default.PagesReplaced    0
            io.dataregion.default.EvictionRate    0
            io.dataregion.default.InitialSize 104857600
            io.dataregion.default.DirtyPages    0
            io.dataregion.default.MaxSize 8589934592
            io.dataregion.default.PagesWritten    0
            io.dataregion.default.PagesReplaceRate    0
            io.dataregion.default.PagesRead    0
            io.dataregion.default.PagesFillFactor 0.9997031688690186
            io.dataregion.default.TotalAllocatedPages    2066
            io.dataregion.default.PhysicalMemorySize    8511920
            io.dataregion.default.PagesReplaceAge 0
            io.dataregion.default.AllocationRate 2066
            io.dataregion.default.OffheapUsedSize 8511920
            io.dataregion.default.LargeEntriesPagesCount    0
            io.dataregion.default.TotalAllocatedSize    8511920
            io.dataregion.default.CheckpointBufferSize    0

            TotalUsedSize - (TotalAllocatedSize - PagesFillFactor *
            TotalAllocatedSize)

            = 8511920 - (8511920 - 0.9997031688690186 * 8511920) =
            8509392


            delete from city;

            SELECT * FROM sys.metrics WHERE name LIKE
            'io.dataregion.default%';

            io.dataregion.default.OffHeapSize 104857600
            io.dataregion.default.PhysicalMemoryPages    2075
            io.dataregion.default.EmptyDataPages 0
            io.dataregion.default.UsedCheckpointBufferSize    0
            io.dataregion.default.TotalThrottlingTime    0
            io.dataregion.default.PagesReplaced    0
            io.dataregion.default.EvictionRate    0
            io.dataregion.default.InitialSize 104857600
            io.dataregion.default.DirtyPages    0
            io.dataregion.default.MaxSize 8589934592
            io.dataregion.default.PagesWritten    0
            io.dataregion.default.PagesReplaceRate    0
            io.dataregion.default.PagesRead    0
            io.dataregion.default.PagesFillFactor 1.0
            io.dataregion.default.TotalAllocatedPages    2075
            io.dataregion.default.PhysicalMemorySize    8549000
            io.dataregion.default.PagesReplaceAge 0
            io.dataregion.default.AllocationRate 9
            io.dataregion.default.OffheapUsedSize 8549000
            io.dataregion.default.LargeEntriesPagesCount    0
            io.dataregion.default.TotalAllocatedSize    8549000
            io.dataregion.default.CheckpointBufferSize    0


            TotalUsedSize - (TotalAllocatedSize - PagesFillFactor *
            TotalAllocatedSize)

            =8549000 - (8549000 - 1.0 * 8549000) = 8549000

            Instead, the value becomes larger?


            在 2022/9/19 15:13, Alexander Polovtcev 写道:
            Sorry, I messed up the metric names a little bit, we
            should use the size metrics, not the page metrics. So
            the correct formula would be: `TotalUsedSize -
            (TotalAllocatedSize - PagesFillFactor *
            TotalAllocatedSize)`

            On Sun, Sep 18, 2022 at 4:29 PM 38797715
            <[email protected]> wrote:

                I've checked TotalUsedPages in
                ignite.dataRegionMetrics(), it is found that the
                logic of this metric is consistent with
                OffheapUsedSize, that is, the value will not be
                updated after data deletion.

                在 2022/9/18 19:42, Alexander Polovtcev 写道:
                Hello, you can check out "TotalUsedPages" and
                "TotalUsedSize" metrics that can be found in the
                Data Region metrics. However, please keep in mind
                that this does not take page fragmentation into
                account.

                On Sat, Sep 17, 2022 at 2:35 PM 38797715
                <[email protected]> wrote:

                    Hi team,

                    We found that if the delete operation is
                    executed, the memory space corresponding to
                    the data will not be released, and this space
                    will be reused later.

                    Therefore, metrics such as OffheapUsedSize
                    will become inaccurate.

                    So, how to calculate the exact amount of
                    memory occupied by data in a cache or a
                    dataregion?



-- With regards,
                Aleksandr Polovtcev



-- With regards,
            Aleksandr Polovtcev



-- With regards,
        Aleksandr Polovtcev



-- With regards,
    Aleksandr Polovtcev



--
With regards,
Aleksandr Polovtcev

Reply via email to