[
https://issues.apache.org/jira/browse/IGNITE-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270170#comment-17270170
]
shivakumar edited comment on IGNITE-13912 at 1/22/21, 2:31 PM:
---------------------------------------------------------------
Hi [[email protected]]
I have attached the reproducer, when you unzip reproducer.zip then there will
be ignite-config.xml file which is used to install ignite cluster. In my setup
I deployed 5 node Ignite cluster with this configuration on kubernetes.
In this config file you can disable SSL, change ipFinder (ex: to vm ipFinder if
you install on Virtual machine).
The JVM OPTS set is
*-server -Xms32g -Xmx32g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC*
In the same reproducer there is a client program(inventory_data.zip) which
creates tables and ingest data, here is the steps to run the client program
once you installed the ignite cluster.
*unzip inventory_data.zip*
*cd inventory_data/*
*mvn clean package*
open this config file and update ignite_db_url
*vim src/main/resources/config.properties*
The config file look like this
{code:java}
ignite_jdbc_driver = org.apache.ignite.IgniteJdbcThinDriver
ignite_db_url =
jdbc:ignite:thin://ignite-service.default.svc.cluster.local:10800
ignite_db_url_parameters = distributedJoins=true
ignite_user = ignite
ignite_pass = ignite
generate_rows = 1000000 {code}
To run the program(This will connect to Ignite cluster and creates table and
ingest one million records)
{code:java}
java -cp target/inventory-1.0-SNAPSHOT.jar:target/libs/*
com.test.app.InventoryCreate src/main/resources/config.properties yes{code}
To continuously start data ingestion, call the same program in loop with 2nd
argument set to "*no*" (This argument will skip table deletion and re-creation
and directly starts data ingestion)
{code:java}
while true ; do java -cp target/inventory-1.0-SNAPSHOT.jar:target/libs/*
com.test.app.InventoryCreate src/main/resources/config.properties no; sleep 2s;
done{code}
Run this program for at-least 40 to 50 minutes and connect to visor shell in
between data ingestion (like every 5 minutes connect to visor shell and run
*cache -a* or *cache* command and be in visor for 2 minute and disconnect)
To connect to visor shell I'm using the same Ignite config file which is used
for server start.
*./ignitevisorcmd.sh -cfg=/opt/ignite/conf/ignite-config.xml*
Monitor the WAL usage and see if you can re-produce.
Please let me know if you need any help.
Regards,
Shiva
was (Author: shm):
Hi [[email protected]]
I have attached the reproducer, when you unzip reproducer.zip then there will
be ignite-config.xml file which is used to install ignite cluster. In my setup
I deployed 5 node Ignite cluster with this configuration on kubernetes.
In this config file you can disable SSL, change ipFinder (ex: to vm ipFinder if
you install on Virtual machine).
The JVM OPTS set is
*-server -Xms32g -Xmx32g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC*
In the same reproducer there is a client program(inventory_data.zip) which
creates tables and ingest data, here is the steps to run the client program
once you installed the ignite cluster.
*unzip inventory_data.zip*
*cd inventory_data/*
*mvn clean package*
open this config file and update ignite_db_url
*vim src/main/resources/config.properties*
The config file look like this
{code:java}
ignite_jdbc_driver = org.apache.ignite.IgniteJdbcThinDriver
ignite_db_url =
jdbc:ignite:thin://ignite-service.default.svc.cluster.local:10800
ignite_db_url_parameters = distributedJoins=true
ignite_user = ignite
ignite_pass = ignite
generate_rows = 1000000 {code}
To run the program(This will connect to Ignite cluster and creates table and
ingest one million records)
*java -cp target/inventory-1.0-SNAPSHOT.jar:target/libs/*
com.test.app.InventoryCreate src/main/resources/config.properties yes*
To continuously start data ingestion, call the same program in loop with 2nd
argument set to "*no*" (This argument will skip table deletion and re-creation
and directly starts data ingestion)
*while true ; do java -cp target/libs/*:target/inventory-1.0-SNAPSHOT.jar
com.test.app.InventoryCreate src/main/resources/config.properties no; sleep 2s;
done*
Run this program for at-least 40 to 50 minutes and connect to visor shell in
between data ingestion (like every 5 minutes connect to visor shell and run
*cache -a* or *cache* command and be in visor for 2 minute and disconnect)
To connect to visor shell I'm using the same Ignite config file which is used
for server start.
*./ignitevisorcmd.sh -cfg=/opt/ignite/conf/ignite-config.xml*
Monitor the WAL usage and see if you can re-produce.
Please let me know if you need any help.
Regards,
Shiva
> Incorrect calculation of WAL segments that should be deleted from WAL archive
> -----------------------------------------------------------------------------
>
> Key: IGNITE-13912
> URL: https://issues.apache.org/jira/browse/IGNITE-13912
> Project: Ignite
> Issue Type: Bug
> Components: persistence
> Reporter: Kirill Tkalenko
> Assignee: Kirill Tkalenko
> Priority: Critical
> Fix For: 2.10
>
> Attachments: reproducer.zip, server1-full-wal-checkpoint.log,
> wal-checkpoint-logs, wal_dir_contents, wal_grows_from_peak.PNG,
> wal_issue_reproduced.PNG, wal_usage.PNG, wal_usage_dec12.PNG,
> wal_usage_dec22nd_binary.PNG
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Now there is an incorrect calculation of WAL segments that should be deleted
> from WAL archive. Since we delete only those segments whose total size should
> not exceed *DataStorageConfiguration#maxWalArchiveSize *
> IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE*, but should be up to
> DataStorageConfiguration#maxWalArchiveSize *
> IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE*. Therefore, an excess of
> *DataStorageConfiguration#maxWalArchiveSize* occurs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
