sandynz commented on pull request #14978:
URL: https://github.com/apache/shardingsphere/pull/14978#issuecomment-1018253837
It's found in a simple performance test.
Table `sbtest1` with 20 millions lines of records that's generated by
sysbench.
Scaling job ran on 4G heap JVM.
It throw exception on inventory dump:
```
[ERROR] 2022-01-21 11:58:23.078 [ShardingSphere-Scaling-execute-4]
o.a.s.d.p.core.task.InventoryTask - get an error when migrating the inventory
data
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
at java.util.HashMap.putVal(HashMap.java:631)
at java.util.HashMap.put(HashMap.java:612)
at
org.apache.shardingsphere.driver.executor.batch.BatchExecutionUnit.mapAddBatchCount(BatchExecutionUnit.java:56)
at
org.apache.shardingsphere.driver.executor.batch.BatchPreparedStatementExecutor.reviseBatchExecutionUnit(BatchPreparedStatementExecutor.java:118)
```
Most memory was consumed by `toBeAckRecords` which is instanceof
`ConcurrentLinkedDeque` in `AbstractBitSetChannel`. It's remove for now,
`AckCallback` is enough for now.
On the latest code, it works, test result:
```
mysql> show scaling list;
+-----------------+---------+----------------------+--------+---------------------+---------------------+
| id | tables | sharding_total_count | active | create_time
| stop_time |
+-----------------+---------+----------------------+--------+---------------------+---------------------+
| 223261682468932 | sbtest1 | 1 | false | 2022-01-21
14:51:00 | 2022-01-21 15:04:45 |
+-----------------+---------+----------------------+--------+---------------------+---------------------+
1 row in set (0.01 sec)
mysql> show scaling status 223261682468932;
+------+-------------+----------+--------+-------------------------------+--------------------------+
| item | data_source | status | active | inventory_finished_percentage |
incremental_idle_minutes |
+------+-------------+----------+--------+-------------------------------+--------------------------+
| 0 | ds_0 | FINISHED | false | 100 | 4
|
+------+-------------+----------+--------+-------------------------------+--------------------------+
1 row in set (0.01 sec)
mysql> check scaling 223261682468932 by type (name=CRC32_MATCH);
+------------+----------------------+----------------------+-----------------------+-------------------------+
| table_name | source_records_count | target_records_count |
records_count_matched | records_content_matched |
+------------+----------------------+----------------------+-----------------------+-------------------------+
| sbtest1 | 22762887 | 22762887 | true
| true |
+------------+----------------------+----------------------+-----------------------+-------------------------+
1 row in set (1 min 47.23 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
