zy-kkk opened a new pull request, #59103:
URL: https://github.com/apache/doris/pull/59103
## Summary
Add memory size monitoring for Hive file cache to help diagnose memory
usage issues.
- Add `hive_file_cache_memory_bytes` Prometheus metric per catalog
- Add `memory_size_bytes` field to `catalog_meta_cache_statistics` system
table
- Implement `getRetainedSizeInBytes()` for `FileCacheKey`,
`FileCacheValue`, and `HiveFileStatus`
## Usage
**System table:**
```sql
SELECT * FROM information_schema.catalog_meta_cache_statistics
WHERE cache_name = 'hive_file_cache' and CATALOG_NAME = 'hive' and
METRIC_NAME = 'memory_size_bytes';
Prometheus:
curl -s http://localhost:8030/metrics | grep
'hive_file_cache_memory_bytes{catalog="hive"}'
Test
-- Query partition to load cache
SELECT * FROM hive.zyk.partition_table2 WHERE pt1 = '10000';
-- Check memory: 996 bytes
SELECT * FROM information_schema.catalog_meta_cache_statistics WHERE
cache_name = 'hive_file_cache' and CATALOG_NAME = 'hive' and METRIC_NAME =
'memory_size_bytes';
+--------------+-----------------+-------------------+--------------+
| CATALOG_NAME | CACHE_NAME | METRIC_NAME | METRIC_VALUE |
+--------------+-----------------+-------------------+--------------+
| hive | hive_file_cache | memory_size_bytes | 996 |
+--------------+-----------------+-------------------+--------------+
-- Query another partition
SELECT * FROM hive.zyk.partition_table2 WHERE pt1 = '10001';
-- Check memory: 1992 bytes (doubled)
SELECT * FROM information_schema.catalog_meta_cache_statistics WHERE
cache_name = 'hive_file_cache' and CATALOG_NAME = 'hive' and METRIC_NAME =
'memory_size_bytes';
+--------------+-----------------+-------------------+--------------+
| CATALOG_NAME | CACHE_NAME | METRIC_NAME | METRIC_VALUE |
+--------------+-----------------+-------------------+--------------+
| hive | hive_file_cache | memory_size_bytes | 1992 |
+--------------+-----------------+-------------------+--------------+
-- Refresh catalog
REFRESH CATALOG hive;
-- Check memory: 0 bytes (cleared)
SELECT * FROM information_schema.catalog_meta_cache_statistics WHERE
cache_name = 'hive_file_cache' and CATALOG_NAME = 'hive' and METRIC_NAME =
'memory_size_bytes';
+--------------+-----------------+-------------------+--------------+
| CATALOG_NAME | CACHE_NAME | METRIC_NAME | METRIC_VALUE |
+--------------+-----------------+-------------------+--------------+
| hive | hive_file_cache | memory_size_bytes | 0 |
+--------------+-----------------+-------------------+--------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
