[
https://issues.apache.org/jira/browse/HDDS-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793239#comment-17793239
]
Pratyush Bhatt commented on HDDS-9840:
--------------------------------------
Tried for 500KB, 800KB and 900KB approx.
500KB:
{code:java}
ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path
ofs://ozone1701449533/vol3/bucket3/dfsg -s 502400 -t 32 --sync HSYNC -p sync222
23/12/05 11:24:05 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:24:05 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
23/12/05 11:24:05 INFO impl.MetricsSystemImpl: ozone-freon metrics system
started
23/12/05 11:24:05 INFO freon.BaseFreonGenerator: Executing test with prefix
sync222 and number-of-tests 1 0.00% |?
| 0/1 Time: 0:00:00|
Disabling FS cache: fs.ofs.impl.disable.cache
0.00% |?
| 0/1 Time: 0:00:01|
23/12/05 11:24:06 WARN impl.MetricsSystemImpl: ozone-freon metrics system
already initialized!
23/12/05 11:24:06 INFO metrics.MetricRegistries: Loaded MetricRegistries class
org.apache.ratis.metrics.impl.MetricRegistriesImpl
12/5/23 11:24:22 AM
============================================================-- Timers
----------------------------------------------------------------------
file-create
count = 1
mean rate = 0.06 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 16072.72 milliseconds
max = 16072.72 milliseconds
mean = 16072.72 milliseconds
stddev = 0.00 milliseconds
median = 16072.72 milliseconds
75% <= 16072.72 milliseconds
95% <= 16072.72 milliseconds
98% <= 16072.72 milliseconds
99% <= 16072.72 milliseconds
99.9% <= 16072.72 milliseconds
Total execution time (sec): 17
Failures: 0
Successful executions: 1 {code}
800KB:
{code:java}
ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path
ofs://ozone1701449533/vol3/bucket3/dfsg -s 819200 -t 32 --sync HSYNC -p sync222
23/12/05 11:24:46 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:24:46 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
23/12/05 11:24:46 INFO impl.MetricsSystemImpl: ozone-freon metrics system
started
23/12/05 11:24:46 INFO freon.BaseFreonGenerator: Executing test with prefix
sync222 and number-of-tests 1 0.00% |?
| 0/1 Time: 0:00:00|
Disabling FS cache: fs.ofs.impl.disable.cache
0.00% |?
| 0/1 Time: 0:00:01|
23/12/05 11:24:47 WARN impl.MetricsSystemImpl: ozone-freon metrics system
already initialized!
23/12/05 11:24:47 INFO metrics.MetricRegistries: Loaded MetricRegistries class
org.apache.ratis.metrics.impl.MetricRegistriesImpl
12/5/23 11:25:13 AM
============================================================-- Timers
----------------------------------------------------------------------
file-create
count = 1
mean rate = 0.04 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 25797.62 milliseconds
max = 25797.62 milliseconds
mean = 25797.62 milliseconds
stddev = 0.00 milliseconds
median = 25797.62 milliseconds
75% <= 25797.62 milliseconds
95% <= 25797.62 milliseconds
98% <= 25797.62 milliseconds
99% <= 25797.62 milliseconds
99.9% <= 25797.62 milliseconds
Total execution time (sec): 27
Failures: 0
Successful executions: 1 {code}
900KB:
{code:java}
ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path
ofs://ozone1701449533/vol3/bucket3/dfsg -s 921600 -t 32 --sync HSYNC -p sync2222
23/12/05 11:25:35 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:25:35 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
23/12/05 11:25:35 INFO impl.MetricsSystemImpl: ozone-freon metrics system
started
23/12/05 11:25:35 INFO freon.BaseFreonGenerator: Executing test with prefix
sync2222 and number-of-tests 1 0.00% |?
| 0/1 Time:
0:00:00| Disabling FS cache: fs.ofs.impl.disable.cache
0.00% |?
| 0/1 Time: 0:00:01|
23/12/05 11:25:36 WARN impl.MetricsSystemImpl: ozone-freon metrics system
already initialized!
23/12/05 11:25:37 INFO metrics.MetricRegistries: Loaded MetricRegistries class
org.apache.ratis.metrics.impl.MetricRegistriesImpl
12/5/23 11:28:10 AM
============================================================-- Timers
----------------------------------------------------------------------
file-create
count = 1
mean rate = 0.01 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 153554.19 milliseconds
max = 153554.19 milliseconds
mean = 153554.19 milliseconds
stddev = 0.00 milliseconds
median = 153554.19 milliseconds
75% <= 153554.19 milliseconds
95% <= 153554.19 milliseconds
98% <= 153554.19 milliseconds
99% <= 153554.19 milliseconds
99.9% <= 153554.19 milliseconds
Total execution time (sec): 155
Failures: 0
Successful executions: 1 {code}
> ozone freon dfsg takes very long time when HSYNC is enabled
> -----------------------------------------------------------
>
> Key: HDDS-9840
> URL: https://issues.apache.org/jira/browse/HDDS-9840
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone CLI
> Reporter: Pratyush Bhatt
> Priority: Major
>
> The command generates 1 100MB file in about 3 seconds when HSYNC is not
> passed.
> {code:java}
> ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path
> ofs://ozone1701449533/vol2/bucket2/dfsg -s 102400000 -t 32 -p nonsync
> 23/12/05 11:11:29 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
> 23/12/05 11:11:29 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 23/12/05 11:11:29 INFO impl.MetricsSystemImpl: ozone-freon metrics system
> started
> 23/12/05 11:11:29 INFO freon.BaseFreonGenerator: Executing test with prefix
> nonsync and number-of-tests 1 0.00% |?
> | 0/1 Time:
> 0:00:00| Disabling FS cache: fs.ofs.impl.disable.cache
> 0.00% |?
> | 0/1 Time: 0:00:01| 23/12/05 11:11:30 INFO
> rpc.RpcClient: Creating Bucket: vol2/bucket2, with bucket layout
> FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to
> DISK and Encryption set to false, Replication Type set to server-side default
> replication type, Namespace Quota set to -1, Space Quota set to -1
> 23/12/05 11:11:31 WARN impl.MetricsSystemImpl: ozone-freon metrics system
> already initialized!
> 23/12/05 11:11:31 INFO metrics.MetricRegistries: Loaded MetricRegistries
> class org.apache.ratis.metrics.impl.MetricRegistriesImpl
> 0.00% |?
> | 0/1 Time: 0:00:02|
> 12/5/23 11:11:33 AM
> ============================================================-- Timers
> ----------------------------------------------------------------------
> file-create
> count = 1
> mean rate = 0.46 calls/second
> 1-minute rate = 0.00 calls/second
> 5-minute rate = 0.00 calls/second
> 15-minute rate = 0.00 calls/second
> min = 2018.26 milliseconds
> max = 2018.26 milliseconds
> mean = 2018.26 milliseconds
> stddev = 0.00 milliseconds
> median = 2018.26 milliseconds
> 75% <= 2018.26 milliseconds
> 95% <= 2018.26 milliseconds
> 98% <= 2018.26 milliseconds
> 99% <= 2018.26 milliseconds
> 99.9% <= 2018.26 milliseconds
> Total execution time (sec): 3
> Failures: 0
> Successful executions: 1 {code}
> But when HSYNC flag is passed, the same command takes 161 seconds even for a
> single 1MB file.
> {code:java}
> ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path
> ofs://ozone1701449533/vol3/bucket3/dfsg -s 1024000 -t 32 --sync HSYNC -p sync2
> 23/12/05 11:17:50 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
> 23/12/05 11:17:50 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 23/12/05 11:17:50 INFO impl.MetricsSystemImpl: ozone-freon metrics system
> started
> 23/12/05 11:17:50 INFO freon.BaseFreonGenerator: Executing test with prefix
> sync2 and number-of-tests 1
> 0.00% |?
> | 0/1 Time: 0:00:00| Disabling FS cache:
> fs.ofs.impl.disable.cache
> 0.00% |?
> | 0/1 Time: 0:00:01| 23/12/05 11:17:51 WARN
> impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
> 23/12/05 11:17:51 INFO metrics.MetricRegistries: Loaded MetricRegistries
> class org.apache.ratis.metrics.impl.MetricRegistriesImpl
> 0.00% |?
> | 0/1 Time: 0:00:02|
> 12/5/23 11:20:31 AM
> ============================================================
> -- Timers
> ----------------------------------------------------------------------
> file-create
> count = 1
> mean rate = 0.01 calls/second
> 1-minute rate = 0.00 calls/second
> 5-minute rate = 0.00 calls/second
> 15-minute rate = 0.00 calls/second
> min = 160038.05 milliseconds
> max = 160038.05 milliseconds
> mean = 160038.05 milliseconds
> stddev = 0.00 milliseconds
> median = 160038.05 milliseconds
> 75% <= 160038.05 milliseconds
> 95% <= 160038.05 milliseconds
> 98% <= 160038.05 milliseconds
> 99% <= 160038.05 milliseconds
> 99.9% <= 160038.05 milliseconds
> Total execution time (sec): 161
> Failures: 0
> Successful executions: 1ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n
> 1 --path ofs://ozone1701449533/vol3/bucket3/dfsg -s 1024000 -t 32 --sync
> HSYNC -p sync2
> 23/12/05 11:17:50 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
> 23/12/05 11:17:50 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 23/12/05 11:17:50 INFO impl.MetricsSystemImpl: ozone-freon metrics system
> started
> 23/12/05 11:17:50 INFO freon.BaseFreonGenerator: Executing test with prefix
> sync2 and number-of-tests 1 0.00% |?
> | 0/1 Time:
> 0:00:00| Disabling FS cache: fs.ofs.impl.disable.cache
> 0.00% |?
> | 0/1 Time: 0:00:01| 23/12/05 11:17:51 WARN
> impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
> 23/12/05 11:17:51 INFO metrics.MetricRegistries: Loaded MetricRegistries
> class org.apache.ratis.metrics.impl.MetricRegistriesImpl
> 0.00% |?
> | 0/1 Time: 0:00:02|
> 12/5/23 11:20:31 AM
> ============================================================-- Timers
> ----------------------------------------------------------------------
> file-create
> count = 1
> mean rate = 0.01 calls/second
> 1-minute rate = 0.00 calls/second
> 5-minute rate = 0.00 calls/second
> 15-minute rate = 0.00 calls/second
> min = 160038.05 milliseconds
> max = 160038.05 milliseconds
> mean = 160038.05 milliseconds
> stddev = 0.00 milliseconds
> median = 160038.05 milliseconds
> 75% <= 160038.05 milliseconds
> 95% <= 160038.05 milliseconds
> 98% <= 160038.05 milliseconds
> 99% <= 160038.05 milliseconds
> 99.9% <= 160038.05 milliseconds
> Total execution time (sec): 161
> Failures: 0
> Successful executions: 1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
