[
https://issues.apache.org/jira/browse/HDDS-16953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-16953:
----------------------------------
Description:
`BasicRootedOzoneClientAdapterImpl.createFile` and `createStreamFile` (behind
`FileSystem.create` / `createNonRecursive`) call `getBucket(ofsPath,
recursive)` and then `OzoneBucket.createFile`. So every
`create` sends at least 2 OM requests: an `InfoBucket` followed by a
`CreateFile`, and a link bucket adds an `InfoVolume` and an `InfoBucket` for
each hop. On write-heavy workloads such as ingest, ETL output and Hive inserts,
`create` is the hottest write metadata operation.
The fetch does three jobs:
1. *{*}Layout check.{*}* `OzoneFSUtils.validateBucketLayout` rejects
OBJECT_STORE. `OMFileCreateRequest` is registered for
`BucketLayout.OBJECT_STORE`, so OM does not reject it.
2. *{*}Replication.{*}* The adapter passes `bucket.getReplicationConfig()` to
`OzoneClientUtils.resolveClientSideReplicationConfig`. *{*}If the bucket
default is EC, the bucket wins{*}*, because "there is no way for file system
interfaces to pass EC replication". Otherwise the FS-supplied factor wins, as
RATIS or as the client-configured type adjusted to that factor.
3. *{*}Auto-create.{*}* With `recursive=true` (always set by `CopyCommands`), a
missing volume or bucket is created.
*Proposal: opt-in fields that let OM apply the file system rule*
{code:java}
message CreateFileRequest {
required KeyArgs keyArgs = 1;
required bool isRecursive = 2;
required bool isOverwrite = 3;
optional uint64 clientID = 4;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 5;
// Set by file system clients, which cannot express EC replication: if the
// bucket default replication is EC, OM uses it instead of the replication
// in keyArgs.
optional bool preferBucketEcReplication = 6;
}
{code}
The second field reproduces the client rule exactly. The non-EC branch of
`resolveClientSideReplicationConfig` does not depend on the bucket. So the
client can compute it with
`bucketReplConfig = null` and send the result. Then:
* if the field is set and the bucket default is EC, OM uses the bucket default;
* otherwise OM uses the client value. If the client sent nothing, OM falls
back to the bucket default,
then the server default, through the existing logic.
That is the same decision the client makes today, with the inputs on the other
side of the wire.
*OM.* `OMFileCreateRequest.preExecute` already reads `bucketInfo`
(`ozoneManager.getBucketInfo`) to feed
`resolveReplicationConfigPreference`, before it calls `allocateBlock`. Both
checks go there, on the
leader, before Ratis, and *{*}before `allocateBlock`{*}*. Otherwise a rejected
`create` would still allocate
SCM blocks. Things to settle in review:
* whether `getBucketInfo` returns the *source* bucket's default replication
for a link bucket (UNVERIFIED);
* whether the layout check can use the factory's already-resolved layout
instead of re-deriving it.
This adds no extra lookup either way. Do not put either check in
`BucketLayoutAwareOMKeyRequestFactory.createRequest`, because that also runs at
apply time on every OM.
*Client.*
* When `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant, the
adapter sends `CreateFile` with
both fields set and does not call `getBucket()` first. Place the constant after
`LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is
assigned at merge. Against any
older OM, including ZDU, keep today's path.
* `RpcClient.createFile` builds the output stream from the replication
returned by the server. Check
that nothing between `createFile` and the stream relies on the client-side
value. `RpcClient` rejects
EC client-side for OM < 2; that is irrelevant here, because such an OM takes
the fallback.
* Auto-create on the error path, as in the mkdirs Jira. On `VOLUME_NOT_FOUND`
/ `BUCKET_NOT_FOUND` with
`recursive=true`, call `getBucket(ofsPath, true)` and retry once, using today's
replication path
because the bucket is now in hand. With `recursive=false`, propagate as today.
* Map `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`.
`FILE_ALREADY_EXISTS` / `NOT_A_FILE` keep
mapping to `FileAlreadyExistsException`.
*Other callers*
`CreateFile` has no S3 Gateway caller (S3 uses `CreateKey`). `freon
OmMetadataGenerator` calls it on any
layout. Neither sets the new fields, so their behavior is unchanged.
*Compatibility*
| |Behavior|
|—|—|
|New client, new OM, bucket exists|1 RPC; OBJECT_STORE rejected; EC bucket ⇒
EC, as today|
|New client, new OM, bucket missing (recursive)|failed `CreateFile` →
`getBucket(…, true)` → `CreateFile`|
|New client, older OM (incl. ZDU)|as today|
|Old client / freon / direct `createFile`, new OM|fields absent ⇒ unchanged|
There are two new optional proto fields, no new result code and no new
configuration.
*Testing*
* *{*}Replication decision table.{*}* In both an OM request test and the
adapter, cover these inputs:
* bucket default: EC, RATIS, or none;
* client-configured `ozone.replication`: unset, RATIS, or EC;
* FS factor: 1, 3, or unsupported;
* the field: set or unset.
For every combination, assert that the new path's effective replication equals
the old client-side
path's. Keep a copy of `resolveClientSideReplicationConfig` in the test as the
oracle.
* Adapter (Mockito):
* Existing bucket: exactly one `CreateFile` with both fields set, and no
`getBucket()`.
* Missing bucket with recursive: one failed call, then `getBucket(…, true)`,
then a retry.
* Missing bucket without recursive: the error propagates.
* Older OMs (including `ZDU`) fall back.
* OM request: with the field set, OBJECT_STORE is rejected *{*}without{*}*
`allocateBlock` being called;
FSO and *{*}LEGACY{*}* succeed. Without the field, OBJECT_STORE still succeeds.
A link to an EC bucket
resolves to EC.
* `AbstractRootedOzoneFileSystemTest`:
* create on an EC bucket writes EC;
* create into a missing bucket auto-creates it;
* create on an OBJECT_STORE bucket fails with `IllegalArgumentException`.
Count RPCs in the adapter test, not with OMMetrics deltas (HDDS-16513).
* Benchmark `TestOfsCreateRpcBenchmark`, same shape as
`TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets; create + close of empty files, single-threaded and 10
threads;
* n=3 per arm, built and run strictly one after the other;
* invariant: one fewer OM RPC per `create` (CreateFile + CommitKey; no
InfoBucket).
was:
`BasicRootedOzoneClientAdapterImpl.createFile` and `createStreamFile` (behind
`FileSystem.create` / `createNonRecursive`) call `getBucket(ofsPath,
recursive)` and then `OzoneBucket.createFile`. So every
`create` sends at least 2 OM requests: an `InfoBucket` followed by a
`CreateFile`, and a link bucket adds an `InfoVolume` and an `InfoBucket` for
each hop. On write-heavy workloads such as ingest, ETL output and Hive inserts,
`create` is the hottest write metadata operation.
The fetch does three jobs:
1. **Layout check.** `OzoneFSUtils.validateBucketLayout` rejects OBJECT_STORE.
`OMFileCreateRequest` is registered for `BucketLayout.OBJECT_STORE`, so OM does
not reject it.
2. **Replication.** The adapter passes `bucket.getReplicationConfig()` to
`OzoneClientUtils.resolveClientSideReplicationConfig`. **If the bucket default
is EC, the bucket wins**, because "there is no way for file system interfaces
to pass EC replication". Otherwise the FS-supplied factor wins, as RATIS or as
the client-configured type adjusted to that factor.
3. **Auto-create.** With `recursive=true` (always set by `CopyCommands`), a
missing volume or bucket is created.
### Proposal: opt-in fields that let OM apply the file system rule
{code}
message CreateFileRequest {
required KeyArgs keyArgs = 1;
required bool isRecursive = 2;
required bool isOverwrite = 3;
optional uint64 clientID = 4;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 5;
// Set by file system clients, which cannot express EC replication: if the
// bucket default replication is EC, OM uses it instead of the replication
// in keyArgs.
optional bool preferBucketEcReplication = 6;
}
{code}
The second field reproduces the client rule exactly. The non-EC branch of
`resolveClientSideReplicationConfig` does not depend on the bucket. So the
client can compute it with
`bucketReplConfig = null` and send the result. Then:
* if the field is set and the bucket default is EC, OM uses the bucket default;
* otherwise OM uses the client value. If the client sent nothing, OM falls back
to the bucket default,
then the server default, through the existing logic.
That is the same decision the client makes today, with the inputs on the other
side of the wire.
*OM.* `OMFileCreateRequest.preExecute` already reads `bucketInfo`
(`ozoneManager.getBucketInfo`) to feed
`resolveReplicationConfigPreference`, before it calls `allocateBlock`. Both
checks go there, on the
leader, before Ratis, and **before `allocateBlock`**. Otherwise a rejected
`create` would still allocate
SCM blocks. Things to settle in review:
* whether `getBucketInfo` returns the *source* bucket's default replication for
a link bucket (UNVERIFIED);
* whether the layout check can use the factory's already-resolved layout
instead of re-deriving it.
This adds no extra lookup either way. Do not put either check in
`BucketLayoutAwareOMKeyRequestFactory.createRequest`, because that also runs at
apply time on every OM.
*Client.*
* When `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant, the
adapter sends `CreateFile` with
both fields set and does not call `getBucket()` first. Place the constant after
`LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is
assigned at merge. Against any
older OM, including ZDU, keep today's path.
* `RpcClient.createFile` builds the output stream from the replication returned
by the server. Check
that nothing between `createFile` and the stream relies on the client-side
value. `RpcClient` rejects
EC client-side for OM < 2; that is irrelevant here, because such an OM takes
the fallback.
* Auto-create on the error path, as in the mkdirs Jira. On `VOLUME_NOT_FOUND` /
`BUCKET_NOT_FOUND` with
`recursive=true`, call `getBucket(ofsPath, true)` and retry once, using today's
replication path
because the bucket is now in hand. With `recursive=false`, propagate as today.
* Map `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`.
`FILE_ALREADY_EXISTS` / `NOT_A_FILE` keep
mapping to `FileAlreadyExistsException`.
### Other callers
`CreateFile` has no S3 Gateway caller (S3 uses `CreateKey`). `freon
OmMetadataGenerator` calls it on any
layout. Neither sets the new fields, so their behavior is unchanged.
### Compatibility
| | Behavior |
|---|---|
| New client, new OM, bucket exists | 1 RPC; OBJECT_STORE rejected; EC bucket ⇒
EC, as today |
| New client, new OM, bucket missing (recursive) | failed `CreateFile` →
`getBucket(…, true)` → `CreateFile` |
| New client, older OM (incl. ZDU) | as today |
| Old client / freon / direct `createFile`, new OM | fields absent ⇒ unchanged |
There are two new optional proto fields, no new result code and no new
configuration.
## Testing
* **Replication decision table.** In both an OM request test and the adapter,
cover these inputs:
* bucket default: EC, RATIS, or none;
* client-configured `ozone.replication`: unset, RATIS, or EC;
* FS factor: 1, 3, or unsupported;
* the field: set or unset.
For every combination, assert that the new path's effective replication equals
the old client-side
path's. Keep a copy of `resolveClientSideReplicationConfig` in the test as the
oracle.
* Adapter (Mockito):
* Existing bucket: exactly one `CreateFile` with both fields set, and no
`getBucket()`.
* Missing bucket with recursive: one failed call, then `getBucket(…, true)`,
then a retry.
* Missing bucket without recursive: the error propagates.
* Older OMs (including `ZDU`) fall back.
* OM request: with the field set, OBJECT_STORE is rejected **without**
`allocateBlock` being called;
FSO and **LEGACY** succeed. Without the field, OBJECT_STORE still succeeds. A
link to an EC bucket
resolves to EC.
* `AbstractRootedOzoneFileSystemTest`:
* create on an EC bucket writes EC;
* create into a missing bucket auto-creates it;
* create on an OBJECT_STORE bucket fails with `IllegalArgumentException`.
Count RPCs in the adapter test, not with OMMetrics deltas (HDDS-16513).
* Benchmark `TestOfsCreateRpcBenchmark`, same shape as
`TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets; create + close of empty files, single-threaded and 10
threads;
* n=3 per arm, built and run strictly one after the other;
* invariant: one fewer OM RPC per `create` (CreateFile + CommitKey; no
InfoBucket).
> Remove the per-call InfoBucket from rooted-ofs create by resolving file
> system replication in OM
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16953
> URL: https://issues.apache.org/jira/browse/HDDS-16953
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> `BasicRootedOzoneClientAdapterImpl.createFile` and `createStreamFile` (behind
> `FileSystem.create` / `createNonRecursive`) call `getBucket(ofsPath,
> recursive)` and then `OzoneBucket.createFile`. So every
> `create` sends at least 2 OM requests: an `InfoBucket` followed by a
> `CreateFile`, and a link bucket adds an `InfoVolume` and an `InfoBucket` for
> each hop. On write-heavy workloads such as ingest, ETL output and Hive
> inserts, `create` is the hottest write metadata operation.
> The fetch does three jobs:
> 1. *{*}Layout check.{*}* `OzoneFSUtils.validateBucketLayout` rejects
> OBJECT_STORE. `OMFileCreateRequest` is registered for
> `BucketLayout.OBJECT_STORE`, so OM does not reject it.
> 2. *{*}Replication.{*}* The adapter passes `bucket.getReplicationConfig()` to
> `OzoneClientUtils.resolveClientSideReplicationConfig`. *{*}If the bucket
> default is EC, the bucket wins{*}*, because "there is no way for file system
> interfaces to pass EC replication". Otherwise the FS-supplied factor wins, as
> RATIS or as the client-configured type adjusted to that factor.
> 3. *{*}Auto-create.{*}* With `recursive=true` (always set by `CopyCommands`),
> a missing volume or bucket is created.
> *Proposal: opt-in fields that let OM apply the file system rule*
> {code:java}
> message CreateFileRequest {
> required KeyArgs keyArgs = 1;
> required bool isRecursive = 2;
> required bool isOverwrite = 3;
> optional uint64 clientID = 4;
> // Set by file system clients: OM rejects the request with
> // NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
> optional bool requireFileSystemLayout = 5;
> // Set by file system clients, which cannot express EC replication: if the
> // bucket default replication is EC, OM uses it instead of the replication
> // in keyArgs.
> optional bool preferBucketEcReplication = 6;
> }
> {code}
> The second field reproduces the client rule exactly. The non-EC branch of
> `resolveClientSideReplicationConfig` does not depend on the bucket. So the
> client can compute it with
> `bucketReplConfig = null` and send the result. Then:
> * if the field is set and the bucket default is EC, OM uses the bucket
> default;
> * otherwise OM uses the client value. If the client sent nothing, OM falls
> back to the bucket default,
> then the server default, through the existing logic.
> That is the same decision the client makes today, with the inputs on the
> other side of the wire.
> *OM.* `OMFileCreateRequest.preExecute` already reads `bucketInfo`
> (`ozoneManager.getBucketInfo`) to feed
> `resolveReplicationConfigPreference`, before it calls `allocateBlock`. Both
> checks go there, on the
> leader, before Ratis, and *{*}before `allocateBlock`{*}*. Otherwise a
> rejected `create` would still allocate
> SCM blocks. Things to settle in review:
> * whether `getBucketInfo` returns the *source* bucket's default replication
> for a link bucket (UNVERIFIED);
> * whether the layout check can use the factory's already-resolved layout
> instead of re-deriving it.
> This adds no extra lookup either way. Do not put either check in
> `BucketLayoutAwareOMKeyRequestFactory.createRequest`, because that also runs
> at apply time on every OM.
> *Client.*
> * When `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant, the
> adapter sends `CreateFile` with
> both fields set and does not call `getBucket()` first. Place the constant
> after
> `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is
> assigned at merge. Against any
> older OM, including ZDU, keep today's path.
> * `RpcClient.createFile` builds the output stream from the replication
> returned by the server. Check
> that nothing between `createFile` and the stream relies on the client-side
> value. `RpcClient` rejects
> EC client-side for OM < 2; that is irrelevant here, because such an OM takes
> the fallback.
> * Auto-create on the error path, as in the mkdirs Jira. On
> `VOLUME_NOT_FOUND` / `BUCKET_NOT_FOUND` with
> `recursive=true`, call `getBucket(ofsPath, true)` and retry once, using
> today's replication path
> because the bucket is now in hand. With `recursive=false`, propagate as today.
> * Map `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`.
> `FILE_ALREADY_EXISTS` / `NOT_A_FILE` keep
> mapping to `FileAlreadyExistsException`.
> *Other callers*
> `CreateFile` has no S3 Gateway caller (S3 uses `CreateKey`). `freon
> OmMetadataGenerator` calls it on any
> layout. Neither sets the new fields, so their behavior is unchanged.
> *Compatibility*
> | |Behavior|
> |—|—|
> |New client, new OM, bucket exists|1 RPC; OBJECT_STORE rejected; EC bucket ⇒
> EC, as today|
> |New client, new OM, bucket missing (recursive)|failed `CreateFile` →
> `getBucket(…, true)` → `CreateFile`|
> |New client, older OM (incl. ZDU)|as today|
> |Old client / freon / direct `createFile`, new OM|fields absent ⇒ unchanged|
> There are two new optional proto fields, no new result code and no new
> configuration.
> *Testing*
> * *{*}Replication decision table.{*}* In both an OM request test and the
> adapter, cover these inputs:
> * bucket default: EC, RATIS, or none;
> * client-configured `ozone.replication`: unset, RATIS, or EC;
> * FS factor: 1, 3, or unsupported;
> * the field: set or unset.
> For every combination, assert that the new path's effective replication
> equals the old client-side
> path's. Keep a copy of `resolveClientSideReplicationConfig` in the test as
> the oracle.
> * Adapter (Mockito):
> * Existing bucket: exactly one `CreateFile` with both fields set, and no
> `getBucket()`.
> * Missing bucket with recursive: one failed call, then `getBucket(…, true)`,
> then a retry.
> * Missing bucket without recursive: the error propagates.
> * Older OMs (including `ZDU`) fall back.
> * OM request: with the field set, OBJECT_STORE is rejected *{*}without{*}*
> `allocateBlock` being called;
> FSO and *{*}LEGACY{*}* succeed. Without the field, OBJECT_STORE still
> succeeds. A link to an EC bucket
> resolves to EC.
> * `AbstractRootedOzoneFileSystemTest`:
> * create on an EC bucket writes EC;
> * create into a missing bucket auto-creates it;
> * create on an OBJECT_STORE bucket fails with `IllegalArgumentException`.
> Count RPCs in the adapter test, not with OMMetrics deltas (HDDS-16513).
> * Benchmark `TestOfsCreateRpcBenchmark`, same shape as
> `TestOfsListStatusRpcBenchmark`:
> * 200 FSO buckets; create + close of empty files, single-threaded and 10
> threads;
> * n=3 per arm, built and run strictly one after the other;
> * invariant: one fewer OM RPC per `create` (CreateFile + CommitKey; no
> InfoBucket).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
