smengcl commented on PR #3603:
URL: https://github.com/apache/ozone/pull/3603#issuecomment-1190960680
> Thanks for this PR @DaveTeng0. As we discussed in the community meeting,
@errose28 , can we verify the ranger client integration works well with the
s3gateway with Grpc transport? Run the CI with Grpc s3gateway settings:
`ozone.om.s3.grpc.server_enabled = true` `ozone.om.transport.class =
org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory`
>
> Also does the update to 2.3.0 work well both with and without
`multi-tenant` enabled?
Hey @neils-dev , sorry for the wait. We were dealing with an issue on our
end:
We bumped `netty` (from 4.1.74) to 4.1.77/4.1.78 internally without bumping
`netty-tcnative`. Our build system failed to pick up netty-tcnative 2.0.48 jar
correctly in the resulting package, leading to missing `netty_tcnative` library
during OM startup (found when netty debugging is enabled). As a result the
client timed out reaching any of the OMs when `hdds.grpc.tls.enabled` and
`ozone.om.s3.grpc.server_enabled` are both set to true. Now the issue is
resolved by bumping netty-tcnative to 2.0.52 (just a few hours ago. Thanks
@adoroszlai ).
I have briefly tested S3 gateway with and without a tenant (default s3v
volume). Works as expected.
Env:
1. `hdds.grpc.tls.enabled=true` is set cluster-wide
2. `ozone.om.s3.grpc.server_enabled=true` is set on OM (only affects OM)
3.
`ozone.om.transport.class=org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory`
is set on S3g
4. Restarted Ozone service and confirmed the configs were applied in leader
OM and S3g's `/conf` endpoint
```bash
$ kinit -kt /path/to/om.keytab om
$ ozone getconf confKey hdds.grpc.tls.enabled
true
$ ozone tenant create tenant1 --om-service-id=ozone1
22/07/21 01:44:14 INFO rpc.RpcClient: Creating Tenant: 'tenant1', with new
volume: 'tenant1'
$ ozone tenant user assign --tenant=tenant1 hive --om-service-id=ozone1
export AWS_ACCESS_KEY_ID='tenant1$hive'
export AWS_SECRET_ACCESS_KEY='<RANDOMACCESSKEY>'
$ kdestroy
$ export AWS_ACCESS_KEY_ID='tenant1$hive'
$ export AWS_SECRET_ACCESS_KEY='<RANDOMACCESSKEY>'
$ alias awsc='aws s3api --endpoint https://<S3G>:9879 --ca-bundle
/path/to/cacerts.pem'
$ awsc list-buckets
{
"Buckets": []
}
$ awsc create-bucket --bucket buck1
{
"Location": "https://<S3G>:9879/buck1"
}
$ awsc list-buckets
{
"Buckets": [
{
"Name": "buck1",
"CreationDate": "2022-07-21T01:49:23.022000+00:00"
}
]
}
$ awsc list-objects --bucket buck1
$ awsc put-object --bucket buck1 --key awscliv2-uploaded.zip --body
awscliv2.zip
$ awsc list-objects --bucket buck1
{
"Contents": [
{
"Key": "awscliv2-uploaded.zip",
"LastModified": "2022-07-21T01:54:12.548000+00:00",
"ETag": "2022-07-21T01:54:12.548Z",
"Size": 47048038,
"StorageClass": "STANDARD"
}
]
}
$ awsc get-object --bucket buck1 --key awscliv2-uploaded.zip awscliv2-got.zip
{
"AcceptRanges": "bytes",
"LastModified": "2022-07-21T01:54:12+00:00",
"ContentLength": 47048038,
"CacheControl": "no-cache",
"ContentType": "application/octet-stream",
"Expires": "2022-07-21T01:54:45+00:00",
"Metadata": {}
}
$ sha256sum *.zip
bb8f11423aaa00be3a18f2cbf301d1d835e3ab17f0d91404ef5ee627ef216e58
awscliv2-got.zip
bb8f11423aaa00be3a18f2cbf301d1d835e3ab17f0d91404ef5ee627ef216e58
awscliv2.zip
```
```
$ kinit -kt /path/to/om.keytab om
$ ozone sh bucket list /tenant1
[ {
"metadata" : { },
"volumeName" : "tenant1",
"name" : "buck1",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 141144114,
"usedNamespace" : 1,
"creationTime" : "2022-07-21T01:49:23.022Z",
"modificationTime" : "2022-07-21T01:49:23.022Z",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "LEGACY",
"owner" : "hive",
"link" : false
} ]
$ ozone sh key list /tenant1/buck1
[ {
"volumeName" : "tenant1",
"bucketName" : "buck1",
"name" : "awscliv2-uploaded.zip",
"dataSize" : 47048038,
"creationTime" : "2022-07-21T01:54:11.868Z",
"modificationTime" : "2022-07-21T01:54:12.548Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
$ ozone fs -ls -R ofs://ozone1/tenant1/
drwxrwxrwx - om om 0 2022-07-21 01:49 ofs://ozone1/tenant1/buck1
-rw-rw-rw- 3 om om 47048038 2022-07-21 01:54
ofs://ozone1/tenant1/buck1/awscliv2-uploaded.zip
# This is a known issue in displaying current logged in user as the
bucket/key owner while it isn't.
```
Similarly tested with default `s3v` volume, works fine. The RangerClient
version used here is an internal 2.3.0-based build.
Thanks,
Siyao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
