steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1006605173
## dependencies
dependencies look good; nothing accidentally referred to.
```
[INFO] +- org.apache.hadoop:hadoop-common:test-jar:tests:3.4.0-SNAPSHOT:test
[INFO] +- com.amazonaws:aws-java-sdk-bundle:jar:1.12.132:compile
[INFO] +- org.assertj:assertj-core:jar:3.12.2:test
```
size is now 264 MB, up from 226. that is huge, but it allows us to dodge all
classpath problems we used to get about version incompatibilities of things
like json parsing, http client etc.
## Test run with `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep
-Dscale`
ITestS3AInputStreamPerformance output shows a bug in the throughput
calculation; no new errors from the SDK though.
```
2022-01-05 17:32:29,352[JUnit-testReadWithNormalPolicy]INFO
scale.ITestS3AInputStreamPerformance(ITestS3AInputStreamPerformance.java:executeSeekReadSequence(437))-Effective
bandwidth-3.478002MB/S
```
## Test run with `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete
-Ddynamo -Ds3guard`
```
[ERROR] Failures:
[ERROR]
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:268->Assert.assertTrue:42->Assert.fail:89
treewalk vs listFiles(/, true) mismatch: between
"s3a://stevel-london/file.txt"
] and
"s3a://stevel-london/dir-no-trailing/file2"
"s3a://stevel-london/file.txt"
"s3a://stevel-london/p1/p2/file"
]
[INFO]
[ERROR] Tests run: 151, Failures: 1, Errors: 0, Skipped: 82
```
## Test run with `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete
-Ddynamo -Ds3guard -Dscale`
This time `ITestS3AContractRootDir` was happy.
## manual command line tests
All executed against AWS London with no problems in any of the commands. as
S3Guard was disabled, I skipped some of the commands.
```
~/P/R/hadoop-3.4.0-SNAPSHOT bin/hadoop s3guard bucket-info -markers aware
$BUCKET
2022-01-06 13:37:48,612 [main] INFO Configuration.deprecation
(Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption.key
is deprecated. Instead, use fs.s3a.encryption.key
2022-01-06 13:37:48,615 [main] INFO Configuration.deprecation
(Configuration.java:logDeprecation(1459)) -
fs.s3a.server-side-encryption-algorithm is deprecated. Instead, use
fs.s3a.encryption.algorithm
2022-01-06 13:37:49,759 [main] INFO impl.DirectoryPolicyImpl
(DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be
kept
Filesystem s3a://stevel-london
Location: eu-west-2
Filesystem s3a://stevel-london is not using S3Guard
S3A Client
Signing Algorithm: fs.s3a.signing-algorithm=(unset)
Endpoint: fs.s3a.endpoint=(unset)
Encryption: fs.s3a.encryption.algorithm=SSE-KMS
Input seek policy: fs.s3a.experimental.input.fadvise=normal
Change Detection Source: fs.s3a.change.detection.source=etag
Change Detection Mode: fs.s3a.change.detection.mode=server
S3A Committers
The "magic" committer is supported in the filesystem
S3A Committer factory class:
mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
S3A Committer name: fs.s3a.committer.name=magic
Store magic committer integration:
fs.s3a.committer.magic.enabled=true
Security
Delegation token support is disabled
Security
The directory marker policy is "keep"
Available Policies: delete, keep, authoritative
Authoritative paths: fs.s3a.authoritative.path=
The S3A connector is compatible with buckets where directory markers
are not deleted
2022-01-06 13:37:50,622 [main] INFO statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((audit_request_execution=1)
(audit_span_creation=2)
(store_exists_probe=1)
(store_io_request=2));
gauges=();
minimums=((store_exists_probe.min=842));
maximums=((store_exists_probe.max=842));
means=((store_exists_probe.mean=(samples=1, sum=842, mean=842.0000)));
```
I'm using the old properties for encryption because I still want tests
against older builds to also be encrypted. Maybe I should set both properties
just to get rid of the warning.
and a storediag command
```
bin/hadoop jar $CLOUDSTORE storediag $BUCKET
Store Diagnostics for stevel (auth:SIMPLE) on stevel-mbp1376/127.0.0.1
======================================================================
Diagnostics for filesystem s3a://stevel-london/
===============================================
S3A FileSystem Connector
ASF Filesystem Connector to Amazon S3 Storage and compatible stores
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
Hadoop information
==================
Hadoop 3.4.0-SNAPSHOT
Compiled by stevel on 2022-01-05T18:07Z
Compiled with protoc 3.7.1
From source with checksum e249b256a31f772e4220f92668721349
Determining OS version
======================
Darwin stevel-mbp1376 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28
20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
Selected System Properties
==========================
aws.accessKeyId = (unset)
aws.secretKey = (unset)
aws.sessionToken = (unset)
aws.region = (unset)
com.amazonaws.regions.RegionUtils.fileOverride = (unset)
com.amazonaws.regions.RegionUtils.disableRemote = (unset)
com.amazonaws.sdk.disableCertChecking = (unset)
com.amazonaws.sdk.ec2MetadataServiceEndpointOverride = (unset)
com.amazonaws.sdk.enableDefaultMetrics = (unset)
com.amazonaws.sdk.enableInRegionOptimizedMode = (unset)
com.amazonaws.sdk.enableThrottledRetry = (unset)
com.amazonaws.services.s3.disableImplicitGlobalClients = (unset)
com.amazonaws.services.s3.enableV4 = (unset)
com.amazonaws.services.s3.enforceV4 = (unset)
Environment Variables
=====================
AWS_ACCESS_KEY_ID = (unset)
AWS_ACCESS_KEY = (unset)
AWS_SECRET_KEY = (unset)
AWS_SECRET_ACCESS_KEY = (unset)
AWS_SESSION_TOKEN = (unset)
AWS_REGION = (unset)
AWS_S3_US_EAST_1_REGIONAL_ENDPOINT = (unset)
AWS_CBOR_DISABLE = (unset)
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI = (unset)
AWS_CONTAINER_CREDENTIALS_FULL_URI = (unset)
AWS_CONTAINER_AUTHORIZATION_TOKEN = (unset)
AWS_EC2_METADATA_DISABLED = "true"
AWS_EC2_METADATA_SERVICE_ENDPOINT = (unset)
AWS_MAX_ATTEMPTS = (unset)
AWS_RETRY_MODE = (unset)
HADOOP_CONF_DIR =
"/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/etc/hadoop"
HADOOP_CREDSTORE_PASSWORD = (unset)
HADOOP_HEAPSIZE = (unset)
HADOOP_HEAPSIZE_MIN = (unset)
HADOOP_HOME = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT"
HADOOP_LOG_DIR = (unset)
HADOOP_OPTIONAL_TOOLS = "hadoop-azure,hadoop-aws,hadoop-openstack"
HADOOP_OPTS = "-Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs
-Dyarn.log.file=hadoop.log
-Dyarn.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT
-Dyarn.root.logger=INFO,console
-Dhadoop.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT
-Dhadoop.id.str=stevel -Dhadoop.root.logger=INFO,console
-Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender"
HADOOP_SHELL_SCRIPT_DEBUG = (unset)
HADOOP_TOKEN = (unset)
HADOOP_TOKEN_FILE_LOCATION = (unset)
HADOOP_TOOLS_HOME = (unset)
HADOOP_TOOLS_OPTIONS = (unset)
HADOOP_YARN_HOME = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT"
HDP_VERSION = (unset)
JAVA_HOME = "/Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home"
LOCAL_DIRS = (unset)
OPENSSL_ROOT_DIR = "/usr/local/opt/openssl/"
PYSPARK_DRIVER_PYTHON = (unset)
SPARK_HOME = (unset)
SPARK_CONF_DIR = (unset)
SPARK_SCALA_VERSION = (unset)
YARN_CONF_DIR = (unset)
Security
========
Security Enabled: false
Keytab login: false
Ticket login: false
Current user: stevel (auth:SIMPLE)
Token count: 0
Hadoop Options
==============
fs.defaultFS = "file:///" [core-default.xml]
2022-01-06 13:41:51,986 [main] INFO Configuration.deprecation
(Configuration.java:logDeprecation(1459)) - fs.default.name is deprecated.
Instead, use fs.defaultFS
fs.default.name = "file:///"
fs.permissions.umask-mode = "022" [core-default.xml]
fs.trash.classname = (unset)
fs.trash.interval = "0" [core-default.xml]
fs.trash.checkpoint.interval = "0" [core-default.xml]
hadoop.tmp.dir = "/tmp/hadoop-stevel" [core-default.xml]
hdp.version = (unset)
yarn.resourcemanager.address = "0.0.0.0:8032" [yarn-default.xml]
yarn.resourcemanager.principal = (unset)
yarn.resourcemanager.webapp.address = "0.0.0.0:8088" [yarn-default.xml]
yarn.resourcemanager.webapp.https.address = "0.0.0.0:8090" [yarn-default.xml]
mapreduce.input.fileinputformat.list-status.num-threads = "1"
[mapred-default.xml]
mapreduce.jobtracker.kerberos.principal = (unset)
mapreduce.job.hdfs-servers.token-renewal.exclude = (unset)
mapreduce.application.framework.path = (unset)
fs.iostatistics.logging.level = "info" [core-site.xml]
Security Options
================
dfs.data.transfer.protection = (unset)
hadoop.http.authentication.simple.anonymous.allowed = "true"
[core-default.xml]
hadoop.http.authentication.type = "simple" [core-default.xml]
hadoop.kerberos.min.seconds.before.relogin = "60" [core-default.xml]
hadoop.kerberos.keytab.login.autorenewal.enabled = "false" [core-default.xml]
hadoop.security.authentication = "simple" [core-default.xml]
hadoop.security.authorization = "false" [core-default.xml]
hadoop.security.credential.provider.path = (unset)
hadoop.security.credstore.java-keystore-provider.password-file = (unset)
hadoop.security.credential.clear-text-fallback = "true" [core-default.xml]
hadoop.security.key.provider.path = (unset)
hadoop.security.crypto.jceks.key.serialfilter = (unset)
hadoop.rpc.protection = "authentication" [core-default.xml]
hadoop.tokens = (unset)
hadoop.token.files = (unset)
Selected Configuration Options
==============================
fs.s3a.session.token = (unset)
fs.s3a.server-side-encryption-algorithm = "SSE-KMS"
[fs.s3a.bucket.stevel-london.server-side-encryption-algorithm via
[core-site.xml]]
fs.s3a.server-side-encryption.key =
"ar*********************************************************************3bf6"
[75] [fs.s3a.bucket.stevel-london.server-side-encryption.key via
[core-site.xml]]
fs.s3a.encryption-algorithm = (unset)
fs.s3a.encryption.key = (unset)
fs.s3a.aws.credentials.provider = "
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
" [core-default.xml]
fs.s3a.endpoint = (unset)
fs.s3a.endpoint.region = (unset)
fs.s3a.signing-algorithm = (unset)
fs.s3a.acl.default = (unset)
fs.s3a.attempts.maximum = "20" [core-default.xml]
fs.s3a.authoritative.path = (unset)
fs.s3a.block.size = "32M" [core-default.xml]
fs.s3a.bucket.probe = "0" [core-site.xml]
fs.s3a.buffer.dir = "/tmp/hadoop-stevel/s3a" [core-default.xml]
fs.s3a.bulk.delete.page.size = (unset)
fs.s3a.change.detection.source = "etag" [core-default.xml]
fs.s3a.change.detection.mode = "server" [core-default.xml]
fs.s3a.change.detection.version.required = "true" [core-default.xml]
fs.s3a.connection.ssl.enabled = "true" [core-default.xml]
fs.s3a.connection.maximum = "256" [core-site.xml]
fs.s3a.connection.establish.timeout = "5000" [core-site.xml]
fs.s3a.connection.request.timeout = "5s" [core-site.xml]
fs.s3a.connection.timeout = "5000" [core-site.xml]
fs.s3a.custom.signers = (unset)
fs.s3a.directory.marker.retention = "keep"
[fs.s3a.bucket.stevel-london.directory.marker.retention via [core-site.xml]]
fs.s3a.downgrade.syncable.exceptions = "false" [core-site.xml]
fs.s3a.etag.checksum.enabled = "false" [core-default.xml]
fs.s3a.experimental.input.fadvise = (unset)
fs.s3a.experimental.aws.s3.throttling = (unset)
fs.s3a.experimental.optimized.directory.operations = (unset)
fs.s3a.fast.buffer.size = (unset)
fs.s3a.fast.upload.buffer = "disk" [core-default.xml]
fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
fs.s3a.impl.disable.cache = (unset)
fs.s3a.list.version = "2" [core-default.xml]
fs.s3a.max.total.tasks = "128" [core-site.xml]
fs.s3a.multipart.size = "32M" [core-site.xml]
fs.s3a.multiobjectdelete.enable = "true" [core-default.xml]
fs.s3a.multipart.purge = "false" [core-site.xml]
fs.s3a.multipart.purge.age = "3600000" [core-site.xml]
fs.s3a.paging.maximum = "5000" [core-default.xml]
fs.s3a.path.style.access = "false" [core-site.xml]
fs.s3a.proxy.host = (unset)
fs.s3a.proxy.port = (unset)
fs.s3a.proxy.username = (unset)
fs.s3a.proxy.password = (unset)
fs.s3a.proxy.domain = (unset)
fs.s3a.proxy.workstation = (unset)
fs.s3a.rename.raises.exceptions = "true" [core-site.xml]
fs.s3a.readahead.range = "524288" [core-site.xml]
fs.s3a.retry.limit = "7" [core-default.xml]
fs.s3a.retry.interval = "500ms" [core-default.xml]
fs.s3a.retry.throttle.limit = "20" [core-default.xml]
fs.s3a.retry.throttle.interval = "100ms" [core-default.xml]
fs.s3a.ssl.channel.mode = "default_jsse" [core-default.xml]
fs.s3a.s3.client.factory.impl = (unset)
fs.s3a.threads.max = "128" [core-site.xml]
fs.s3a.threads.keepalivetime = "60" [core-default.xml]
fs.s3a.user.agent.prefix = (unset)
fs.s3a.assumed.role.arn =
"arn:aws:iam::152813717728:role/stevel-assumed-role" [core-site.xml]
fs.s3a.assumed.role.sts.endpoint = "sts.eu-west-2.amazonaws.com"
[core-site.xml]
fs.s3a.assumed.role.sts.endpoint.region = "eu-west-2" [core-site.xml]
fs.s3a.assumed.role.session.name = (unset)
fs.s3a.assumed.role.session.duration = "12h" [core-site.xml]
fs.s3a.assumed.role.credentials.provider =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" [core-default.xml]
fs.s3a.assumed.role.policy = (unset)
fs.s3a.metadatastore.impl =
"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore" [core-default.xml]
fs.s3a.metadatastore.authoritative = "false" [core-default.xml]
fs.s3a.metadatastore.authoritative.dir.ttl = (unset)
fs.s3a.metadatastore.fail.on.write.error = "true" [core-default.xml]
fs.s3a.metadatastore.metadata.ttl = "15m" [core-default.xml]
fs.s3a.s3guard.consistency.retry.interval = "2s" [core-default.xml]
fs.s3a.s3guard.consistency.retry.limit = "7" [core-default.xml]
fs.s3a.s3guard.ddb.table = (unset)
fs.s3a.s3guard.ddb.region = "eu-west-2"
[fs.s3a.bucket.stevel-london.s3guard.ddb.region via [core-site.xml]]
fs.s3a.s3guard.ddb.background.sleep = "25ms" [core-default.xml]
fs.s3a.s3guard.ddb.max.retries = "9" [core-default.xml]
fs.s3a.s3guard.ddb.table.capacity.read = "0" [core-default.xml]
fs.s3a.s3guard.ddb.table.capacity.write = "0" [core-default.xml]
fs.s3a.s3guard.ddb.table.create = "false" [core-default.xml]
fs.s3a.s3guard.ddb.throttle.retry.interval = "100ms" [core-default.xml]
fs.s3a.s3guard.local.max_records = (unset)
fs.s3a.s3guard.local.ttl = (unset)
fs.s3a.committer.name = "magic" [core-site.xml]
fs.s3a.committer.magic.enabled = "true" [core-default.xml]
fs.s3a.committer.staging.abort.pending.uploads = (unset)
fs.s3a.committer.staging.conflict-mode = "append" [core-default.xml]
fs.s3a.committer.staging.tmp.path = "tmp/staging" [core-default.xml]
fs.s3a.committer.threads = "8" [core-default.xml]
fs.s3a.committer.staging.unique-filenames = "false" [core-site.xml]
mapreduce.outputcommitter.factory.scheme.s3a =
"org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory" [mapred-default.xml]
mapreduce.fileoutputcommitter.marksuccessfuljobs = (unset)
fs.s3a.delegation.token.binding = (unset)
fs.s3a.delegation.token.secondary.bindings = (unset)
fs.s3a.audit.referrer.enabled = (unset)
fs.s3a.audit.referrer.filter = (unset)
fs.s3a.audit.reject.out.of.span.operations = "true" [core-site.xml]
fs.s3a.audit.request.handlers = (unset)
fs.s3a.audit.service.classname = (unset)
Required Classes
================
All these classes must be on the classpath
class: org.apache.hadoop.fs.s3a.S3AFileSystem
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: com.amazonaws.services.s3.AmazonS3
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
class: com.amazonaws.ClientConfiguration
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
class: java.lang.System
Optional Classes
================
These classes are needed in some versions of Hadoop.
And/or for optional features to work.
class: com.amazonaws.services.dynamodbv2.AmazonDynamoDB
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
class: com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
class: com.fasterxml.jackson.annotation.JacksonAnnotation
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-annotations-2.13.0.jar
class: com.fasterxml.jackson.core.JsonParseException
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-core-2.13.0.jar
class: com.fasterxml.jackson.databind.ObjectMapper
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-databind-2.13.0.jar
class: org.joda.time.Interval
Not found on classpath: org.joda.time.Interval
class: org.apache.hadoop.fs.s3a.s3guard.S3Guard
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.Invoker
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: com.amazonaws.services.s3.model.SelectObjectContentRequest
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
class: org.apache.hadoop.fs.s3a.select.SelectInputStream
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.impl.RenameOperation
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.impl.NetworkBinding
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.impl.DirectoryPolicy
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.hadoop.fs.s3a.audit.AuditManagerS3A
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
class: org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding
Not found on classpath:
org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding
class: org.wildfly.openssl.OpenSSLProvider
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/wildfly-openssl-1.0.7.Final.jar
At least one optional class was missing -the filesystem client *may* still
work
S3A Config validation
=====================
Buffer configuration option fs.s3a.buffer.dir = /tmp/hadoop-stevel/s3a
Temporary files created in /tmp/hadoop-stevel/s3a
Configuration options with prefix fs.s3a.ext.
=============================================
Locating implementation class for Filesystem scheme s3a://
==========================================================
FileSystem for s3a:// is: org.apache.hadoop.fs.s3a.S3AFileSystem
Loaded from:
file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
Endpoints
=========
Attempting to list and connect to public service endpoints,
without any authentication credentials.
This is just testing the reachability of the URLs.
If the request fails with any network error it is likely
to be configuration problem with address, proxy, etc%n
If it is some authentication error, then don't worry so much%n-look for the
results of the filesystem operations
Endpoint: https://stevel-london.s3.amazonaws.com/
=================================================
Canonical hostname s3-w.eu-west-2.amazonaws.com
IP address 52.95.148.69
Proxy: none
Connecting to https://stevel-london.s3.amazonaws.com/
Response: 403 : Forbidden
HTTP response 403 from https://stevel-london.s3.amazonaws.com/: Forbidden
Using proxy: false
Transfer-Encoding: chunked
null: HTTP/1.1 403 Forbidden
Server: AmazonS3
x-amz-request-id: 8YZH01Y5KCVP5TM9
x-amz-id-2:
0XTVOvJ5Z7p1FiMmGomfymm3sqqNp7Qbf0n8dcshA8npKgA72d5M4mY90P1Quvv/+jGojQw7KcU=
Date: Thu, 06 Jan 2022 13:41:52 GMT
x-amz-bucket-region: eu-west-2
Content-Type: application/xml
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access
Denied</Message><RequestId>8YZH01Y5KCVP5TM9</RequestId><HostId>0XTVOvJ5Z7p1FiMmGomfymm3sqqNp7Qbf0n8dcshA8npKgA72d5M4mY90P1Quvv/+jGojQw7KcU=</HostId></Error>
Endpoint: https://sts.eu-west-2.amazonaws.com/
==============================================
Canonical hostname 52.94.48.43
IP address 52.94.48.43
Proxy: none
Connecting to https://sts.eu-west-2.amazonaws.com/
Response: 200 : OK
HTTP response 200 from https://sts.eu-west-2.amazonaws.com/: OK
Using proxy: false
Transfer-Encoding: chunked
null: HTTP/1.1 200 OK
X-Cache: Miss from cloudfront
Server: Server
X-Content-Type-Options: nosniff
X-Amz-Cf-Pop: IAD79-C1
Permissions-Policy: interest-cohort=()
Connection: keep-alive
Last-Modified: Mon, 03 Jan 2022 20:29:37 GMT
x-amz-rid: NFFCXAKR69RKGD9S6BWE
Date: Thu, 06 Jan 2022 13:41:55 GMT
Via: 1.1 39174a6a452e175e6e614ff396a4ca4f.cloudfront.net (CloudFront)
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=300
X-Amz-Cf-Id: 0aTRgu2cx13Puh4L0m8tfbZ68KFe821nDGOxRGXpcu1VTyFiXE30DA==
Vary:
accept-encoding,Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
Set-Cookie: aws_lang=en; Domain=.amazon.com;
Path=/,aws-priv=eyJ2IjoxLCJldSI6MSwic3QiOjB9; Version=1; Comment="Anonymous
cookie for privacy regulations"; Domain=.aws.amazon.com; Max-Age=31536000;
Expires=Fri, 06-Jan-2023 13:41:55 GMT; Path=/
x-amz-id-1: NFFCXAKR69RKGD9S6BWE
X-XSS-Protection: 1; mode=block
Content-Security-Policy-Report-Only: default-src *; connect-src *; font-src
* data:; frame-src *; img-src * data:; media-src *; object-src *; script-src
'unsafe-eval' 'unsafe-inline' *; style-src 'unsafe-inline' *; report-uri
https://prod-us-west-2.csp-report.marketing.aws.dev/submit
Content-Type: text/html;charset=UTF-8
<!doctype html>
<html class="no-js aws-lng-en_US aws-with-target" lang="en-US"
data-static-assets="https://a0.awsstatic.com"; data-js-version="1.0.413"
data-css-version="1.0.400">
<head>
<meta http-equiv="Content-Security-Policy" content="default-src 'self'
data: https://a0.awsstatic.com; connect-src 'self'
https://112-tzm-766.mktoresp.com https://112-tzm-766.mktoutil.com
https://a0.awsstatic.com https://a0.p.awsstatic.com https://a1.awsstatic.com
https://amazonwebservices.d2.sc.omtrdc.net
https://amazonwebservicesinc.tt.omtrdc.net
https://api.regional-table.region-services.aws.a2z.com
https://api.us-west-2.prod.pricing.aws.a2z.com https://aws.amazon.com
https://aws.demdex.net https://b0.p.awsstatic.com https://c0.b0.p.awsstatic.com
https://calculator.aws https://cm.everesttech.net https://d0.awsstatic.com
https://d1.awsstatic.com https://d1fgizr415o1r6.cloudfront.net
https://d3borx6sfvnesb.cloudfront.net https://dc.ads.linkedin.com
https://dftu77xade0tc.cloudfront.net https://dpm.demdex.net https://fls-na
WARNING: this unauthenticated operation was not rejected.
This may mean the store is world-readable.
Check this by pasting https://sts.eu-west-2.amazonaws.com/ into your browser
Test filesystem s3a://stevel-london/
====================================
Trying some operations against the filesystem
Starting with some read operations, then trying to write
2022-01-06 13:41:55,180 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: Creating filesystem for
s3a://stevel-london/
2022-01-06 13:41:55,201 [main] INFO Configuration.deprecation
(Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption.key
is deprecated. Instead, use fs.s3a.encryption.key
2022-01-06 13:41:55,201 [main] INFO Configuration.deprecation
(Configuration.java:logDeprecation(1459)) -
fs.s3a.server-side-encryption-algorithm is deprecated. Instead, use
fs.s3a.encryption.algorithm
2022-01-06 13:41:55,983 [main] INFO impl.DirectoryPolicyImpl
(DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be
kept
2022-01-06 13:41:55,985 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - Creating filesystem for
s3a://stevel-london/: duration 0:00:807
S3AFileSystem{uri=s3a://stevel-london,
workingDir=s3a://stevel-london/user/stevel, inputPolicy=normal,
partSize=33554432, enableMultiObjectsDelete=true, maxKeys=5000,
readAhead=524288, blockSize=33554432, multiPartThreshold=134217728,
s3EncryptionAlgorithm='SSE_KMS',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2db2cd5,
auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A:
STARTED, auditor=LoggingAuditor{ID='b09ad642-5323-4ebd-b67a-3870d6efbeed',
headerEnabled=true, rejectOutOfSpan=true}}, metastore=NullMetadataStore,
authoritativeStore=false, authoritativePath=[], useListV1=false,
magicCommitter=true,
boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=384,
available=384, waiting=0}, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@70e659aa[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0],
credentials=AWSCredentialProviderList[refcount= 1:
[TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider,
EnvironmentVariableCredentialsProvider,
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@615f972],
delegation tokens=disabled, DirectoryMarkerRetention{policy='keep'},
instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
Implementation class class org.apache.hadoop.fs.s3a.S3AFileSystem
Path Capabilities
=================
fs.capability.etags.available true
fs.capability.etags.preserved.in.rename false
fs.capability.paths.checksums false
fs.capability.multipart.uploader true
fs.capability.outputstream.abortable true
fs.s3a.capability.magic.committer true
fs.s3a.capability.select.sql true
fs.s3a.capability.directory.marker.aware true
fs.s3a.capability.directory.marker.policy.keep true
fs.s3a.capability.directory.marker.policy.delete false
fs.s3a.capability.directory.marker.policy.authoritative false
fs.s3a.capability.directory.marker.action.keep true
fs.s3a.capability.directory.marker.action.delete false
2022-01-06 13:41:55,993 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: GetFileStatus
s3a://stevel-london/
root entry S3AFileStatus{path=s3a://stevel-london/; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null
2022-01-06 13:41:56,027 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - GetFileStatus s3a://stevel-london/:
duration 0:00:034
2022-01-06 13:41:56,027 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: First 25 entries of
listStatus(s3a://stevel-london/)
s3a://stevel-london/ : scanned 0 entries
2022-01-06 13:41:56,601 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - First 25 entries of
listStatus(s3a://stevel-london/): duration 0:00:574
2022-01-06 13:41:56,602 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: First 25 entries of
listFiles(s3a://stevel-london/)
Files listing provided by:
FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator
against s3a://stevel-london/; listing count 1; isTruncated=false;
counters=((object_continue_list_request=0)
(object_continue_list_request.failures=0) (object_list_request.failures=0)
(object_list_request=1));
gauges=();
minimums=((object_continue_list_request.min=-1)
(object_list_request.failures.min=-1)
(object_continue_list_request.failures.min=-1) (object_list_request.min=50));
maximums=((object_list_request.max=50) (object_continue_list_request.max=-1)
(object_continue_list_request.failures.max=-1)
(object_list_request.failures.max=-1));
means=((object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000))
(object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000))
(object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000))
(object_list_request.mean=(samples=1, sum=50, mean=50.0000)));
]}
2022-01-06 13:41:56,663 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - First 25 entries of
listFiles(s3a://stevel-london/): duration 0:00:061
Security and Delegation Tokens
==============================
Security is disabled
Filesystem s3a://stevel-london does not/is not configured to issue
delegation tokens (at least while security is disabled)
2022-01-06 13:41:56,663 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: probe for a directory which
does not yet exist s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
2022-01-06 13:41:56,738 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - probe for a directory which does not yet
exist s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration
0:00:075
Filesystem Write Operations
===========================
2022-01-06 13:41:56,738 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: creating a directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
2022-01-06 13:41:56,925 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - creating a directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:187
2022-01-06 13:41:56,926 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: create directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
2022-01-06 13:41:56,994 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - create directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:067
2022-01-06 13:41:56,994 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
2022-01-06 13:41:57,060 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration
0:00:066
2022-01-06 13:41:57,060 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: creating a file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
Output stream summary:
FSDataOutputStream{wrappedStream=S3ABlockOutputStream{WriteOperationHelper
{bucket=stevel-london}, blockSize=33554432
Statistics=counters=((object_multipart_aborted=0) (stream_write_total_time=0)
(op_hsync=0) (stream_write_exceptions_completing_upload=0)
(stream_write_total_data=7) (action_executor_acquired=1) (op_abort.failures=0)
(stream_write_exceptions=0) (op_hflush=0) (op_abort=0)
(action_executor_acquired.failures=0) (stream_write_queue_duration=0)
(multipart_upload_completed=0) (stream_write_block_uploads=1)
(object_multipart_aborted.failures=0) (multipart_upload_completed.failures=0)
(stream_write_bytes=7));
gauges=((stream_write_block_uploads_pending=1)
(stream_write_block_uploads_data_pending=0));
minimums=((object_multipart_aborted.failures.min=-1)
(multipart_upload_completed.failures.min=-1) (op_abort.min=-1)
(action_executor_acquired.min=0) (op_abort.failures.min=-1)
(action_executor_acquired.failures.min=-1) (object_multipart_aborted.min=-1)
(multipart_upload_completed.min=-1));
maximums=((object_multipart_aborted.max=-1)
(multipart_upload_completed.failures.max=-1) (op_abort.max=-1)
(action_executor_acquired.failures.max=-1) (multipart_upload_completed.max=-1)
(op_abort.failures.max=-1) (object_multipart_aborted.failures.max=-1)
(action_executor_acquired.max=0));
means=((op_abort.failures.mean=(samples=0, sum=0, mean=0.0000))
(action_executor_acquired.mean=(samples=1, sum=0, mean=0.0000))
(action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000))
(multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000))
(object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000))
(op_abort.mean=(samples=0, sum=0, mean=0.0000))
(object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000))
(multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)));
}}
2022-01-06 13:41:57,291 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - creating a file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration
0:00:230
2022-01-06 13:41:57,291 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: Listing
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
2022-01-06 13:41:57,331 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - Listing
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:040
2022-01-06 13:41:57,331 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: Reading a file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
input stream summary: org.apache.hadoop.fs.FSDataInputStream@26f3d90c:
S3AInputStream{s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
wrappedStream=closed read policy=normal pos=7 nextReadPos=7 contentLength=7
contentRangeStart=0 contentRangeFinish=7 remainingInCurrentRequest=0
ChangeTracker{ETagChangeDetectionPolicy mode=Server,
revisionId='c96643be2404fdc0054404530b198c21'}
StreamStatistics{counters=((stream_read_bytes_discarded_in_close=0)
(stream_read_operations_incomplete=0) (stream_read_close_operations=1)
(stream_read_fully_operations=0) (stream_read_opened=1)
(stream_read_seek_backward_operations=0) (stream_read_closed=1)
(stream_read_operations=1) (stream_read_bytes_backwards_on_seek=0)
(stream_read_seek_bytes_discarded=0) (action_http_get_request.failures=0)
(stream_read_seek_policy_changed=1) (stream_aborted=0)
(stream_read_unbuffered=0) (stream_read_seek_operations=0)
(stream_read_total_bytes=7) (stream_read_exceptions=0)
(stream_read_version_mismatches=0) (stream_read_seek_forward_operations=0)
(stream_read_bytes_discarded_in_abort=0) (stream_read_bytes=7)
(action_http_get_request=1) (stream_read_seek_bytes_skipped=0));
gauges=((stream_read_gauge_input_policy=0));
minimums=((action_http_get_request.min=45)
(action_http_get_request.failures.min=-1));
maximums=((action_http_get_request.max=45)
(action_http_get_request.failures.max=-1));
means=((action_http_get_request.mean=(samples=1, sum=45, mean=45.0000))
(action_http_get_request.failures.mean=(samples=0, sum=0, mean=0.0000)));
}}
2022-01-06 13:41:57,432 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - Reading a file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration
0:00:101
2022-01-06 13:41:57,432 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: Renaming file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file under
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir
2022-01-06 13:41:58,446 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - Renaming file
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file under
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir: duration
0:01:014
2022-01-06 13:41:58,447 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir/subfile
2022-01-06 13:41:58,595 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir/subfile:
duration 0:00:148
2022-01-06 13:41:58,595 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: delete dir
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2
2022-01-06 13:41:58,854 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - delete dir
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2: duration
0:00:259
2022-01-06 13:41:58,854 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2
2022-01-06 13:41:58,921 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - probing path
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2: duration
0:00:067
2022-01-06 13:41:58,921 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:<init>(56)) - Starting: delete directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
2022-01-06 13:41:59,019 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(115)) - delete directory
s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:098
2022-01-06 13:41:59,023 [main] INFO statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_executor_acquired=1)
(action_http_get_request=1)
(action_http_head_request=17)
(audit_request_execution=50)
(audit_span_creation=18)
(directories_created=2)
(directories_deleted=1)
(files_copied=2)
(files_copied_bytes=14)
(files_created=1)
(files_deleted=4)
(object_bulk_delete_request=2)
(object_copy_requests=2)
(object_delete_objects=5)
(object_delete_request=2)
(object_list_request=21)
(object_metadata_request=17)
(object_put_bytes=7)
(object_put_request=3)
(object_put_request_completed=3)
(op_create=1)
(op_delete=2)
(op_get_file_status=6)
(op_get_file_status.failures=4)
(op_list_files=2)
(op_list_status=1)
(op_mkdirs=2)
(op_open=1)
(op_rename=2)
(store_io_request=51)
(stream_read_bytes=7)
(stream_read_close_operations=1)
(stream_read_closed=1)
(stream_read_opened=1)
(stream_read_operations=1)
(stream_read_seek_policy_changed=1)
(stream_read_total_bytes=7)
(stream_write_block_uploads=1)
(stream_write_bytes=7)
(stream_write_total_data=14));
gauges=((stream_write_block_uploads_pending=1));
minimums=((action_executor_acquired.min=0)
(action_http_get_request.min=45)
(action_http_head_request.min=27)
(object_bulk_delete_request.min=44)
(object_delete_request.min=38)
(object_list_request.min=32)
(object_put_request.min=73)
(op_create.min=41)
(op_delete.min=38)
(op_get_file_status.failures.min=66)
(op_get_file_status.min=3)
(op_list_files.min=40)
(op_list_status.min=571)
(op_mkdirs.min=185)
(op_rename.min=399));
maximums=((action_executor_acquired.max=0)
(action_http_get_request.max=45)
(action_http_head_request.max=110)
(object_bulk_delete_request.max=61)
(object_delete_request.max=41)
(object_list_request.max=558)
(object_put_request.max=167)
(op_create.max=41)
(op_delete.max=87)
(op_get_file_status.failures.max=148)
(op_get_file_status.max=67)
(op_list_files.max=53)
(op_list_status.max=571)
(op_mkdirs.max=186)
(op_rename.max=428));
means=((action_executor_acquired.mean=(samples=1, sum=0, mean=0.0000))
(action_http_get_request.mean=(samples=1, sum=45, mean=45.0000))
(action_http_head_request.mean=(samples=17, sum=629, mean=37.0000))
(object_bulk_delete_request.mean=(samples=2, sum=105, mean=52.5000))
(object_delete_request.mean=(samples=2, sum=79, mean=39.5000))
(object_list_request.mean=(samples=21, sum=1344, mean=64.0000))
(object_put_request.mean=(samples=3, sum=360, mean=120.0000))
(op_create.mean=(samples=1, sum=41, mean=41.0000))
(op_delete.mean=(samples=2, sum=125, mean=62.5000))
(op_get_file_status.failures.mean=(samples=4, sum=355, mean=88.7500))
(op_get_file_status.mean=(samples=2, sum=70, mean=35.0000))
(op_list_files.mean=(samples=2, sum=93, mean=46.5000))
(op_list_status.mean=(samples=1, sum=571, mean=571.0000))
(op_mkdirs.mean=(samples=2, sum=371, mean=185.5000))
(op_rename.mean=(samples=2, sum=827, mean=413.5000)));
JVM: memory=89262960
```
I'm actually going to list that on the commands to run during validation
because it could potentially find a problem on the command line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
