steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-530748126
# +1
As well as all the automated tests, did some manual command line operations.
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-530564476
ok, full test with ddb non-auth happy; repeating with auth
Tomorrow I'll build the
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-530380233
my branch with changes to the assertion
https://github.com/steveloughran/hadoop/tree/inco
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-530357583
last test run:
```
[ERROR] Failures:
[ERROR]
ITestS3GuardFsck.testIDetectPare
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-52599
test runs before your last commit. First fine; second with -Dauth failed
```
[ER
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-529980416
OK, done a quick scan. Changes all LGTM
-
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528905689
Overall then, last iteration has a working CLI
* One of the tests is brittle in paralle
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528894617
OK, latest review is good with etags; modtime is something we can worry
about as an extra
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528883425
The testCLIFsckWithParam test works standalone. Looks to me like a race
condition -the fs
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528873116
and
```
[ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
25
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528871326
got a test run failure in a new test
```
testCLIFsckWithParam(org.apache.hadoop.fs.
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528867021
Did another test run on an unversioned bucket where the the DDB table was
built up with a
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528866219
e.g
```
bin/hadoop s3guard fsck -check
s3a://hwdev-steve-ireland-new/etc/hadoop/she
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528864203
OK, latest patch is better.
* still warns of no etag on a dir;
* when you pass a pa
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-528858428
working on the CLI, but overreporting errors, especially on versioning
Warns of no
steveloughran commented on issue #1208: HADOOP-16423. S3Guard fsck: Check
metadata consistency between S3 and metadatastore (log)
URL: https://github.com/apache/hadoop/pull/1208#issuecomment-523927611
## Overall
there are checks, but as it doesnt recurse from me it's hard to valide t
16 matches
Mail list logo
