[jira] [Commented] (HADOOP-14291) S3a "Bad Request" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362439#comment-16362439 ] Steve Loughran commented on HADOOP-14291: - I've got a separate app to do the diagnostics: https://github.com/steveloughran/cloudstore this is something which could be pulled into hadoop itself > S3a "Bad Request" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Major > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14291) S3a "Bad Request" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986342#comment-15986342 ] Steve Loughran commented on HADOOP-14291: - + proxies, IP Address of proxies and S3 endpoint + if bucket requires encryption and client doesn't have it turned on, that also fails. Something which can be probed Maybe the strategy would be to bootstrap comms to the endpoint: lookup, remote server time (in HTTP responses?), then * List buckets & see if the user is down as an owner for the target bucket (warn but don't fail if not, as it may be someone else's public one, one in same org,...) * attempt to read bucket root / * pick a file & do a HEAD * try to read the file * if enabled, attempt to write then delete an entry (some UUID on root; avoids directory creation issues) > S3a "Bad Request" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14291) S3a "Bad Request" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974462#comment-15974462 ] Steve Loughran commented on HADOOP-14291: - No, MD5 hashes aren't secure, even if you concat them together. Building a hash table via Hadoop MR or spark is straightforward (especially the latter); save it to HDFS under the right file structure and you have O(1) md5 -> secret conversion. Maybe just include the last 4 digits, the way web sites that leak credit card details do. > S3a "Bad Request" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14291) S3a "Bad Request" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972705#comment-15972705 ] Steve Loughran commented on HADOOP-14291: - Wiki page created: https://wiki.apache.org/hadoop/S3ABadRequest now someone needs to add this to the translated error, and ideally, the diagnostics > S3a "Bad Request" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
