[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833804#comment-16833804
]
Steve Loughran commented on HADOOP-16221:
-
bq. The retries on a MetadataStore are pretty robust where failures should be
pretty rare.
they are now we're handling throttling. I'd find it unlikely that you'd be in a
situation where you have W access to S3 and yet DDB writes fail unless
permissions or client config are broken
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Assignee: Ben Roling
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831010#comment-16831010
]
Ben Roling commented on HADOOP-16221:
-
Thanks [~fabbri] for sharing your perspective. I understand where you are
coming from.
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Assignee: Ben Roling
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830848#comment-16830848
]
Aaron Fabbri commented on HADOOP-16221:
---
Sorry I didn't see this until now. Thanks for the contribution and
documentation.
I'll give some background on the existing logic at least.
As you can see we generally chose to fall back to raw s3 behavior when there
are failures with the Metadata Store. S3Guard was targeted to existing S3
customers so that made sense to me.
The MetadataStore is conceptually a "trailling log of metadata changes made to
S3". You can also think of it as a consistency hint. There are are few
guarantees with the semantics that S3 exposes (e.g. no upper bound on eventual
consistency time–think about what that means for your write. You need a write
journal w/ fast scalable queries and transactions to really solve this but
you'd be better off ditching S3 for a real storage system IMO..).
We are logging things that already happened in S3. With error semantics, if you
mutate s3 but fail to mutate MetadataStore I thought you should either (1) roll
back transaction and return failure or (2) don't rollback and return success.
#1 is seen as too complex and slow to do right above S3 but #2 returns success
after successful mutation of S3 state.
So this was an intentional decision, not to return failure when you
successfully write a file to S3. As you note, there is no roll back.
I can see the argument for doing it the new way as well.. My bias is that it is
important to know whether or not you actually wrote data to the backing store.
Spent some time in finance (the wrong write can cost you) and storage companies
which sort of formed my bias.
Essentially both options are "wrong". Before, we'd return success but give up
consistency hints on that path, now we return failure even though we wrote the
data to S3.
In lieu of a real storage system, I think having this well-documented and
configurable is fine. The retries on a MetadataStore are pretty robust where
failures should be pretty rare.
Hope this background was interesting. Feel free to email me if you ever need a
review. My email filters tend to catch a lot of stuff that I should have
noticed.
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Assignee: Ben Roling
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830322#comment-16830322
]
Ben Roling commented on HADOOP-16221:
-
Awesome, thanks!
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Assignee: Ben Roling
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830188#comment-16830188
]
Hudson commented on HADOOP-16221:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16480 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16480/])
HADOOP-16221. S3Guard: add option to fail operation on metadata write (stevel:
rev 0af4011580878566213016af0c32633eabd15100)
* (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (add)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/MetadataPersistenceException.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMetadataPersistenceException.java
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Retries.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Assignee: Ben Roling
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store
[
https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805149#comment-16805149
]
Ben Roling commented on HADOOP-16221:
-
I'm preparing a patch for this. I appreciate any feedback, especially with
regard to default behavior.
> S3Guard: fail write that doesn't update metadata store
> --
>
> Key: HADOOP-16221
> URL: https://issues.apache.org/jira/browse/HADOOP-16221
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Ben Roling
>Priority: Major
>
> Right now, a failure to write to the S3Guard metadata store (e.g. DynamoDB)
> is [merely
> logged|https://github.com/apache/hadoop/blob/rel/release-3.1.2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2708-L2712].
> It does not fail the S3AFileSystem write operation itself. As such, the
> writer has no idea that anything went wrong. The implication of this is that
> S3Guard doesn't always provide the consistency it advertises.
> For example [this
> article|https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/]
> states:
> {quote}If a Hadoop S3A client creates or moves a file, and then a client
> lists its directory, that file is now guaranteed to be included in the
> listing.
> {quote}
> Unfortunately, this is sort of untrue and could result in exactly the sort of
> problem S3Guard is supposed to avoid:
> {quote}Missing data that is silently dropped. Multi-step Hadoop jobs that
> depend on output of previous jobs may silently omit some data. This omission
> happens when a job chooses which files to consume based on a directory
> listing, which may not include recently-written items.
> {quote}
> Imagine the typical multi-job Hadoop processing pipeline. Job 1 runs and
> succeeds, but one (or more) S3Guard metadata write failed under the covers.
> Job 2 picks up the output directory from Job 1 and runs its processing,
> potentially seeing an inconsistent listing, silently missing some of the Job
> 1 output files.
> S3Guard should at least provide a configuration option to fail if the
> metadata write fails. It seems even ideally this should be the default?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
