[ 
https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061659#comment-16061659
 ] 

Sean Mackrory commented on HADOOP-14576:
----------------------------------------

{quote}do you think this is something to need for an initial preview?{quote}

No. The fix would be DynamoDB-specific, so it's unlikely to be disruptive to 
other S3 development.

{quote}how long does it go into this state —can we spin until it is 
fixed?{quote}

The workload that hit this has run numerous times before and never had an 
issue. So it's possible that's because the window in which it happens is small 
enough to have been missed until now. I've been trying to pump data into a 
table and haven't been able to reproduce it. The workload in question was 
Hive's parallel rename (which gets aborted on the first failure), though, and 
not something like a MapReduce job, so there's no indication of how parallel or 
subsequent tasks would have been affected. It's also possible that other 
unplanned glitches server-side might look the same but take much longer. I'd 
suggest we consider reusing retryBackoff() from the getVersionMarker logic in 
the event of failures?

My concern with failing over to S3 for non-auth read is what happens when 
you're listing stuff that isn't consistent on S3 yet. IMO non-auth mode is 
really just to enable lazily loading data that already existed or that is added 
outside of S3Guard. I don't think it should weaken guarantees in the presence 
of partitioning events in DynamoDB.

> DynamoDB tables may leave ACTIVE state after initial connection
> ---------------------------------------------------------------
>
>                 Key: HADOOP-14576
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14576
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Sean Mackrory
>
> We currently only anticipate tables not being in the ACTIVE state when first 
> connecting. It is possible for a table to be in the ACTIVE state and move to 
> an UPDATING state during partitioning events. Attempts to read or write 
> during that time will result in an AmazonServerException getting thrown. We 
> should try to handle that better...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to