[
https://issues.apache.org/jira/browse/HADOOP-16875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041944#comment-17041944
]
Rafael Acevedo commented on HADOOP-16875:
-----------------------------------------
Hi Steve,
I see. About pay-as-you-go tables, they actually cost a bit higher when the
load is stable(we're running a spark streaming app so IO is quite stable and
the table never gets idle).
On Hadoop 3.2, DDB WCUs and RCUs are ~200, for a single sparj streaming app,
which results in approximately $115 per month(not taking storage into account).
What are the improvements on trunk that could reduce costs?
About implementing support for other backends, do you have any considerations
for postgres/mysql? Maybe I can try working on this.
Thanks!
> S3Guard: add support for other MetadataStores
> ---------------------------------------------
>
> Key: HADOOP-16875
> URL: https://issues.apache.org/jira/browse/HADOOP-16875
> Project: Hadoop Common
> Issue Type: Wish
> Affects Versions: 3.2.1
> Reporter: Rafael Acevedo
> Priority: Major
>
> Hi all,
>
> Are there any plans to add other MetadataStore implementations for S3Guard?
> DynamoDB costs are too high when the read capacity/write capacity are high.
>
> Maybe a Postgres/MySQL implementation is simple enough to implement and offer
> strong consistency.
> Another idea is to implement a Cassandra/Scylla MetadataStore(for better
> write scalability), but we should pay attention to consistency.
>
> Any thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
