[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975555#comment-15975555
]
Giovanni Matteo Fumarola commented on YARN-3663:
------------------------------------------------
Thanks [~curino] for the accurate review. Let me go in order.
Conf
1. Every process should have at most 1 connection open to the StateStore except
for Router. It has 10. This is for performance and scalability reason of the
Database. FederationInterceptor open one connection for each application
running in the node. More than that can cause a possible DoS to StateStore.
2. Done.
SQLFederationStateStore
1. Done.
2. Done.
3. Done.
4. Done.
5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it
in 2 lines.
6. Done.
7. We can improve/reduce the amount of code later under a different Jira. I am
quite confident in this part and I would avoid to change it.
8. Done.
9. We can improve/reduce the amount of code later under a different Jira. I am
quite confident in this part and I would avoid to change it.
10. That is a safe check, since admin's cluster can setup their own database.
11. Done.
12. Done.
13. Done.
14. Done.
15. Done.
FederationStateStoreUtils
1. Done
HSQLDBFederationStateStore
1. Done
2. I tried to move in a file but I am adding a lot of complexity and I did not
find a way to let it work. I tried to format in SQL standard inside the file
itself but the autoformat messes it up.
3. Done.
4. Fixed it - good catch.
5. In that way I only insert a new application. That code is like if (! app
exists){add app}.
6. Done.
Stored procedure
1. Done.
2. Done.
3. Increased to 6000.
4. My bad, it was a for a performance test - introduced in the patch by mistake.
5. We can DEREGISTER it or put it in a LOST state.
6. We did not run in any issues with those settings. I would keep like this.
7. Let's open a JIRA about it.
8. Done.
9. Increase to 6000.
10. As I said before, let's keep this one since we have one stored procedure
for each function. We can reduce/improve later.
Top level concerns:
1. The current design was not edited over the last year. However in case we
need to evolve the schema we will need to add them.
2. This cannot happen because we have input validators.
3. For that scenario, we have to add only the stored procedure for the specific
database.
4. I edited some tests to cover some scenario we did not check. However, the
current state covers all the possible scenario.
> Federation State and Policy Store (DBMS implementation)
> -------------------------------------------------------
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Affects Versions: YARN-2915
> Reporter: Giovanni Matteo Fumarola
> Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch,
> YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and
> Policy Store, which implements YARN-3662 APIs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
