[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[
https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122143#comment-16122143
]
Jian He commented on YARN-6413:
---
So the registry path is currently like this:
{code}
/users/{username}/{serviceclass}/{instancename}/components/{componentname}
{code}, see
[document|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/registry/yarn-registry.html]
For native-service, ApplicationServiceRecordKey#appId will need to be appName
and ContainerServiceRecordKey#containerId will need to be containerName,
probably rename it to be containerKey or appKey ?
And the new method RegistryUtil#getPathForServiceRecordKey, the generated path
is inconsistent with the document - which will beak the callers of registry
API. Could you make it in consistent with the document ?
> Decouple Yarn Registry API from ZK
> --
>
> Key: YARN-6413
> URL: https://issues.apache.org/jira/browse/YARN-6413
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: amrmproxy, api, resourcemanager
>Reporter: Ellen Hui
>Assignee: Ellen Hui
> Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch,
> 0003-Registry-API-api-only.patch, 0004-Registry-API-api-stubbed.patch
>
>
> Right now the Yarn Registry API (defined in the RegistryOperations interface)
> is a very thin layer over Zookeeper. This jira proposes changing the
> interface to abstract away the implementation details so that we can write a
> FS-based implementation of the registry service, which will be used to
> support AMRMProxy HA.
> The new interface will use register/delete/resolve APIs instead of
> Zookeeper-specific operations like mknode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120653#comment-16120653 ] Jian He commented on YARN-6413: --- Looks like the ApplicationServiceRecordKey etc. is still using the appId, as said before, that will be in conflict with what exists today ? > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch, > 0003-Registry-API-api-only.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100749#comment-16100749 ] Jian He commented on YARN-6413: --- bq. Right now yarn-native-services doesn't compile for me with a pom.xml error (pulled just now), is the branch healthy? I just fixed this, can you pull the latest branch? bq. I can add a select-multiple method with a filter of some sort. what do you mean by this ? as long as it doesn't change the functionality, I think it's fine. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099352#comment-16099352 ] Ellen Hui commented on YARN-6413: - bq: looks like RegistryUtils#extractServiceRecords is removed. Do you mind change RegistryDNSServer to use the new methods ? (i.e. make this patch compile with yarn-native-services branch) Ah, I see. RegistryUtils#extractServiceRecords is one of the methods that depends on having the hierarchical namespace, which is why it was removed. I can add a select-multiple method with a filter of some sort, would that work? The point of this from our end was to abstract away the path. Right now yarn-native-services doesn't compile for me with a pom.xml error (pulled just now), is the branch healthy? bq: The DNS today depends on the ZK path layout to reconstruct the service record. So changing the zk path will break DNS. Ok, I will put the ZK path layout back to the way it was before. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099345#comment-16099345 ] Jian He commented on YARN-6413: --- bq. I did not remove any methods, I just didn't add the ones that exist in yarn-native-services but not trunk. I thought doing so would probably cause more conflicts than not. looks like RegistryUtils#extractServiceRecords is removed. Do you mind change RegistryDNSServer to use the new methods ? (i.e. make this patch compile with yarn-native-services branch) bq. My understanding was that DNS went through ZK directly without going through the interface, so it wouldn't be affected by the service records setting up the path differently. I can change the path construction back for the ZK impl if it needs that. The DNS today depends on the ZK path layout to reconstruct the service record. So changing the zk path will break DNS. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098902#comment-16098902 ] Ellen Hui commented on YARN-6413: - bq. In RegistryDNSServer, all the methods removed are essential to registryDNS - the implementation of RegistryDNS is depending on the zookeeper’s listener functionality. This is one big difference from the state store implementation. we can not remove those methods. I did not remove any methods, I just didn't add the ones that exist in yarn-native-services but not trunk. I thought doing so would probably cause more conflicts than not. bq. The path is used by RegistryDNSServer for reconstructing the DNS record(e.g. BaseServiceRecordProcessor#getContainerName). If we change the path, everything there will break. Also, the registry documentation needs to change. My understanding was that DNS went through ZK directly without going through the interface, so it wouldn't be affected by the service records setting up the path differently. I can change the path construction back for the ZK impl if it needs that. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093802#comment-16093802 ] Jian He commented on YARN-6413: --- In RegistryDNSServer, all the methods removed are essential to registryDNS - the implementation of RegistryDNS is depending on the zookeeper’s listener functionality. This is one big difference from the state store implementation. we can not remove those methods. bq. The path was rearranged in order to flatten the namespace The path is used by RegistryDNSServer for reconstructing the DNS record(e.g. BaseServiceRecordProcessor#getContainerName). If we change the path, everything there will break. Also, the registry documentation needs to change. can you make sure the patch does not break existing stuff ? it’s fine to abstract out the interface, but breaking existing functionalities will be another big problem. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[
https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058691#comment-16058691
]
Jian He commented on YARN-6413:
---
Hi Ellem, sorry, I was looking at the wrong patch. some comments on the latest
patch:
- The patch removed some methods which are actually used by RegistryDNS, can
you check the RegistryDNSServer class and see if we need to add them back or
make it use the alternative methods ?
- RegistryOpertions: InvalidRecordException and NoRecordException can be
removed from the method signature
- getPathForServiceRecordKey: the patch changed the path naming , any reason
for not keeping the original naming?
- remove the getSubpathsForUserServiceRecordKey method, it's unused
- should use LOG.error instead
{code}
} catch (InvalidRegistryKeyException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
{code}
> Decouple Yarn Registry API from ZK
> --
>
> Key: YARN-6413
> URL: https://issues.apache.org/jira/browse/YARN-6413
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: amrmproxy, api, resourcemanager
>Reporter: Ellen Hui
>Assignee: Ellen Hui
> Attachments: 0001-Registry-API-v2.patch
>
>
> Right now the Yarn Registry API (defined in the RegistryOperations interface)
> is a very thin layer over Zookeeper. This jira proposes changing the
> interface to abstract away the implementation details so that we can write a
> FS-based implementation of the registry service, which will be used to
> support AMRMProxy HA.
> The new interface will use register/delete/resolve APIs instead of
> Zookeeper-specific operations like mknode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050803#comment-16050803 ] Ellen Hui commented on YARN-6413: - I think you were looking at the old patch, I have removed it now. The current patch should not have formatting changes. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049782#comment-16049782 ] Jian He commented on YARN-6413: --- e.g. Several places in RegistryCli has been reformatted. Also, looks like the entire file of RegistryOperationsFactory and RegistryUtils are reformatted. Typically, we don't introduce large trunks of reformatting-only changes because that makes the review harder, as people would have to read unnecessary code changes. And it also makes the history hard to track, when user are trying to find logic changes in history. sometimes it's also harder to merge back. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, > 0001-WIP-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049646#comment-16049646 ] Ellen Hui commented on YARN-6413: - [~jianhe] I think I had cleaned up all of the formatting changes in the latest patch, could you point me to the files that have lots of format changes? There are a couple files that were renamed, so they will show up as large diffs, but the actual changes were small there. > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, > 0001-WIP-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048702#comment-16048702 ] Jian He commented on YARN-6413: --- [~ellenfkh], there are a lot of format only changes, can you clean those up ? > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, > 0001-WIP-Registry-API-v2.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
