Re: [Freeipa-devel] Designing better API compatibility
On 04/09/2015 09:16 AM, Jan Cholasta wrote: Dne 8.4.2015 v 16:44 Martin Kosek napsal(a): On 03/20/2015 05:00 PM, Petr Vobornik wrote: On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 Ok. So to sum up this thread (and do the actual changes in Trac), in FreeIPA 4.2, we would: 1) Prepare the API UI browser or generated API documentation so that people could finally see the existing API without having to read the code or inspect jquery sent by the Web UI. https://fedorahosted.org/freeipa/ticket/3129 This is not related to API compatibility, it just uses the same metadata. It is not related to API compatibility per se, but very related to better API consumption and a low hanging fruit we could start with, since we have the metadata already 2) Have option for the ipa tool to send version-less command to the server which should thus behave as if it is the same version. Bonus points if defaults are not filled in this case to prevent unrecoverable Unkown Option errors. https://fedorahosted.org/freeipa/ticket/4768 Not sending version and not computing defaults are very different things and their implemetantion will be very different too. I would not mix them together. We are now getting more in the design, but the idea was that sending the defaults may force server to refuse serving the command even if the caller did not explicitly requested that option. Even if the caller did not care about the new default option in 4.x, he would not be able to call the command as it would be always sent to the old server. Rest would be left for later releases. Please holler if there is disagreement with this plan. I agree with Nathaniel that we should do thin client ASAP. I agree too, but given it is not realistic for 4.2, we need to do at least something in 4.2 for projects which need to use the CLI against older versions. Skipping version and client defaults seemed as the low hanging fruit that could help them. If there is a better idea about what else can be done in 4.2, I am open to it. Martin -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 04/09/2015 09:35 AM, Martin Kosek wrote: On 04/09/2015 09:16 AM, Jan Cholasta wrote: Dne 8.4.2015 v 16:44 Martin Kosek napsal(a): On 03/20/2015 05:00 PM, Petr Vobornik wrote: On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 Ok. So to sum up this thread (and do the actual changes in Trac), in FreeIPA 4.2, we would: 1) Prepare the API UI browser or generated API documentation so that people could finally see the existing API without having to read the code or inspect jquery sent by the Web UI. https://fedorahosted.org/freeipa/ticket/3129 This is not related to API compatibility, it just uses the same metadata. It is not related to API compatibility per se, but very related to better API consumption and a low hanging fruit we could start with, since we have the metadata already +1 2) Have option for the ipa tool to send version-less command to the server which should thus behave as if it is the same version. Bonus points if defaults are not filled in this case to prevent unrecoverable Unkown Option errors. https://fedorahosted.org/freeipa/ticket/4768 Not sending version and not computing defaults are very different things and their implemetantion will be very different too. I would not mix them together. We are now getting more in the design, but the idea was that sending the defaults may force server to refuse serving the command even if the caller did not explicitly requested that option. Even if the caller did not care about the new default option in 4.x, he would not be able to call the command as it would be always sent to the old server. +1 that not sending defaults is essential for this case. IMHO we should not send them at all. Rest would be left for later releases. Please holler if there is disagreement with this plan. I agree with Nathaniel that we should do thin client ASAP. I agree too, but given it is not realistic for 4.2, we need to do at least something in 4.2 for projects which need to use the CLI against older versions. Skipping version and client defaults seemed as the low hanging fruit that could help them. If there is a better idea about what else can be done in 4.2, I am open to it. Martin -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
Dne 8.4.2015 v 16:44 Martin Kosek napsal(a): On 03/20/2015 05:00 PM, Petr Vobornik wrote: On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 Ok. So to sum up this thread (and do the actual changes in Trac), in FreeIPA 4.2, we would: 1) Prepare the API UI browser or generated API documentation so that people could finally see the existing API without having to read the code or inspect jquery sent by the Web UI. https://fedorahosted.org/freeipa/ticket/3129 This is not related to API compatibility, it just uses the same metadata. 2) Have option for the ipa tool to send version-less command to the server which should thus behave as if it is the same version. Bonus points if defaults are not filled in this case to prevent unrecoverable Unkown Option errors. https://fedorahosted.org/freeipa/ticket/4768 Not sending version and not computing defaults are very different things and their implemetantion will be very different too. I would not mix them together. Rest would be left for later releases. Please holler if there is disagreement with this plan. I agree with Nathaniel that we should do thin client ASAP. Thanks, Martin -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
Dne 9.4.2015 v 09:45 Petr Vobornik napsal(a): On 04/09/2015 09:35 AM, Martin Kosek wrote: On 04/09/2015 09:16 AM, Jan Cholasta wrote: Dne 8.4.2015 v 16:44 Martin Kosek napsal(a): On 03/20/2015 05:00 PM, Petr Vobornik wrote: On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 Ok. So to sum up this thread (and do the actual changes in Trac), in FreeIPA 4.2, we would: 1) Prepare the API UI browser or generated API documentation so that people could finally see the existing API without having to read the code or inspect jquery sent by the Web UI. https://fedorahosted.org/freeipa/ticket/3129 This is not related to API compatibility, it just uses the same metadata. It is not related to API compatibility per se, but very related to better API consumption and a low hanging fruit we could start with, since we have the metadata already +1 2) Have option for the ipa tool to send version-less command to the server which should thus behave as if it is the same version. Bonus points if defaults are not filled in this case to prevent unrecoverable Unkown Option errors. https://fedorahosted.org/freeipa/ticket/4768 Not sending version and not computing defaults are very different things and their implemetantion will be very different too. I would not mix them together. We are now getting more in the design, but the idea was that sending the defaults may force server to refuse serving the command even if the caller did not explicitly requested that option. Even if the caller did not care about the new default option in 4.x, he would not be able to call the command as it would be always sent to the old server. +1 that not sending defaults is essential for this case. IMHO we should not send them at all. I agree with that, I'm just saying it won't be as simple as it sounds and certainly not as simple as not sending the version. Rest would be left for later releases. Please holler if there is disagreement with this plan. I agree with Nathaniel that we should do thin client ASAP. I agree too, but given it is not realistic for 4.2, we need to do at least something in 4.2 for projects which need to use the CLI against older versions. Skipping version and client defaults seemed as the low hanging fruit that could help them. If there is a better idea about what else can be done in 4.2, I am open to it. Martin -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 03/20/2015 05:00 PM, Petr Vobornik wrote: On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 Ok. So to sum up this thread (and do the actual changes in Trac), in FreeIPA 4.2, we would: 1) Prepare the API UI browser or generated API documentation so that people could finally see the existing API without having to read the code or inspect jquery sent by the Web UI. https://fedorahosted.org/freeipa/ticket/3129 2) Have option for the ipa tool to send version-less command to the server which should thus behave as if it is the same version. Bonus points if defaults are not filled in this case to prevent unrecoverable Unkown Option errors. https://fedorahosted.org/freeipa/ticket/4768 Rest would be left for later releases. Please holler if there is disagreement with this plan. Thanks, Martin -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 03/20/2015 02:19 PM, Simo Sorce wrote: On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote: Hi guys, I would like to resurrect the discussion we had during DevConf.cz time, about API compatibility in the FreeIPA server. So right now, we maintain the backward compatibility, old clients can talk to newer servers. Forward compatibility is not maintained. Unfortunately, this is not very helpful in real deployments, where the server will often be some RHEL/CentOS system and the client may be the newest Fedora - with newer API than the server. This is the toughest part we need to solve. There 3 main areas we wanted to attack with respect to compatibility: 1) API publishing and/or API browser This is mostly documentation/interactive browser to see the supported API of the server. It should not be difficult, it would just consume the metadata already generated by the server. Ticket: https://fedorahosted.org/freeipa/ticket/3129 2) Forward compatibility of the direct API consumers Until now, to keep newer clients working against older server, we are using the following trick in the ipa-client-install: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install#n1649 It mostly works, one just needs to know the minimal version that needs to be supported. It would be more user friendly, however, if this check is done on the server automatically, without user having to research it. This applies both for ipalib python lib consumer and for direct JSON-RPC consumers. Ticket: https://fedorahosted.org/freeipa/ticket/4739 3) Forward compatibility of the ipa client tool There are different approaches how to fix this, the generally accepted idea was to implement very thin client, which would download and cache metadata from the server on the client and generate the CLI from it. We would only need to have separate client only plugins, basically implementing interactive_promt_callback from existing server side plugins. Tickets: https://fedorahosted.org/freeipa/ticket/4739, https://fedorahosted.org/freeipa/ticket/4768 Now, question is what we can do in 4.2. I do not think we can manage to rewrite ipa command in the thin client, but we should do at least some portion of 1) and 3). I could not decipher that from our Devconf.cz notes. To me, the simplest way of fixing forward compatibility seems to be following steps (besides not making API backwards incompatible - i.e. what we do already): - keep sending API version from client to server - server should not refuse newer API versions - only raise error when an unknown option or unknown command is used This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 20.3.2015 14:19, Simo Sorce wrote: This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Currently new clients do not work with older servers, right? Maybe we should do one more (last!) release like that and do a big cut after that. It would make the design so much easier if the new (supposedly thin) client does not need to support ancient servers which had only 'fat' clients. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote: Hi guys, I would like to resurrect the discussion we had during DevConf.cz time, about API compatibility in the FreeIPA server. So right now, we maintain the backward compatibility, old clients can talk to newer servers. Forward compatibility is not maintained. Unfortunately, this is not very helpful in real deployments, where the server will often be some RHEL/CentOS system and the client may be the newest Fedora - with newer API than the server. This is the toughest part we need to solve. There 3 main areas we wanted to attack with respect to compatibility: 1) API publishing and/or API browser This is mostly documentation/interactive browser to see the supported API of the server. It should not be difficult, it would just consume the metadata already generated by the server. Ticket: https://fedorahosted.org/freeipa/ticket/3129 2) Forward compatibility of the direct API consumers Until now, to keep newer clients working against older server, we are using the following trick in the ipa-client-install: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install#n1649 It mostly works, one just needs to know the minimal version that needs to be supported. It would be more user friendly, however, if this check is done on the server automatically, without user having to research it. This applies both for ipalib python lib consumer and for direct JSON-RPC consumers. Ticket: https://fedorahosted.org/freeipa/ticket/4739 3) Forward compatibility of the ipa client tool There are different approaches how to fix this, the generally accepted idea was to implement very thin client, which would download and cache metadata from the server on the client and generate the CLI from it. We would only need to have separate client only plugins, basically implementing interactive_promt_callback from existing server side plugins. Tickets: https://fedorahosted.org/freeipa/ticket/4739, https://fedorahosted.org/freeipa/ticket/4768 Now, question is what we can do in 4.2. I do not think we can manage to rewrite ipa command in the thin client, but we should do at least some portion of 1) and 3). I could not decipher that from our Devconf.cz notes. To me, the simplest way of fixing forward compatibility seems to be following steps (besides not making API backwards incompatible - i.e. what we do already): - keep sending API version from client to server - server should not refuse newer API versions - only raise error when an unknown option or unknown command is used This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. When plugins change the behavior, they should check for client version and base it's action on it (sort of the capabilities we already have). This is the simple way, it would work well with the global API number and thin client also. Long term we need to provide versioned APIs and it is better if the client falls back to known good old APIs, because the server certainly can't fall forward. So this is the simple version. Simo, Nathaniel (and others), I know you proposed versions the commands themselves, but I am now not sure how exactly you wanted to do it. What exactly would it mean for the typical extension of our API - adding new parameter and how user command extensions should be treated (command parameter added). I'll let Nathaniel weight in on the technical side as he had better ideas IIRC. From my POV the client needs to find out what calls the server can support and use appropriate ones. At the same time newer servers must support older call versions so that older clients can still work. Simo. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: On 03/20/2015 02:19 PM, Simo Sorce wrote: On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote: Hi guys, I would like to resurrect the discussion we had during DevConf.cz time, about API compatibility in the FreeIPA server. So right now, we maintain the backward compatibility, old clients can talk to newer servers. Forward compatibility is not maintained. Unfortunately, this is not very helpful in real deployments, where the server will often be some RHEL/CentOS system and the client may be the newest Fedora - with newer API than the server. This is the toughest part we need to solve. There 3 main areas we wanted to attack with respect to compatibility: 1) API publishing and/or API browser This is mostly documentation/interactive browser to see the supported API of the server. It should not be difficult, it would just consume the metadata already generated by the server. Ticket: https://fedorahosted.org/freeipa/ticket/3129 2) Forward compatibility of the direct API consumers Until now, to keep newer clients working against older server, we are using the following trick in the ipa-client-install: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install#n1649 It mostly works, one just needs to know the minimal version that needs to be supported. It would be more user friendly, however, if this check is done on the server automatically, without user having to research it. This applies both for ipalib python lib consumer and for direct JSON-RPC consumers. Ticket: https://fedorahosted.org/freeipa/ticket/4739 3) Forward compatibility of the ipa client tool There are different approaches how to fix this, the generally accepted idea was to implement very thin client, which would download and cache metadata from the server on the client and generate the CLI from it. We would only need to have separate client only plugins, basically implementing interactive_promt_callback from existing server side plugins. Tickets: https://fedorahosted.org/freeipa/ticket/4739, https://fedorahosted.org/freeipa/ticket/4768 Now, question is what we can do in 4.2. I do not think we can manage to rewrite ipa command in the thin client, but we should do at least some portion of 1) and 3). I could not decipher that from our Devconf.cz notes. To me, the simplest way of fixing forward compatibility seems to be following steps (besides not making API backwards incompatible - i.e. what we do already): - keep sending API version from client to server - server should not refuse newer API versions - only raise error when an unknown option or unknown command is used This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. Simo. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On Fri, 2015-03-20 at 15:30 +0100, Petr Spacek wrote: On 20.3.2015 14:19, Simo Sorce wrote: This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Currently new clients do not work with older servers, right? Maybe we should do one more (last!) release like that and do a big cut after that. It would make the design so much easier if the new (supposedly thin) client does not need to support ancient servers which had only 'fat' clients. People are using 3.3 and 4.1 now, we want to support them too, whatever it takes. For future clients we can do whatever fancy automated thin client and what not, and in time drop support for very old releases, but 3.3 and 4.2 are going to be around for quite a while and we need to support those server in the clients. Simo. -- Simo Sorce * Red Hat, Inc * New York -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: On 03/20/2015 02:19 PM, Simo Sorce wrote: On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote: Hi guys, I would like to resurrect the discussion we had during DevConf.cz time, about API compatibility in the FreeIPA server. So right now, we maintain the backward compatibility, old clients can talk to newer servers. Forward compatibility is not maintained. Unfortunately, this is not very helpful in real deployments, where the server will often be some RHEL/CentOS system and the client may be the newest Fedora - with newer API than the server. This is the toughest part we need to solve. There 3 main areas we wanted to attack with respect to compatibility: 1) API publishing and/or API browser This is mostly documentation/interactive browser to see the supported API of the server. It should not be difficult, it would just consume the metadata already generated by the server. Ticket: https://fedorahosted.org/freeipa/ticket/3129 2) Forward compatibility of the direct API consumers Until now, to keep newer clients working against older server, we are using the following trick in the ipa-client-install: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install# n1649 It mostly works, one just needs to know the minimal version that needs to be supported. It would be more user friendly, however, if this check is done on the server automatically, without user having to research it. This applies both for ipalib python lib consumer and for direct JSON-RPC consumers. Ticket: https://fedorahosted.org/freeipa/ticket/4739 3) Forward compatibility of the ipa client tool There are different approaches how to fix this, the generally accepted idea was to implement very thin client, which would download and cache metadata from the server on the client and generate the CLI from it. We would only need to have separate client only plugins, basically implementing interactive_promt_callback from existing server side plugins. Tickets: https://fedorahosted.org/freeipa/ticket/4739, https://fedorahosted.org/freeipa/ticket/4768 Now, question is what we can do in 4.2. I do not think we can manage to rewrite ipa command in the thin client, but we should do at least some portion of 1) and 3). I could not decipher that from our Devconf.cz notes. To me, the simplest way of fixing forward compatibility seems to be following steps (besides not making API backwards incompatible - i.e. what we do already): - keep sending API version from client to server - server should not refuse newer API versions - only raise error when an unknown option or unknown command is used This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) -- Petr^2 Spacek -- Manage your
Re: [Freeipa-devel] Designing better API compatibility
On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: On 03/20/2015 02:19 PM, Simo Sorce wrote: On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote: Hi guys, I would like to resurrect the discussion we had during DevConf.cz time, about API compatibility in the FreeIPA server. So right now, we maintain the backward compatibility, old clients can talk to newer servers. Forward compatibility is not maintained. Unfortunately, this is not very helpful in real deployments, where the server will often be some RHEL/CentOS system and the client may be the newest Fedora - with newer API than the server. This is the toughest part we need to solve. There 3 main areas we wanted to attack with respect to compatibility: 1) API publishing and/or API browser This is mostly documentation/interactive browser to see the supported API of the server. It should not be difficult, it would just consume the metadata already generated by the server. Ticket: https://fedorahosted.org/freeipa/ticket/3129 2) Forward compatibility of the direct API consumers Until now, to keep newer clients working against older server, we are using the following trick in the ipa-client-install: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install# n1649 It mostly works, one just needs to know the minimal version that needs to be supported. It would be more user friendly, however, if this check is done on the server automatically, without user having to research it. This applies both for ipalib python lib consumer and for direct JSON-RPC consumers. Ticket: https://fedorahosted.org/freeipa/ticket/4739 3) Forward compatibility of the ipa client tool There are different approaches how to fix this, the generally accepted idea was to implement very thin client, which would download and cache metadata from the server on the client and generate the CLI from it. We would only need to have separate client only plugins, basically implementing interactive_promt_callback from existing server side plugins. Tickets: https://fedorahosted.org/freeipa/ticket/4739, https://fedorahosted.org/freeipa/ticket/4768 Now, question is what we can do in 4.2. I do not think we can manage to rewrite ipa command in the thin client, but we should do at least some portion of 1) and 3). I could not decipher that from our Devconf.cz notes. To me, the simplest way of fixing forward compatibility seems to be following steps (besides not making API backwards incompatible - i.e. what we do already): - keep sending API version from client to server - server should not refuse newer API versions - only raise error when an unknown option or unknown command is used This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this
Re: [Freeipa-devel] Designing better API compatibility
Simo Sorce wrote: On Fri, 2015-03-20 at 15:30 +0100, Petr Spacek wrote: On 20.3.2015 14:19, Simo Sorce wrote: This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Currently new clients do not work with older servers, right? Maybe we should do one more (last!) release like that and do a big cut after that. It would make the design so much easier if the new (supposedly thin) client does not need to support ancient servers which had only 'fat' clients. People are using 3.3 and 4.1 now, we want to support them too, whatever it takes. For future clients we can do whatever fancy automated thin client and what not, and in time drop support for very old releases, but 3.3 and 4.2 are going to be around for quite a while and we need to support those server in the clients. I think 3.0 should be the baseline. It will around another few years at least. rob -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 03/20/2015 04:16 PM, Petr Spacek wrote: On 20.3.2015 15:51, Nathaniel McCallum wrote: On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote: On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote: Correct. I see 2 approaches here: a) Thin client, which simply downloads metadata from the (old) server and won't use unsupported commands/parameters b) Not-so-thin client that knows the minimal API versions of commands/parameters (can be annotated in the code), that would ping the server first to identify it's version, validate that the chosen set of commands/parameters is supported on that server and then send the commands with that version. If we have a recognizable error the client can take an optimistic approach, send the command normally, if it gets an error that the server does not understand it, it checks the version in the reply and falls back to an older baseline version of the command (if possible) or bails out with an error. My understanding was that: 1. We already publish all the information necessary to implement a thin client, and have for some time. We certainly have *some* data but real thin client will most likely require some changes. Some information like return types and so on are missing. 2. Thus, the thin client would work on both new and old versions since it just simply translates from user input into JSON/XML. 3. Only plugins with specific client behavior would need to be ported to the thin client. A prime example of this is otptoken-add-yubikey. My preference is solidly for implementing the thin client first. Once we have decoupled the client from the current plugin framework, server- side changes can be made in isolation. This decoupling is the move that is essentially necessary to provide proper API versioning. And if this can't land for 4.2, land it in the next release. I'd rather do API-stability correctly and a release later than rushed with compromises. We have to live with this forever. + all votes I have :-) +1 -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Designing better API compatibility
On 03/20/2015 04:46 PM, Rob Crittenden wrote: Simo Sorce wrote: On Fri, 2015-03-20 at 15:30 +0100, Petr Spacek wrote: On 20.3.2015 14:19, Simo Sorce wrote: This is not sufficient, older 3.3 and 4.x servers can't be changed and we MUST be compatible with those. Basically the plan MUST work with already released servers, this is a constraint that cannot be releaxed, please work within this limitations. Currently new clients do not work with older servers, right? Maybe we should do one more (last!) release like that and do a big cut after that. It would make the design so much easier if the new (supposedly thin) client does not need to support ancient servers which had only 'fat' clients. People are using 3.3 and 4.1 now, we want to support them too, whatever it takes. For future clients we can do whatever fancy automated thin client and what not, and in time drop support for very old releases, but 3.3 and 4.2 are going to be around for quite a while and we need to support those server in the clients. I think 3.0 should be the baseline. It will around another few years at least. rob Yes, 3.0 is base line. It is the version shipped in RHEL/CentOS-6 and until that dies*, we should be friendly with it. * Looking at https://access.redhat.com/support/policy/updates/errata, this looks like year ~2020 :-) -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
