Hi Samitha, 1. Is it possible to migrate subscriptions with the applications? Further, I guess existing tokens will be invalid once migration happened right? 2. would you still plan to provide a UI for the import-export tool? it would be really nice if we can.
Regards Roshan On Tue, Oct 30, 2018 at 5:17 PM Samitha Chathuranga <[email protected]> wrote: > Hi, > > This is an update on the status of this migration implementation. > > REST API and related Impl/DAO layer changes are done as discussed > previously and below is a summary on that. [1]. I'm creating the WUM update > for 2.6.0 for these changes (PRs : [2] & [3] ) and will be released within > few days. > > *Description on the WUM update* > > Cross tenant APIs/Apps export support through REST APIs is required for > the APIM 2.6.0/2.x to APIM 3.0.0 migration. This was not available > previously. Changes done are described below. > *For exporting APIs* Get APIs list > > - Change Publisher REST API to enable Cross tenant API list retrieval > - Add new parameter to the /apis GET to specify tenant > - Use conditions to return APIs of another tenant > - System parameter set at server startup : migrationMode > - Allowed only If the user is a super tenant admin and server is > started with -DmigrationMode=true property set. > > Export APIs > > - Import-export REST API (web app) changes (via starting tenantFlow) > to enable Cross tenant API Exporting. > - This will be allowed only to super-tenant admin user and only if the > ‘migrationMode’ system property is set. > > *For exporting Applications* Get list of Applications For Applications > export > > - Change Admin REST API - /applications GET to get list of all the > apps for the given tenant/cross tenants too. (previously, only getting the > applications accessible to a single user was possible ) > - Add new parameter to the /applications GET to specify tenant domain > - Validate special permissions to allow only the super tenant admin > users, only when the server is started with migrationMode=true property, to > get cross tenant application list > > For exporting Applications > > - Change Admin REST API - /export/applications GET to export Apps of > the given tenant/cross tenants too. > - Return application keys too (conditionally) to use for migration > - Validate special permissions to allow only the super tenant admin > users, only when the server is started with migrationMode=true property, > for cross tenant apps export > > ------ > > Tested these changes though multiple approaches and no issues found. > > Created applications in store via different tenant users, tenant admins > and tested the admin REST API whether cross tenant application list > retrieval and App exporting is working as expected, adhering to the > migrationMode system property and super admin user role without violating > the expected policies. > > Created APIs in publisher via different tenant users, tenant admins and > tested the Publisher REST API and Import-export REST API, whether the cross > tenant api list retrieval and api exporting is working as expected , > adhering to the migrationMode property and super admin user role without > violating the expected policies. > > And also executed product integration tests on a pack in which > migrationMode system property is enabled too. > > [1] https://github.com/wso2/product-apim/issues/3901 > [2] https://github.com/wso2-support/carbon-apimgt/pull/1138 > [3] https://github.com/wso2-support/product-apim/pull/325 > > Regards, > Samitha > > On Fri, Sep 21, 2018 at 11:11 AM Samitha Chathuranga <[email protected]> > wrote: > >> Hi, >> >> @Amila Maha Arachchi <[email protected]> >> >>> For me, writing a separate API is easier for us and the users both. >>> >> As Nuwan has described above, supporting to be migrated from any version >> (2.1.0/2.2.0/2.5.0..) can be a complex and error prone task and so not >> healthy for us. Mainly it would need adding new REST APIs or changing >> existing APIs to all these older versions and that is cumbersome. >> >> @Dushan Abeyruwan <[email protected]> >> >>> The other question is; its better CLI tool having a verbose option to be >>> implemented. maybe through (-v) which prints all the necessary details >>> around the migration state rather someone to enable that via log4j >>> separately. >>> >> Verbose mode is already configured in import-export tool and so available >> for all the commands. So it is upto us to drop some logs from non-verbose >> mode and include only in verbose mode. My feeling is that it is better to >> log as much information in a non verbose mode too, because that would help >> us dig through errors by default. Migration is not something done always >> hence the logs are not a thing which would continuously pile up. And if we >> limit some logs to verbose mode, then we may require to request our >> customers to enable verbose mode and do migration again and send the logs. >> >> Regards, >> Samitha >> >> On Fri, Sep 21, 2018 at 10:02 AM Nuwan Dias <[email protected]> wrote: >> >>> Let's take this incrementally. We need to do lots of changes on every >>> version to support this kind of migration. In addition to that the changes >>> between the versions are not identical. So every migration needs duplicated >>> work and testing to be carried out. Plus whatever changes we do to each of >>> those old version have to shipped as WUM updates and the clients needs to >>> get those WUM updates for proceeding. We all know that allowing too many >>> options ends up with too many problems, in the end of the day ending up >>> with too many frustrated customers. >>> >>> The migration to API Manager 3.0 is going to be the most sophisticated >>> API Manager migration we have done so far. Given our history of migration >>> experiences trying to do too many thing first up is not a good idea. So I >>> really want us to focus on 2.6 to 3.0 migration and make it really smooth >>> and robust before attempting anything else. >>> >>> Thanks, >>> NuwanD. >>> >>> On Fri, Sep 21, 2018 at 9:51 AM Amila Maha Arachchi <[email protected]> >>> wrote: >>> >>>> Dushan has a very good point (but I think Dushan has or expressed it >>>> wrong). It seems that it has been decided to go with Approach 1. i.e. >>>> adding the migration capabilities to the existing REST apis of 2.6.0. This >>>> means, people using other versions such as 2.1.0, 2.5.0 wont be able to >>>> migrate to 3.0 without coming to 2.6.0. >>>> >>>> If we had a separate API which can export from any version (lets say >>>> from any version from 2.1.0 upwards) and import to 3.0, that is the ideal >>>> solution. It is easy to fix issues as well. It is true that this tool will >>>> be thrown away after sometime. But, we shouldn't expect all the code we >>>> write to be used forever. >>>> >>>> For me, writing a separate API is easier for us and the users both. >>>> >>>> On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan <[email protected]> >>>> wrote: >>>> >>>>> Hi Samitha, >>>>> Are we proceeding with the approach-2? if so, how does such >>>>> migration approach benefited for someone who is trying to migrate from >>>>> 2.5.0 or before? will there be any intermittent steps or this tool would >>>>> take care of such intermittent steps? >>>>> >>>>> The other question is; its better CLI tool having a verbose option to >>>>> be implemented. maybe through (-v) which prints all the necessary details >>>>> around the migration state rather someone to enable that via log4j >>>>> separately. >>>>> >>>>> Cheers, >>>>> Dushan >>>>> >>>>> On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Following is an update on the current progress of this task. >>>>>> >>>>>> Tasks done upto: >>>>>> >>>>>> 1. 2.6.0 REST API and implementation changes. >>>>>> >>>>>> - Change GET applications REST API to enable get all apps of a >>>>>> given tenant too, instead of all tenants >>>>>> - Support cross tenant APIs GET for migration only for super admin >>>>>> - Return application keys too (conditionally) to use for migration >>>>>> - Support cross tenant application export for migration for super >>>>>> admin >>>>>> >>>>>> 2. Export APIs as a bulk with export-apis command with import-export >>>>>> tool >>>>>> >>>>>> - All the APIs are exported by single command but implementation >>>>>> is as iterative api export, set by set separately (export sets of APIs >>>>>> separately, instead of exporting all at once) >>>>>> - API export resume capability (error handling) with the support >>>>>> of two files migration-apis-export-metadata.yaml and >>>>>> last-succeeded-api.log >>>>>> - *--force* flag with *export-apis* command will forcefully >>>>>> export all the apis from beginning, disregarding the fact that >>>>>> all/some >>>>>> apis were exported successfully by last execution of command. >>>>>> - Able to skip exporting particular APIs (if they were failed >>>>>> once and not needed to export if execute the command next time without >>>>>> --force option) >>>>>> >>>>>> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0 >>>>>> import-export tool related to API import/export process. >>>>>> >>>>>> Slight changes on the design of the approach is updated in the same >>>>>> doc shared previously. [1] >>>>>> Will be updating this thread on the progress and appreciate any >>>>>> concerns. >>>>>> >>>>>> [1] >>>>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >>>>>> >>>>>> Regards, >>>>>> Samitha >>>>>> >>>>>> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> This is an update on the up to now state on this task. Had several >>>>>>> discussions, design reviews within team and below doc [1] contains the >>>>>>> overall design of migration process. And each operation executed at API >>>>>>> export/import process (for migration) is detailed under Section 7 >>>>>>> (*Using >>>>>>> the Import Export CLI tool for migration - User Steps and Steps >>>>>>> executed in >>>>>>> each command*). >>>>>>> >>>>>>> Appreciate any comments and suggestions. >>>>>>> >>>>>>> Meanwhile I have worked on changing required REST APIs for this >>>>>>> migration process and are almost done. API/App export related operations >>>>>>> from CLI tool is done upto an extent allowing to be finalized based on >>>>>>> finalized design. >>>>>>> >>>>>>> [1] >>>>>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >>>>>>> >>>>>>> Regards, >>>>>>> Samitha >>>>>>> >>>>>>> On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Nuwan, >>>>>>>> >>>>>>>> One main reason is that, if we follow approach 1, migration story >>>>>>>> cannot be done in single step. Specially two CLI tools will have >>>>>>>> to be used for the two sub steps of exporting and importing. >>>>>>>> >>>>>>>> And also normal API/Application export flow will change as major >>>>>>>> modifications will be required for certain aspects. i.e. >>>>>>>> >>>>>>>> - bulk export/import >>>>>>>> - correlate the difference in the format of archives supported >>>>>>>> by 2.6 vs 3.0 >>>>>>>> - exporting and importing additional artifacts such as throttle >>>>>>>> policies, access tokens, user role permissions >>>>>>>> >>>>>>>> Importing the 2.6 format API/Applications to 3.0 would require >>>>>>>> completely new effort too. So anyway we will have to change the 3.0 >>>>>>>> import >>>>>>>> tool highly. >>>>>>>> >>>>>>>> And my perception is that normal import export and migration >>>>>>>> oriented export-import are tow separate tasks and thought it is better >>>>>>>> to >>>>>>>> isolate the two objectives. (as similarly we do in earlier versions). >>>>>>>> We >>>>>>>> will also have to think about specific permissions required for the >>>>>>>> migration oriented export-import in same tool. >>>>>>>> >>>>>>>> Eventhough we may go with Approach 2, we can reuse the same code >>>>>>>> bases when possible. >>>>>>>> >>>>>>>> But anyway it is a matter of decision, whether we manage the normal >>>>>>>> import/export and migration-oriented-import/export separately. >>>>>>>> Approach 1 >>>>>>>> would result with some higher level of import/export while in approach >>>>>>>> 2, >>>>>>>> we could highlight it as a migration. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Samitha >>>>>>>> >>>>>>>> On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I would rather go with approach 1. In approach 2 you are >>>>>>>>> suggesting to develop a separate web app and a separate tool, why do >>>>>>>>> you >>>>>>>>> think it is better than enhancing the current App and CLI? Its >>>>>>>>> obviously >>>>>>>>> going to be a lot more work than approach 1, plus at the end of the >>>>>>>>> day we >>>>>>>>> will throw away one of those tools because we don't need two tools >>>>>>>>> doing >>>>>>>>> the same thing. >>>>>>>>> >>>>>>>>> On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> This is regarding the initial design on possible approaches to >>>>>>>>>>> proceed with upgrading APIM 2.6.0(to be released) to 3.0.0. The >>>>>>>>>>> migration >>>>>>>>>>> process has been already decided to implement through an >>>>>>>>>>> export/import >>>>>>>>>>> approach as a database migration approach won't be realistic as for >>>>>>>>>>> older >>>>>>>>>>> versions. So the simple generic approach is that each artifact >>>>>>>>>>> specially >>>>>>>>>>> APIs and Applications would be exported from 2.6.0 environment and >>>>>>>>>>> then >>>>>>>>>>> will be imported in to 3.0.0 environment. >>>>>>>>>>> >>>>>>>>>>> Below I have listed the possible two approaches for both these >>>>>>>>>>> two steps (importing & exporting) followed by their pros and cons. >>>>>>>>>>> >>>>>>>>>>> Exporting : >>>>>>>>>>> >>>>>>>>>>> Approach 1 : Change currently existing Api Import/export web app >>>>>>>>>>> and CLI tool in 2.6 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Can use API Import/export Web app and CLI tool and modify >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> (REST APIs for export api/applications is currently >>>>>>>>>>> provided via this WEB app. CLI tool uses this web app) >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Required changes: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> For app export: Consumer key, Consumer Secret, Token >>>>>>>>>>> exporting should be supported in CLI tool >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Will have to change admin REST api implementation for >>>>>>>>>>> above task and for exporting other artifacts mentioned at the >>>>>>>>>>> end. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Modify CLI tool to support exporting in bulk. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Pros >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Will not have to develop/maintain a Web app and CLI tool >>>>>>>>>>> from the scratch >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Cons >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> API export flow will change as modifications will be >>>>>>>>>>> required >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Have to manage and consider also about normal export >>>>>>>>>>> feature when doing changes for migration-exporting. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Migration story cannot be done in single step. Exporting >>>>>>>>>>> and importing will be 2 step process. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Approach 2 : Develop a separate tool without using API >>>>>>>>>>> Import/export Web app or CLI tool. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Create a WEB App introducing new REST APIs required and >>>>>>>>>>> develop separate CLI tool. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Pros >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Migration story can be done in single step. (Exporting >>>>>>>>>>> and importing can be done in single command.) >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> API/Application import-export tool won’t have any effect >>>>>>>>>>> or change. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Don’t have to manage 2 separate use cases in single tool. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Cons >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Will have to develop/maintain a Web app and/or CLI tool >>>>>>>>>>> from the scratch >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Importing : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Has to create separate REST APIs for importing 2.6-exported >>>>>>>>>>> APIs/Applications/other artifacts as 3.0.0 expecting archives in >>>>>>>>>>> 3.0 >>>>>>>>>>> specific format. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Approach 1 >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Change/improve API Import/export CLI tool in 3.0 to support >>>>>>>>>>> importing 2.6-exported API/application archives. >>>>>>>>>>> >>>>>>>>>>> Approach 2 >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Develop separate CLI tool from the scratch which will be >>>>>>>>>>> used for exporting from 2.6 also >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Other artifacts required to be exported/imported (other than >>>>>>>>>>> APIs/Applications) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Throttling Policies >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Custom API Lifecycles >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Workflows >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> User Role Permissions >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Custom sequences (at least for reference -> has to decide >>>>>>>>>>> regarding this) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> My perception is that proceeding with Approach 2 under both >>>>>>>>>>> exporting and importing would be better considering the pros and >>>>>>>>>>> cons of >>>>>>>>>>> each. >>>>>>>>>>> >>>>>>>>>>> Highly appreciate your thoughts on this. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Samitha >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Samitha Chathuranga* >>>>>>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>> Mobile: +94715123761 >>>>>>>>>>> >>>>>>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Nuwan Dias >>>>>>>>> >>>>>>>>> Director - WSO2, Inc. http://wso2.com >>>>>>>>> email : [email protected] >>>>>>>>> Phone : +94 777 775 729 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Samitha Chathuranga* >>>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>>> lean.enterprise.middleware >>>>>>>> Mobile: +94715123761 >>>>>>>> >>>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Samitha Chathuranga* >>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>> lean.enterprise.middleware >>>>>>> Mobile: +94715123761 >>>>>>> >>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Samitha Chathuranga* >>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>> lean.enterprise.middleware >>>>>> Mobile: +94715123761 >>>>>> >>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dushan Abeyruwan >>>>> Architect(Associate Director)-Delivery/Support >>>>> Technical Support,MV >>>>> PMC Member Apache Synpase >>>>> WSO2 Inc. http://wso2.com/ >>>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >>>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >>>>> <https://www.linkedin.com/in/dushanabeyruwan>* >>>>> Mobile:(001)408-987-1348 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>> >>>> >>>> -- >>>> *Amila Maharachchi* >>>> Director of Engineering >>>> WSO2, Inc.; http://wso2.com >>>> >>>> Blog: http://maharachchi.blogspot.com >>>> Mobile: +94719371446 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> >>> -- >>> *Nuwan Dias* | Director | WSO2 Inc. >>> (m) +94 777 775 729 | (e) [email protected] >>> [image: Signature.jpg] >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >> >> >> -- >> *Samitha Chathuranga* >> *Senior Software Engineer*, *WSO2 Inc.* >> lean.enterprise.middleware >> Mobile: +94715123761 >> >> [image: http://wso2.com/signature] <http://wso2.com/signature> >> > > > -- > *Samitha Chathuranga* > *Senior Software Engineer*, *WSO2 Inc.* > lean.enterprise.middleware > Mobile: +94715123761 > > [image: http://wso2.com/signature] <http://wso2.com/signature> > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
