Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

Hi Roshan,

1. Is it possible to migrate subscriptions with the applications? Further,
> I guess existing tokens will be invalid once migration happened right?
>
Migrating subscriptions will be facilitated in the migration process.
And we are not planning to migrate the tokens ATM so, existing tokens will
be invalid.

2. would you still plan to provide a UI for the import-export tool? it
> would be really nice if we can.
>
We are sorry to inform that we do not have a plan to provide a UI for
import-export tool.

Regards,
Samitha

On Wed, Oct 31, 2018 at 3:28 PM roshan wijesena 
wrote:

> Hi Samitha,
>
> 1. Is it possible to migrate subscriptions with the applications? Further,
> I guess existing tokens will be invalid once migration happened right?
> 2. would you still plan to provide a UI for the import-export tool? it
> would be really nice if we can.
>
> Regards
> Roshan
>
> On Tue, Oct 30, 2018 at 5:17 PM Samitha Chathuranga 
> wrote:
>
>> Hi,
>>
>> This is an update on the status of this migration implementation.
>>
>> REST API and related Impl/DAO layer changes are done as discussed
>> previously and below is a summary on that. [1]. I'm creating the WUM update
>> for 2.6.0 for these changes (PRs : [2] & [3] ) and will be released within
>> few days.
>>
>> *Description on the WUM update*
>>
>> Cross tenant APIs/Apps export support through REST APIs is required for
>> the APIM 2.6.0/2.x to APIM 3.0.0 migration. This was not available
>> previously. Changes done are described below.
>> *For exporting APIs* Get APIs list
>>
>>- Change Publisher REST API to enable Cross tenant API list retrieval
>>- Add new parameter to the /apis GET to specify tenant
>>- Use conditions to return APIs of another tenant
>>- System parameter set at server startup : migrationMode
>>- Allowed only If the user is a super tenant admin and server is
>>started with -DmigrationMode=true property set.
>>
>> Export APIs
>>
>>- Import-export REST API (web app) changes (via starting tenantFlow)
>>to enable Cross tenant API Exporting.
>>- This will be allowed only to super-tenant admin user and only if
>>the ‘migrationMode’ system property is set.
>>
>> *For exporting Applications* Get list of Applications For Applications
>> export
>>
>>- Change Admin REST API - /applications GET to get list of all the
>>apps for the given tenant/cross tenants too. (previously, only getting the
>>applications accessible to a single user was possible )
>>- Add new parameter to the /applications GET to specify tenant domain
>>- Validate special permissions to allow only the super tenant admin
>>users, only when the server is started with migrationMode=true property, 
>> to
>>get cross tenant application list
>>
>> For exporting Applications
>>
>>- Change Admin REST API - /export/applications GET to export Apps of
>>the given tenant/cross tenants too.
>>- Return application keys too (conditionally) to use for migration
>>- Validate special permissions to allow only the super tenant admin
>>users, only when the server is started with migrationMode=true property,
>>for cross tenant apps export
>>
>> --
>>
>> Tested these changes though multiple approaches and no issues found.
>>
>> Created applications in store via different tenant users, tenant admins
>> and tested the admin REST API whether cross tenant application list
>> retrieval and App exporting is working as expected, adhering to the
>> migrationMode system property and super admin user role without violating
>> the expected policies.
>>
>> Created APIs in publisher via different tenant users, tenant admins and
>> tested the Publisher REST API and Import-export REST API, whether the cross
>> tenant api list retrieval and api exporting is working as expected ,
>> adhering to the migrationMode property and super admin user role without
>> violating the expected policies.
>>
>> And also executed product integration tests on a pack in which
>> migrationMode system property is enabled too.
>>
>> [1] https://github.com/wso2/product-apim/issues/3901
>> [2] https://github.com/wso2-support/carbon-apimgt/pull/1138
>> [3] https://github.com/wso2-support/product-apim/pull/325
>>
>> Regards,
>> Samitha
>>
>> On Fri, Sep 21, 2018 at 11:11 AM Samitha Chathuranga 
>> wrote:
>>
>>> Hi,
>>>
>>> @Amila Maha Arachchi 
>>>
 For me, writing a separate API is easier for us and the users both.

>>> As Nuwan has described above, supporting to be migrated from any version
>>> (2.1.0/2.2.0/2.5.0..) can  be a complex and error prone task and so not
>>> healthy for us. Mainly it would need adding new REST APIs or changing
>>> existing APIs to all these older versions and that is cumbersome.
>>>
>>> @Dushan Abeyruwan 
>>>
 The other question is; its better CLI tool having a verbose option to
 be implemented.  maybe through (-v) which prints all the necessary details
 around the migration state rather someone to 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[roshan wijesena]

Hi Samitha,

1. Is it possible to migrate subscriptions with the applications? Further,
I guess existing tokens will be invalid once migration happened right?
2. would you still plan to provide a UI for the import-export tool? it
would be really nice if we can.

Regards
Roshan

On Tue, Oct 30, 2018 at 5:17 PM Samitha Chathuranga 
wrote:

> Hi,
>
> This is an update on the status of this migration implementation.
>
> REST API and related Impl/DAO layer changes are done as discussed
> previously and below is a summary on that. [1]. I'm creating the WUM update
> for 2.6.0 for these changes (PRs : [2] & [3] ) and will be released within
> few days.
>
> *Description on the WUM update*
>
> Cross tenant APIs/Apps export support through REST APIs is required for
> the APIM 2.6.0/2.x to APIM 3.0.0 migration. This was not available
> previously. Changes done are described below.
> *For exporting APIs* Get APIs list
>
>- Change Publisher REST API to enable Cross tenant API list retrieval
>- Add new parameter to the /apis GET to specify tenant
>- Use conditions to return APIs of another tenant
>- System parameter set at server startup : migrationMode
>- Allowed only If the user is a super tenant admin and server is
>started with -DmigrationMode=true property set.
>
> Export APIs
>
>- Import-export REST API (web app) changes (via starting tenantFlow)
>to enable Cross tenant API Exporting.
>- This will be allowed only to super-tenant admin user and only if the
>‘migrationMode’ system property is set.
>
> *For exporting Applications* Get list of Applications For Applications
> export
>
>- Change Admin REST API - /applications GET to get list of all the
>apps for the given tenant/cross tenants too. (previously, only getting the
>applications accessible to a single user was possible )
>- Add new parameter to the /applications GET to specify tenant domain
>- Validate special permissions to allow only the super tenant admin
>users, only when the server is started with migrationMode=true property, to
>get cross tenant application list
>
> For exporting Applications
>
>- Change Admin REST API - /export/applications GET to export Apps of
>the given tenant/cross tenants too.
>- Return application keys too (conditionally) to use for migration
>- Validate special permissions to allow only the super tenant admin
>users, only when the server is started with migrationMode=true property,
>for cross tenant apps export
>
> --
>
> Tested these changes though multiple approaches and no issues found.
>
> Created applications in store via different tenant users, tenant admins
> and tested the admin REST API whether cross tenant application list
> retrieval and App exporting is working as expected, adhering to the
> migrationMode system property and super admin user role without violating
> the expected policies.
>
> Created APIs in publisher via different tenant users, tenant admins and
> tested the Publisher REST API and Import-export REST API, whether the cross
> tenant api list retrieval and api exporting is working as expected ,
> adhering to the migrationMode property and super admin user role without
> violating the expected policies.
>
> And also executed product integration tests on a pack in which
> migrationMode system property is enabled too.
>
> [1] https://github.com/wso2/product-apim/issues/3901
> [2] https://github.com/wso2-support/carbon-apimgt/pull/1138
> [3] https://github.com/wso2-support/product-apim/pull/325
>
> Regards,
> Samitha
>
> On Fri, Sep 21, 2018 at 11:11 AM Samitha Chathuranga 
> wrote:
>
>> Hi,
>>
>> @Amila Maha Arachchi 
>>
>>> For me, writing a separate API is easier for us and the users both.
>>>
>> As Nuwan has described above, supporting to be migrated from any version
>> (2.1.0/2.2.0/2.5.0..) can  be a complex and error prone task and so not
>> healthy for us. Mainly it would need adding new REST APIs or changing
>> existing APIs to all these older versions and that is cumbersome.
>>
>> @Dushan Abeyruwan 
>>
>>> The other question is; its better CLI tool having a verbose option to be
>>> implemented.  maybe through (-v) which prints all the necessary details
>>> around the migration state rather someone to enable that via log4j
>>> separately.
>>>
>> Verbose mode is already configured in import-export tool and so available
>> for all the commands. So it is upto us to drop some logs from non-verbose
>> mode and include only in verbose mode. My feeling is that it is better to
>> log as much information in a non verbose mode too, because that would help
>> us dig through errors by default. Migration is not something done always
>> hence the logs are not a thing which would continuously pile up. And if we
>> limit some logs to verbose mode, then we may require to request our
>> customers to enable verbose mode and do migration again and send the logs.
>>
>> Regards,
>> Samitha
>>
>> On Fri, Sep 21, 2018 at 10:02 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

Hi,

This is an update on the status of this migration implementation.

REST API and related Impl/DAO layer changes are done as discussed
previously and below is a summary on that. [1]. I'm creating the WUM update
for 2.6.0 for these changes (PRs : [2] & [3] ) and will be released within
few days.

*Description on the WUM update*

Cross tenant APIs/Apps export support through REST APIs is required for the
APIM 2.6.0/2.x to APIM 3.0.0 migration. This was not available previously.
Changes done are described below.
*For exporting APIs* Get APIs list

   - Change Publisher REST API to enable Cross tenant API list retrieval
   - Add new parameter to the /apis GET to specify tenant
   - Use conditions to return APIs of another tenant
   - System parameter set at server startup : migrationMode
   - Allowed only If the user is a super tenant admin and server is started
   with -DmigrationMode=true property set.

Export APIs

   - Import-export REST API (web app) changes (via starting tenantFlow) to
   enable Cross tenant API Exporting.
   - This will be allowed only to super-tenant admin user and only if the
   ‘migrationMode’ system property is set.

*For exporting Applications* Get list of Applications For Applications
export

   - Change Admin REST API - /applications GET to get list of all the apps
   for the given tenant/cross tenants too. (previously, only getting the
   applications accessible to a single user was possible )
   - Add new parameter to the /applications GET to specify tenant domain
   - Validate special permissions to allow only the super tenant admin
   users, only when the server is started with migrationMode=true property, to
   get cross tenant application list

For exporting Applications

   - Change Admin REST API - /export/applications GET to export Apps of the
   given tenant/cross tenants too.
   - Return application keys too (conditionally) to use for migration
   - Validate special permissions to allow only the super tenant admin
   users, only when the server is started with migrationMode=true property,
   for cross tenant apps export

--

Tested these changes though multiple approaches and no issues found.

Created applications in store via different tenant users, tenant admins and
tested the admin REST API whether cross tenant application list retrieval
and App exporting is working as expected, adhering to the migrationMode
system property and super admin user role without violating the expected
policies.

Created APIs in publisher via different tenant users, tenant admins and
tested the Publisher REST API and Import-export REST API, whether the cross
tenant api list retrieval and api exporting is working as expected ,
adhering to the migrationMode property and super admin user role without
violating the expected policies.

And also executed product integration tests on a pack in which
migrationMode system property is enabled too.

[1] https://github.com/wso2/product-apim/issues/3901
[2] https://github.com/wso2-support/carbon-apimgt/pull/1138
[3] https://github.com/wso2-support/product-apim/pull/325

Regards,
Samitha

On Fri, Sep 21, 2018 at 11:11 AM Samitha Chathuranga 
wrote:

> Hi,
>
> @Amila Maha Arachchi 
>
>> For me, writing a separate API is easier for us and the users both.
>>
> As Nuwan has described above, supporting to be migrated from any version
> (2.1.0/2.2.0/2.5.0..) can  be a complex and error prone task and so not
> healthy for us. Mainly it would need adding new REST APIs or changing
> existing APIs to all these older versions and that is cumbersome.
>
> @Dushan Abeyruwan 
>
>> The other question is; its better CLI tool having a verbose option to be
>> implemented.  maybe through (-v) which prints all the necessary details
>> around the migration state rather someone to enable that via log4j
>> separately.
>>
> Verbose mode is already configured in import-export tool and so available
> for all the commands. So it is upto us to drop some logs from non-verbose
> mode and include only in verbose mode. My feeling is that it is better to
> log as much information in a non verbose mode too, because that would help
> us dig through errors by default. Migration is not something done always
> hence the logs are not a thing which would continuously pile up. And if we
> limit some logs to verbose mode, then we may require to request our
> customers to enable verbose mode and do migration again and send the logs.
>
> Regards,
> Samitha
>
> On Fri, Sep 21, 2018 at 10:02 AM Nuwan Dias  wrote:
>
>> Let's take this incrementally. We need to do lots of changes on every
>> version to support this kind of migration. In addition to that the changes
>> between the versions are not identical. So every migration needs duplicated
>> work and testing to be carried out. Plus whatever changes we do to each of
>> those old version have to shipped as WUM updates and the clients needs to
>> get those WUM updates for proceeding. We all know that allowing too many
>> options ends up with 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

Hi,

@Amila Maha Arachchi 

> For me, writing a separate API is easier for us and the users both.
>
As Nuwan has described above, supporting to be migrated from any version
(2.1.0/2.2.0/2.5.0..) can  be a complex and error prone task and so not
healthy for us. Mainly it would need adding new REST APIs or changing
existing APIs to all these older versions and that is cumbersome.

@Dushan Abeyruwan 

> The other question is; its better CLI tool having a verbose option to be
> implemented.  maybe through (-v) which prints all the necessary details
> around the migration state rather someone to enable that via log4j
> separately.
>
Verbose mode is already configured in import-export tool and so available
for all the commands. So it is upto us to drop some logs from non-verbose
mode and include only in verbose mode. My feeling is that it is better to
log as much information in a non verbose mode too, because that would help
us dig through errors by default. Migration is not something done always
hence the logs are not a thing which would continuously pile up. And if we
limit some logs to verbose mode, then we may require to request our
customers to enable verbose mode and do migration again and send the logs.

Regards,
Samitha

On Fri, Sep 21, 2018 at 10:02 AM Nuwan Dias  wrote:

> Let's take this incrementally. We need to do lots of changes on every
> version to support this kind of migration. In addition to that the changes
> between the versions are not identical. So every migration needs duplicated
> work and testing to be carried out. Plus whatever changes we do to each of
> those old version have to shipped as WUM updates and the clients needs to
> get those WUM updates for proceeding. We all know that allowing too many
> options ends up with too many problems, in the end of the day ending up
> with too many frustrated customers.
>
> The migration to API Manager 3.0 is going to be the most sophisticated API
> Manager migration we have done so far. Given our history of migration
> experiences trying to do too many thing first up is not a good idea. So I
> really want us to focus on 2.6 to 3.0 migration and make it really smooth
> and robust before attempting anything else.
>
> Thanks,
> NuwanD.
>
> On Fri, Sep 21, 2018 at 9:51 AM Amila Maha Arachchi 
> wrote:
>
>> Dushan has a very good point (but I think Dushan has or expressed it
>> wrong). It seems that it has been decided to go with Approach 1. i.e.
>> adding the migration capabilities to the existing REST apis of 2.6.0. This
>> means, people using other versions such as 2.1.0, 2.5.0 wont be able to
>> migrate to 3.0 without coming to 2.6.0.
>>
>> If we had a separate API which can export from any version (lets say from
>> any version from 2.1.0 upwards) and import to 3.0, that is the ideal
>> solution. It is easy to fix issues as well. It is true that this tool will
>> be thrown away after sometime. But, we shouldn't expect all the code we
>> write to be used forever.
>>
>> For me, writing a separate API is easier for us and the users both.
>>
>> On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan  wrote:
>>
>>> Hi Samitha,
>>>   Are we proceeding with the approach-2? if so, how does such migration
>>> approach benefited for someone who is trying to migrate from 2.5.0 or
>>> before? will there be any intermittent steps or this tool would take care
>>> of such intermittent steps?
>>>
>>> The other question is; its better CLI tool having a verbose option to be
>>> implemented.  maybe through (-v) which prints all the necessary details
>>> around the migration state rather someone to enable that via log4j
>>> separately.
>>>
>>> Cheers,
>>> Dushan
>>>
>>> On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga 
>>> wrote:
>>>
 Hi all,

 Following is an update on the current progress of this task.

 Tasks done upto:

 1. 2.6.0 REST API and implementation changes.

- Change GET applications REST API to enable get all apps of a
given tenant too, instead of all tenants
- Support cross tenant APIs GET for migration only for super admin
- Return application keys too (conditionally) to use for migration
- Support cross tenant application export for migration for super
admin

 2. Export APIs as a bulk with export-apis command with import-export
 tool

- All the APIs are exported by single command but implementation is
as iterative api export, set by set separately (export sets of APIs
separately, instead of exporting all at once)
- API export resume capability (error handling) with the support of
two files migration-apis-export-metadata.yaml and last-succeeded-api.log
- *--force* flag with *export-apis* command will forcefully export
all the apis from beginning, disregarding the fact that all/some apis 
 were
exported successfully by last execution of command.
- Able to skip exporting particular 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Nuwan Dias]

Let's take this incrementally. We need to do lots of changes on every
version to support this kind of migration. In addition to that the changes
between the versions are not identical. So every migration needs duplicated
work and testing to be carried out. Plus whatever changes we do to each of
those old version have to shipped as WUM updates and the clients needs to
get those WUM updates for proceeding. We all know that allowing too many
options ends up with too many problems, in the end of the day ending up
with too many frustrated customers.

The migration to API Manager 3.0 is going to be the most sophisticated API
Manager migration we have done so far. Given our history of migration
experiences trying to do too many thing first up is not a good idea. So I
really want us to focus on 2.6 to 3.0 migration and make it really smooth
and robust before attempting anything else.

Thanks,
NuwanD.

On Fri, Sep 21, 2018 at 9:51 AM Amila Maha Arachchi  wrote:

> Dushan has a very good point (but I think Dushan has or expressed it
> wrong). It seems that it has been decided to go with Approach 1. i.e.
> adding the migration capabilities to the existing REST apis of 2.6.0. This
> means, people using other versions such as 2.1.0, 2.5.0 wont be able to
> migrate to 3.0 without coming to 2.6.0.
>
> If we had a separate API which can export from any version (lets say from
> any version from 2.1.0 upwards) and import to 3.0, that is the ideal
> solution. It is easy to fix issues as well. It is true that this tool will
> be thrown away after sometime. But, we shouldn't expect all the code we
> write to be used forever.
>
> For me, writing a separate API is easier for us and the users both.
>
> On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan  wrote:
>
>> Hi Samitha,
>>   Are we proceeding with the approach-2? if so, how does such migration
>> approach benefited for someone who is trying to migrate from 2.5.0 or
>> before? will there be any intermittent steps or this tool would take care
>> of such intermittent steps?
>>
>> The other question is; its better CLI tool having a verbose option to be
>> implemented.  maybe through (-v) which prints all the necessary details
>> around the migration state rather someone to enable that via log4j
>> separately.
>>
>> Cheers,
>> Dushan
>>
>> On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga 
>> wrote:
>>
>>> Hi all,
>>>
>>> Following is an update on the current progress of this task.
>>>
>>> Tasks done upto:
>>>
>>> 1. 2.6.0 REST API and implementation changes.
>>>
>>>- Change GET applications REST API to enable get all apps of a given
>>>tenant too, instead of all tenants
>>>- Support cross tenant APIs GET for migration only for super admin
>>>- Return application keys too (conditionally) to use for migration
>>>- Support cross tenant application export for migration for super
>>>admin
>>>
>>> 2. Export APIs as a bulk with export-apis command with import-export tool
>>>
>>>- All the APIs are exported by single command but implementation is
>>>as iterative api export, set by set separately (export sets of APIs
>>>separately, instead of exporting all at once)
>>>- API export resume capability (error handling) with the support of
>>>two files migration-apis-export-metadata.yaml and last-succeeded-api.log
>>>- *--force* flag with *export-apis* command will forcefully export
>>>all the apis from beginning, disregarding the fact that all/some apis 
>>> were
>>>exported successfully by last execution of command.
>>>- Able to skip exporting particular APIs (if they were failed once
>>>and not needed to export if execute the command next time without --force
>>>option)
>>>
>>> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0
>>> import-export tool related to API import/export process.
>>>
>>> Slight changes on the design of the approach is updated in the same doc
>>> shared previously. [1]
>>> Will be updating this thread on the progress and appreciate any concerns.
>>>
>>> [1]
>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing
>>>
>>> Regards,
>>> Samitha
>>>
>>> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga 
>>> wrote:
>>>
 Hi all,

 This is an update on the up to now state on this task. Had several
 discussions, design reviews within team and below doc [1] contains the
 overall design of migration process. And each operation executed at API
 export/import process (for migration) is detailed under Section 7 (*Using
 the Import Export CLI tool for migration - User Steps and Steps executed in
 each command*).

 Appreciate any comments and suggestions.

 Meanwhile I have worked on changing required REST APIs for this
 migration process and are almost done. API/App export related operations
 from CLI tool is done upto an extent allowing to be finalized based on
 finalized design.

 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Amila Maha Arachchi]

Dushan has a very good point (but I think Dushan has or expressed it
wrong). It seems that it has been decided to go with Approach 1. i.e.
adding the migration capabilities to the existing REST apis of 2.6.0. This
means, people using other versions such as 2.1.0, 2.5.0 wont be able to
migrate to 3.0 without coming to 2.6.0.

If we had a separate API which can export from any version (lets say from
any version from 2.1.0 upwards) and import to 3.0, that is the ideal
solution. It is easy to fix issues as well. It is true that this tool will
be thrown away after sometime. But, we shouldn't expect all the code we
write to be used forever.

For me, writing a separate API is easier for us and the users both.

On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan  wrote:

> Hi Samitha,
>   Are we proceeding with the approach-2? if so, how does such migration
> approach benefited for someone who is trying to migrate from 2.5.0 or
> before? will there be any intermittent steps or this tool would take care
> of such intermittent steps?
>
> The other question is; its better CLI tool having a verbose option to be
> implemented.  maybe through (-v) which prints all the necessary details
> around the migration state rather someone to enable that via log4j
> separately.
>
> Cheers,
> Dushan
>
> On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga 
> wrote:
>
>> Hi all,
>>
>> Following is an update on the current progress of this task.
>>
>> Tasks done upto:
>>
>> 1. 2.6.0 REST API and implementation changes.
>>
>>- Change GET applications REST API to enable get all apps of a given
>>tenant too, instead of all tenants
>>- Support cross tenant APIs GET for migration only for super admin
>>- Return application keys too (conditionally) to use for migration
>>- Support cross tenant application export for migration for super
>>admin
>>
>> 2. Export APIs as a bulk with export-apis command with import-export tool
>>
>>- All the APIs are exported by single command but implementation is
>>as iterative api export, set by set separately (export sets of APIs
>>separately, instead of exporting all at once)
>>- API export resume capability (error handling) with the support of
>>two files migration-apis-export-metadata.yaml and last-succeeded-api.log
>>- *--force* flag with *export-apis* command will forcefully export
>>all the apis from beginning, disregarding the fact that all/some apis were
>>exported successfully by last execution of command.
>>- Able to skip exporting particular APIs (if they were failed once
>>and not needed to export if execute the command next time without --force
>>option)
>>
>> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0
>> import-export tool related to API import/export process.
>>
>> Slight changes on the design of the approach is updated in the same doc
>> shared previously. [1]
>> Will be updating this thread on the progress and appreciate any concerns.
>>
>> [1]
>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing
>>
>> Regards,
>> Samitha
>>
>> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga 
>> wrote:
>>
>>> Hi all,
>>>
>>> This is an update on the up to now state on this task. Had several
>>> discussions, design reviews within team and below doc [1] contains the
>>> overall design of migration process. And each operation executed at API
>>> export/import process (for migration) is detailed under Section 7 (*Using
>>> the Import Export CLI tool for migration - User Steps and Steps executed in
>>> each command*).
>>>
>>> Appreciate any comments and suggestions.
>>>
>>> Meanwhile I have worked on changing required REST APIs for this
>>> migration process and are almost done. API/App export related operations
>>> from CLI tool is done upto an extent allowing to be finalized based on
>>> finalized design.
>>>
>>> [1]
>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing
>>>
>>> Regards,
>>> Samitha
>>>
>>> On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga 
>>> wrote:
>>>
 Hi Nuwan,

 One main reason is that, if we follow approach 1, migration story
 cannot be done in single step. Specially two CLI tools will have to be
 used for the two sub steps of exporting and importing.

 And also normal API/Application export flow will change as major
 modifications will be required for certain aspects. i.e.

- bulk export/import
- correlate the difference in the format of archives supported by
2.6 vs 3.0
- exporting and importing additional artifacts such as throttle
policies, access tokens, user role permissions

 Importing the 2.6 format API/Applications to 3.0 would require
 completely new effort too. So anyway we will have to change the 3.0 import
 tool highly.

 And my perception is that normal import export and migration oriented

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Dushan Abeyruwan]

Hi Samitha,
  Are we proceeding with the approach-2? if so, how does such migration
approach benefited for someone who is trying to migrate from 2.5.0 or
before? will there be any intermittent steps or this tool would take care
of such intermittent steps?

The other question is; its better CLI tool having a verbose option to be
implemented.  maybe through (-v) which prints all the necessary details
around the migration state rather someone to enable that via log4j
separately.

Cheers,
Dushan

On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga 
wrote:

> Hi all,
>
> Following is an update on the current progress of this task.
>
> Tasks done upto:
>
> 1. 2.6.0 REST API and implementation changes.
>
>- Change GET applications REST API to enable get all apps of a given
>tenant too, instead of all tenants
>- Support cross tenant APIs GET for migration only for super admin
>- Return application keys too (conditionally) to use for migration
>- Support cross tenant application export for migration for super admin
>
> 2. Export APIs as a bulk with export-apis command with import-export tool
>
>- All the APIs are exported by single command but implementation is as
>iterative api export, set by set separately (export sets of APIs
>separately, instead of exporting all at once)
>- API export resume capability (error handling) with the support of
>two files migration-apis-export-metadata.yaml and
>last-succeeded-api.log
>- *--force* flag with *export-apis* command will forcefully export all
>the apis from beginning, disregarding the fact that all/some apis were
>exported successfully by last execution of command.
>- Able to skip exporting particular APIs (if they were failed once and
>not needed to export if execute the command next time without --force
>option)
>
> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0
> import-export tool related to API import/export process.
>
> Slight changes on the design of the approach is updated in the same doc
> shared previously. [1]
> Will be updating this thread on the progress and appreciate any concerns.
>
> [1] https://docs.google.com/document/d/1N8ZXPf1-
> dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing
>
> Regards,
> Samitha
>
> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga 
> wrote:
>
>> Hi all,
>>
>> This is an update on the up to now state on this task. Had several
>> discussions, design reviews within team and below doc [1] contains the
>> overall design of migration process. And each operation executed at API
>> export/import process (for migration) is detailed under Section 7 (*Using
>> the Import Export CLI tool for migration - User Steps and Steps executed in
>> each command*).
>>
>> Appreciate any comments and suggestions.
>>
>> Meanwhile I have worked on changing required REST APIs for this migration
>> process and are almost done. API/App export related operations from CLI
>> tool is done upto an extent allowing to be finalized based on finalized
>> design.
>>
>> [1] https://docs.google.com/document/d/1N8ZXPf1-
>> dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing
>>
>> Regards,
>> Samitha
>>
>> On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga 
>> wrote:
>>
>>> Hi Nuwan,
>>>
>>> One main reason is that, if we follow approach 1, migration story
>>> cannot be done in single step. Specially two CLI tools will have to be
>>> used for the two sub steps of exporting and importing.
>>>
>>> And also normal API/Application export flow will change as major
>>> modifications will be required for certain aspects. i.e.
>>>
>>>- bulk export/import
>>>- correlate the difference in the format of archives supported by
>>>2.6 vs 3.0
>>>- exporting and importing additional artifacts such as throttle
>>>policies, access tokens, user role permissions
>>>
>>> Importing the 2.6 format API/Applications to 3.0 would require
>>> completely new effort too. So anyway we will have to change the 3.0 import
>>> tool highly.
>>>
>>> And my perception is that normal import export and migration oriented
>>> export-import are tow separate tasks and thought it is better to isolate
>>> the two objectives. (as similarly we do in earlier versions). We will also
>>> have to think about specific permissions required for the migration
>>> oriented export-import in same tool.
>>>
>>> Eventhough we may go with Approach 2, we can reuse the same code bases
>>> when possible.
>>>
>>> But anyway it is a matter of decision, whether we manage the normal
>>> import/export and migration-oriented-import/export separately. Approach
>>> 1 would result with some higher level of import/export while in approach 2,
>>> we could highlight it as a migration.
>>>
>>> Regards,
>>> Samitha
>>>
>>> On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias  wrote:
>>>
 I would rather go with approach 1. In approach 2 you are suggesting to
 develop a separate web app and a separate tool, why do you think it is

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

Hi all,

This is an update on the up to now state on this task. Had several
discussions, design reviews within team and below doc [1] contains the
overall design of migration process. And each operation executed at API
export/import process (for migration) is detailed under Section 7 (*Using
the Import Export CLI tool for migration - User Steps and Steps executed in
each command*).

Appreciate any comments and suggestions.

Meanwhile I have worked on changing required REST APIs for this migration
process and are almost done. API/App export related operations from CLI
tool is done upto an extent allowing to be finalized based on finalized
design.

[1]
https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing

Regards,
Samitha

On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga 
wrote:

> Hi Nuwan,
>
> One main reason is that, if we follow approach 1, migration story cannot
> be done in single step. Specially two CLI tools will have to be used for
> the two sub steps of exporting and importing.
>
> And also normal API/Application export flow will change as major
> modifications will be required for certain aspects. i.e.
>
>- bulk export/import
>- correlate the difference in the format of archives supported by 2.6
>vs 3.0
>- exporting and importing additional artifacts such as throttle
>policies, access tokens, user role permissions
>
> Importing the 2.6 format API/Applications to 3.0 would require completely
> new effort too. So anyway we will have to change the 3.0 import tool highly.
>
> And my perception is that normal import export and migration oriented
> export-import are tow separate tasks and thought it is better to isolate
> the two objectives. (as similarly we do in earlier versions). We will also
> have to think about specific permissions required for the migration
> oriented export-import in same tool.
>
> Eventhough we may go with Approach 2, we can reuse the same code bases
> when possible.
>
> But anyway it is a matter of decision, whether we manage the normal
> import/export and migration-oriented-import/export separately. Approach 1
> would result with some higher level of import/export while in approach 2,
> we could highlight it as a migration.
>
> Regards,
> Samitha
>
> On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias  wrote:
>
>> I would rather go with approach 1. In approach 2 you are suggesting to
>> develop a separate web app and a separate tool, why do you think it is
>> better than enhancing the current App and CLI? Its obviously going to be a
>> lot more work than approach 1, plus at the end of the day we will throw
>> away one of those tools because we don't need two tools doing the same
>> thing.
>>
>> On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga 
>> wrote:
>>
>>>
>>> Hi all,

 This is regarding the initial design on possible approaches to proceed
 with upgrading APIM 2.6.0(to be released) to 3.0.0. The migration process
 has been already decided to implement through an export/import approach as
 a database migration approach won't be realistic as for older versions. So
 the simple generic approach is that each artifact specially APIs and
 Applications would be exported from 2.6.0 environment and then will be
 imported in to 3.0.0 environment.

 Below I have listed the possible two approaches for both these two
 steps (importing & exporting) followed by their pros and cons.

 Exporting :

 Approach 1 : Change currently existing Api Import/export web app and
 CLI tool in 2.6



-

Can use API Import/export Web app and CLI tool and modify
-

   (REST APIs for export api/applications is currently provided via
   this WEB app. CLI tool uses this web app)
   -

Required changes:
-

   For app export: Consumer key, Consumer Secret, Token exporting
   should be supported in CLI tool
   -

   Will have to change admin REST api implementation for above task
   and for exporting other artifacts mentioned at the end.
   -

   Modify CLI tool to support exporting in bulk.



-

Pros
-

   Will not have to develop/maintain a Web app and CLI tool from
   the scratch
   -

Cons
-

   API export flow will change as modifications will be required
   -

   Have to manage and consider also about normal export feature
   when doing changes for migration-exporting.
   -

   Migration story cannot be done in single step. Exporting and
   importing will be 2 step process.



 Approach 2 : Develop a separate tool without using API Import/export
 Web app or CLI tool.


-

Create a WEB App introducing new REST APIs required 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

Hi Nuwan,

One main reason is that, if we follow approach 1, migration story cannot be
done in single step. Specially two CLI tools will have to be used for the
two sub steps of exporting and importing.

And also normal API/Application export flow will change as major
modifications will be required for certain aspects. i.e.

   - bulk export/import
   - correlate the difference in the format of archives supported by 2.6 vs
   3.0
   - exporting and importing additional artifacts such as throttle
   policies, access tokens, user role permissions

Importing the 2.6 format API/Applications to 3.0 would require completely
new effort too. So anyway we will have to change the 3.0 import tool highly.

And my perception is that normal import export and migration oriented
export-import are tow separate tasks and thought it is better to isolate
the two objectives. (as similarly we do in earlier versions). We will also
have to think about specific permissions required for the migration
oriented export-import in same tool.

Eventhough we may go with Approach 2, we can reuse the same code bases when
possible.

But anyway it is a matter of decision, whether we manage the normal
import/export and migration-oriented-import/export separately. Approach 1
would result with some higher level of import/export while in approach 2,
we could highlight it as a migration.

Regards,
Samitha

On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias  wrote:

> I would rather go with approach 1. In approach 2 you are suggesting to
> develop a separate web app and a separate tool, why do you think it is
> better than enhancing the current App and CLI? Its obviously going to be a
> lot more work than approach 1, plus at the end of the day we will throw
> away one of those tools because we don't need two tools doing the same
> thing.
>
> On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga 
> wrote:
>
>>
>> Hi all,
>>>
>>> This is regarding the initial design on possible approaches to proceed
>>> with upgrading APIM 2.6.0(to be released) to 3.0.0. The migration process
>>> has been already decided to implement through an export/import approach as
>>> a database migration approach won't be realistic as for older versions. So
>>> the simple generic approach is that each artifact specially APIs and
>>> Applications would be exported from 2.6.0 environment and then will be
>>> imported in to 3.0.0 environment.
>>>
>>> Below I have listed the possible two approaches for both these two steps
>>> (importing & exporting) followed by their pros and cons.
>>>
>>> Exporting :
>>>
>>> Approach 1 : Change currently existing Api Import/export web app and CLI
>>> tool in 2.6
>>>
>>>
>>>
>>>-
>>>
>>>Can use API Import/export Web app and CLI tool and modify
>>>-
>>>
>>>   (REST APIs for export api/applications is currently provided via
>>>   this WEB app. CLI tool uses this web app)
>>>   -
>>>
>>>Required changes:
>>>-
>>>
>>>   For app export: Consumer key, Consumer Secret, Token exporting
>>>   should be supported in CLI tool
>>>   -
>>>
>>>   Will have to change admin REST api implementation for above task
>>>   and for exporting other artifacts mentioned at the end.
>>>   -
>>>
>>>   Modify CLI tool to support exporting in bulk.
>>>
>>>
>>>
>>>-
>>>
>>>Pros
>>>-
>>>
>>>   Will not have to develop/maintain a Web app and CLI tool from the
>>>   scratch
>>>   -
>>>
>>>Cons
>>>-
>>>
>>>   API export flow will change as modifications will be required
>>>   -
>>>
>>>   Have to manage and consider also about normal export feature when
>>>   doing changes for migration-exporting.
>>>   -
>>>
>>>   Migration story cannot be done in single step. Exporting and
>>>   importing will be 2 step process.
>>>
>>>
>>>
>>> Approach 2 : Develop a separate tool without using API Import/export Web
>>> app or CLI tool.
>>>
>>>
>>>-
>>>
>>>Create a WEB App introducing new REST APIs required and develop
>>>separate CLI tool.
>>>
>>>
>>>
>>>-
>>>
>>>Pros
>>>-
>>>
>>>   Migration story can be done in single step. (Exporting and
>>>   importing can be done in single command.)
>>>   -
>>>
>>>   API/Application import-export tool won’t have any effect or
>>>   change.
>>>   -
>>>
>>>   Don’t have to manage 2 separate use cases in single tool.
>>>   -
>>>
>>>Cons
>>>-
>>>
>>>   Will have to develop/maintain a Web app and/or CLI tool from the
>>>   scratch
>>>
>>>
>>>
>>> Importing :
>>>
>>>
>>>-
>>>
>>>Has to create separate REST APIs for importing 2.6-exported
>>>APIs/Applications/other artifacts as 3.0.0 expecting archives in 3.0
>>>specific format.
>>>
>>>
>>> Approach 1
>>>
>>>-
>>>
>>>Change/improve API Import/export CLI tool in 3.0 to support
>>>importing 2.6-exported API/application archives.
>>>
>>> Approach 2
>>>
>>>-
>>>
>>>Develop separate CLI tool from the 

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Nuwan Dias]

I would rather go with approach 1. In approach 2 you are suggesting to
develop a separate web app and a separate tool, why do you think it is
better than enhancing the current App and CLI? Its obviously going to be a
lot more work than approach 1, plus at the end of the day we will throw
away one of those tools because we don't need two tools doing the same
thing.

On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga 
wrote:

>
> Hi all,
>>
>> This is regarding the initial design on possible approaches to proceed
>> with upgrading APIM 2.6.0(to be released) to 3.0.0. The migration process
>> has been already decided to implement through an export/import approach as
>> a database migration approach won't be realistic as for older versions. So
>> the simple generic approach is that each artifact specially APIs and
>> Applications would be exported from 2.6.0 environment and then will be
>> imported in to 3.0.0 environment.
>>
>> Below I have listed the possible two approaches for both these two steps
>> (importing & exporting) followed by their pros and cons.
>>
>> Exporting :
>>
>> Approach 1 : Change currently existing Api Import/export web app and CLI
>> tool in 2.6
>>
>>
>>
>>-
>>
>>Can use API Import/export Web app and CLI tool and modify
>>-
>>
>>   (REST APIs for export api/applications is currently provided via
>>   this WEB app. CLI tool uses this web app)
>>   -
>>
>>Required changes:
>>-
>>
>>   For app export: Consumer key, Consumer Secret, Token exporting
>>   should be supported in CLI tool
>>   -
>>
>>   Will have to change admin REST api implementation for above task
>>   and for exporting other artifacts mentioned at the end.
>>   -
>>
>>   Modify CLI tool to support exporting in bulk.
>>
>>
>>
>>-
>>
>>Pros
>>-
>>
>>   Will not have to develop/maintain a Web app and CLI tool from the
>>   scratch
>>   -
>>
>>Cons
>>-
>>
>>   API export flow will change as modifications will be required
>>   -
>>
>>   Have to manage and consider also about normal export feature when
>>   doing changes for migration-exporting.
>>   -
>>
>>   Migration story cannot be done in single step. Exporting and
>>   importing will be 2 step process.
>>
>>
>>
>> Approach 2 : Develop a separate tool without using API Import/export Web
>> app or CLI tool.
>>
>>
>>-
>>
>>Create a WEB App introducing new REST APIs required and develop
>>separate CLI tool.
>>
>>
>>
>>-
>>
>>Pros
>>-
>>
>>   Migration story can be done in single step. (Exporting and
>>   importing can be done in single command.)
>>   -
>>
>>   API/Application import-export tool won’t have any effect or change.
>>   -
>>
>>   Don’t have to manage 2 separate use cases in single tool.
>>   -
>>
>>Cons
>>-
>>
>>   Will have to develop/maintain a Web app and/or CLI tool from the
>>   scratch
>>
>>
>>
>> Importing :
>>
>>
>>-
>>
>>Has to create separate REST APIs for importing 2.6-exported
>>APIs/Applications/other artifacts as 3.0.0 expecting archives in 3.0
>>specific format.
>>
>>
>> Approach 1
>>
>>-
>>
>>Change/improve API Import/export CLI tool in 3.0 to support importing
>>2.6-exported API/application archives.
>>
>> Approach 2
>>
>>-
>>
>>Develop separate CLI tool from the scratch which will be used for
>>exporting from 2.6 also
>>
>>
>>
>> Other artifacts required to be exported/imported (other than
>> APIs/Applications)
>>
>>
>>-
>>
>>Throttling Policies
>>-
>>
>>Custom API Lifecycles
>>-
>>
>>Workflows
>>-
>>
>>User Role Permissions
>>-
>>
>>Custom sequences (at least for reference -> has to decide regarding
>>this)
>>
>>
>>
>> My perception is that proceeding with Approach 2 under both exporting and
>> importing would be better considering the pros and cons of each.
>>
>> Highly appreciate your thoughts on this.
>>
>> Regards,
>>
>> Samitha
>>
>>
>> --
>> *Samitha Chathuranga*
>> *Senior Software Engineer*, *WSO2 Inc.*
>> lean.enterprise.middleware
>> Mobile: +94715123761
>>
>> [image: http://wso2.com/signature] 
>>
>
>
>

-- 
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Migration approach for APIM 2.6.0 to 3.0.0

[Samitha Chathuranga]

> Hi all,
>
> This is regarding the initial design on possible approaches to proceed
> with upgrading APIM 2.6.0(to be released) to 3.0.0. The migration process
> has been already decided to implement through an export/import approach as
> a database migration approach won't be realistic as for older versions. So
> the simple generic approach is that each artifact specially APIs and
> Applications would be exported from 2.6.0 environment and then will be
> imported in to 3.0.0 environment.
>
> Below I have listed the possible two approaches for both these two steps
> (importing & exporting) followed by their pros and cons.
>
> Exporting :
>
> Approach 1 : Change currently existing Api Import/export web app and CLI
> tool in 2.6
>
>
>
>-
>
>Can use API Import/export Web app and CLI tool and modify
>-
>
>   (REST APIs for export api/applications is currently provided via
>   this WEB app. CLI tool uses this web app)
>   -
>
>Required changes:
>-
>
>   For app export: Consumer key, Consumer Secret, Token exporting
>   should be supported in CLI tool
>   -
>
>   Will have to change admin REST api implementation for above task
>   and for exporting other artifacts mentioned at the end.
>   -
>
>   Modify CLI tool to support exporting in bulk.
>
>
>
>-
>
>Pros
>-
>
>   Will not have to develop/maintain a Web app and CLI tool from the
>   scratch
>   -
>
>Cons
>-
>
>   API export flow will change as modifications will be required
>   -
>
>   Have to manage and consider also about normal export feature when
>   doing changes for migration-exporting.
>   -
>
>   Migration story cannot be done in single step. Exporting and
>   importing will be 2 step process.
>
>
>
> Approach 2 : Develop a separate tool without using API Import/export Web
> app or CLI tool.
>
>
>-
>
>Create a WEB App introducing new REST APIs required and develop
>separate CLI tool.
>
>
>
>-
>
>Pros
>-
>
>   Migration story can be done in single step. (Exporting and
>   importing can be done in single command.)
>   -
>
>   API/Application import-export tool won’t have any effect or change.
>   -
>
>   Don’t have to manage 2 separate use cases in single tool.
>   -
>
>Cons
>-
>
>   Will have to develop/maintain a Web app and/or CLI tool from the
>   scratch
>
>
>
> Importing :
>
>
>-
>
>Has to create separate REST APIs for importing 2.6-exported
>APIs/Applications/other artifacts as 3.0.0 expecting archives in 3.0
>specific format.
>
>
> Approach 1
>
>-
>
>Change/improve API Import/export CLI tool in 3.0 to support importing
>2.6-exported API/application archives.
>
> Approach 2
>
>-
>
>Develop separate CLI tool from the scratch which will be used for
>exporting from 2.6 also
>
>
>
> Other artifacts required to be exported/imported (other than
> APIs/Applications)
>
>
>-
>
>Throttling Policies
>-
>
>Custom API Lifecycles
>-
>
>Workflows
>-
>
>User Role Permissions
>-
>
>Custom sequences (at least for reference -> has to decide regarding
>this)
>
>
>
> My perception is that proceeding with Approach 2 under both exporting and
> importing would be better considering the pros and cons of each.
>
> Highly appreciate your thoughts on this.
>
> Regards,
>
> Samitha
>
>
> --
> *Samitha Chathuranga*
> *Senior Software Engineer*, *WSO2 Inc.*
> lean.enterprise.middleware
> Mobile: +94715123761
>
> [image: http://wso2.com/signature] 
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture