On 2 September 2018 at 15:07, Harsha Kumara <[email protected]> wrote:
> According to my understanding we going to do the migration on demand. > Which means particular resource will be migrated when read is made for that > particular resource. Let's say we already migrated a particular API, I > think still we will need to do a check on version column to decide whether > we need a migration or not right? Which might add a overhead. > With on the fly migrations the validation to see if a migration should be performed or not when data is retrieved cannot be avoided. But the overhead is negligible since we are only retrieving an additional column for a specific entity(example: API x). we just need an additional if condition to compare the string of this column with the current product version string. > API Manager generally contains artifacts which are reside in the file > system. In this case we will require to migrate all the file system > artifacts we can't perform on the fly upgrades to these APIs. Also > operations like key validation might access database directly to retrieve > information. In this case we will need to ensure all components are going > through the standard interfaces to retrieve data including our dependent > components. > We are talking about APIM 3 specifically here, AFAIK we dont have anything in the file system. We are using the DB as a single source of truth to simplify managing everything. In general having a uniform method of accessing data by all components is desirable in order to avoid bugs. The DOA layer is that uniform method and the migration will be performed at the DAO layer. > What will be the impact for the extensions that allow users to extend the > produce features. Having partially migrated data, specially during the list > operation might cause issues for them. > Not clear exactly on the impact of extensions that you have mentioned. Yes partially migrated data can be a concern if they start using the product before migration has been completed properly. I mentioned a strategy for avoiding this earlier by triggering a GET on each entity in order to force the on the fly migration upfront. > Let's say customer is upgrading from 3.0.0 to 3.5.0 without using the > intermediate versions, do we require to have product version specific > migration methods to handle this scenario or will we write single method to > gradually migrate from 3.0.0 to 3.5.0. Sometimes migrating version of API > will be associate with modify the data of several tables which can share > data across several APIs, will it be good idea to have partially migrated > data? > >From a developer point of view it makes sense to code the migration for each version separately since this is easier to manage. The migration implementation can decide which migrations need to be run based on the version of the data. See the sample code I have provided in the beginning to see what this might look like. Agreed that partially migrated data can be inconsistent, hence I have proposed triggering the on the fly migration of everything upfront. > Instead of having a partial migration which will happen on demand, how > about having full migration during a startup of the server. If we going to > follow this approach, I believe we need to adapt this everywhere. Otherwise > our dependent component will have their own way of migrating artifacts > while we are using on the fly migrations. > Agreed. Best approach is triggering the migration upfront as stated earlier. If we can do this for APIM first we can propose that all other components as well > On Wed, Aug 22, 2018 at 1:17 PM Uvindra Dias Jayasinha <[email protected]> > wrote: > >> See my responses inline >> >> On 22 August 2018 at 11:54, Nuwan Dias <[email protected]> wrote: >> >>> I have a few questions on the design. >>> >>> 1. Who will be updating the Audit table and at which point? >>> >> >> This can be done whenever our DAOFactory is called to create an instance >> of a given DAO object when a DOA operation is performed for the first >> time. We already have a setup() method that adds per-requiste data into >> the DB required for the product to function(Resource Categories, Default >> Labels, API types)[1]. So inserting into Audit table can be performed at >> this point as well. >> >> >> [1] https://github.com/wso2/carbon-apimgt/blob/master/ >> components/apimgt/org.wso2.carbon.apimgt.core/src/main/ >> java/org/wso2/carbon/apimgt/core/dao/impl/DAOFactory.java#L379 >> >> >>> 2. Who will be calling the method in the interface and at which point? I >>> understood it as it will be called upon the GET operation of each entity. >>> If that's the case it'll cause problems specially with listing operations. >>> >> >> The method will be called by the DAO layer when it is retrieving >> data(example: getAPI(String apiID) ). In the listing operation you have >> mentioned we call a separate method to get the summarized data of all APIs. >> So the idea here is that we do this change only when we are getting a >> specific entity instead of a list. The advantage of using a GET is that the >> caller does not need to change the entity themselves. As far as they are >> concerned they are doing a read operation. >> >> NuwanD mentioned that we have been following the method of doing on the >> fly migrations when a PUT is done on an entity. The disadvantage of this is >> that we need to maintain code that knows to render data that hasn't been >> migrated when a GET is performed on this. So 5 releases down the line we >> are going to have to have separate code that knows how to render data from >> all the previous releases. What I'm proposing is instead of having multiple >> rendering logic for the GET, we can decouple this as multiple migration >> logic which will reside in a separate class. So we dont need to have >> migration code in PUT and rendering code in GET, only migration code >> invoked on GET. This is cleaner and involves less code. >> >> One disadvantage in the method I have proposed is, what if one of the >> data displayed during listing needs to be migrated? In that case since the >> retrieval of the specific data entity hasn't been called migration is yet >> to take place. So the listing non migrated data, which is wrong. We can >> overcome this limitation by simply getting the IDs of all entities and >> doing a GET for each specific entity via the REST API upfront. We can ship >> a script that does this and customer only needs to run this script the >> first time before using the product and everything will get migrated. >> >> 3. Do we need an interface for each entity type? If yes, that doesn't >> sound right because then we'll end up with just 1 implementation of each >> interface, which is not very useful. The number of implementations to >> maintain is also a concern due to the high number of entity types we'll end >> up with. >> >> The reason for using an interface in this case is not for polymorphic >> reasons. This is simply to decouple the migration logic from our standard >> DAO code. But in this case I concede that we don't have to have an >> interface, we can simply have a separate class that takes care of this. >> Since DOA layer is not unit tested we don't need to have migration code as >> an interface implementation in order to inject it via constructor to >> improve testability. >> >> We could have a single class that includes all the migration logic for >> each entity to make this more manageable. >> >> Thanks, >> NuwanD. >> >>> >>> On Mon, Aug 20, 2018 at 5:09 PM Uvindra Dias Jayasinha <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On 20 August 2018 at 16:58, Ishara Cooray <[email protected]> wrote: >>>> >>>>> For me, PRODUCT_VERSION column in every table seems to be redundant. >>>>> Could you please explain the reason for introducing this column in >>>>> each table? Is this for auditing? >>>>> >>>> >>>> The reason for this is so the on the fly migration code is able to >>>> detect if it needs to migrate a given row. For example if running version >>>> 3.1.0, >>>> >>>> 1. A row from a table is retrieved >>>> 2. If the value of the PRODUCT_VERSION column of the row is 3.0.0 >>>> migration code will run to convert the data and update the value of >>>> PRODUCT_VERSION to 3.1.0 once row migration has occurred. >>>> 3. On subsequent retrievals of the same row since PRODUCT_VERSION is >>>> 3.1.0 migration code will not execute against the row. >>>> >>>> >>>>> Thanks & Regards, >>>>> Ishara Cooray >>>>> Senior Software Engineer >>>>> Mobile : +9477 262 9512 >>>>> WSO2, Inc. | http://wso2.com/ >>>>> Lean . Enterprise . Middleware >>>>> >>>>> On Mon, Aug 20, 2018 at 4:03 PM, Uvindra Dias Jayasinha < >>>>> [email protected]> wrote: >>>>> >>>>>> Small calcification regarding this statement, >>>>>> >>>>>> For the 3.0.0 release we just need to implement steps *1* and *2* >>>>>>> above. Step *3* can be done for all subsequent releases. >>>>>>> >>>>>> >>>>>> I specifically meant the changes to the DB schema when it comes to >>>>>> steps 1 and 2 . Obviously no migration logic will be needed for 3.0.0 >>>>>> itself >>>>>> >>>>>> On 20 August 2018 at 15:58, Uvindra Dias Jayasinha <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> In the past the APIM product has relied on an external component >>>>>>> such as a migration client for upgrading from a given product version >>>>>>> to a >>>>>>> higher version.The end user was required to configure the latest product >>>>>>> that they are upgrading to against their current data(databases, synapse >>>>>>> files, registry) and run the migration client manually to upgrade the >>>>>>> product. This can be a cumbersome and error prone process to accomplish >>>>>>> for >>>>>>> end users, making product version upgrades time consuming. >>>>>>> >>>>>>> To overcome the above problem on the fly upgrades were proposed >>>>>>> where the product code detects if relevant data being accessed needs to >>>>>>> be >>>>>>> migrated to the latest version and migrates the data when the code is >>>>>>> executed when the respective feature is used. Upgrading product data is >>>>>>> much easier from 3.0.0 onwards because all data related to the product >>>>>>> is >>>>>>> stored in a central database.This means that the end user does not need >>>>>>> to >>>>>>> get involved in upgrading, it happens without them even being aware as >>>>>>> they >>>>>>> use the latest version of the product by pointing it against their >>>>>>> current >>>>>>> database.This makes for a more pleasant user experience when upgrading, >>>>>>> putting the burden of the upgrade to be added by the developer into the >>>>>>> functional code itself. >>>>>>> >>>>>>> The following outlines a design that can be supported from 3.0.0 >>>>>>> onwards to outline a uniform way of handling product upgrades. This is >>>>>>> inspired by the methodology used by FlywayDB to enable DB migrations[1] >>>>>>> but also taking into account the requirement of being able to run on the >>>>>>> fly at runtime(Note: DB schema changes between releases will need to be >>>>>>> handled via DB vendor specific scripts prepared by the team to be run by >>>>>>> the customer against their DB). >>>>>>> >>>>>>> >>>>>>> *1.* A new table will be added to the schema called >>>>>>> PRODUCT_VERSION_AUDIT to track the product version upgrades that take >>>>>>> place >>>>>>> on a given dataset >>>>>>> >>>>>>> PRODUCT_VERSION_AUDIT >>>>>>> VERSION VARCHAR(5) >>>>>>> CREATED_TIME TIMESTAMP(6) >>>>>>> >>>>>>> If a user begins using APIM version 3.0.0 and then upgrades to >>>>>>> version 3.1.0 the table will contain the following values, >>>>>>> >>>>>>> VERSION CREATED_TIME >>>>>>> 3.0.0 2018-11-11 13:23:44 >>>>>>> 3.1.0 2019-10-14 9:26:22 >>>>>>> >>>>>>> This gives a historical view of the product versions a customer has >>>>>>> been using. A new row will be inserted into the table when a given >>>>>>> product >>>>>>> version is started for the first time. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *2*. Each table in the database will have a new column called >>>>>>> PRODUCT_VERSION(VARCHAR(5)) added. When a row is inserted for the first >>>>>>> time it will populate this column with the current product version being >>>>>>> used. >>>>>>> For example the AM_API table could have the following entries for a >>>>>>> customer using APIM 3.0.0, >>>>>>> >>>>>>> UUID PROVIDER NAME VERSION CONTEXT PRODUCT_VERSION >>>>>>> 123e4567-e89b-12d3-a456-426655440000 admin abc 1.0.0 /abc 3.0.0 >>>>>>> 00112233-4455-6677-8899-aabbccddeeff admin xyz 1.0.0 /xyz 3.0.0 >>>>>>> >>>>>>> >>>>>>> Lets assume when upgrading to 3.1.0 the leading '/' character in the >>>>>>> context needs to be removed. On the fly migration code will run when a >>>>>>> given row is accessed by the DAO layer to remove the '/'. Once the >>>>>>> migration of the row is completed the PRODUCT_VERSION column will be >>>>>>> updated with the value 3.1.0 to signify that the migration for this row >>>>>>> has >>>>>>> been completed. The PRODUCT_VERSION column can be validated to check if >>>>>>> migration code needs to be executed. So assuming the API abc is accessed >>>>>>> first the table will look as follows after migration, >>>>>>> >>>>>>> >>>>>>> UUID PROVIDER NAME VERSION CONTEXT PRODUCT_VERSION >>>>>>> 123e4567-e89b-12d3-a456-426655440000 admin abc 1.0.0 abc 3.1.0 >>>>>>> 00112233-4455-6677-8899-aabbccddeeff admin xyz 1.0.0 /xyz 3.0.0 >>>>>>> >>>>>>> >>>>>>> As a pre-requisite the product team will need to create respective >>>>>>> DB scripts for the schema changes that will take place with a given >>>>>>> release. This will only include schema modifications. Customer will >>>>>>> need to >>>>>>> run these manually against their DB but actual data migration will take >>>>>>> place automatically under the hood. >>>>>>> >>>>>>> >>>>>>> *3*. New Java interfaces will be added for each DB entity that will >>>>>>> responsible for migrating the respective entity. For example for APIs we >>>>>>> can have, >>>>>>> >>>>>>> public interface APIMigrator { >>>>>>> API migrate(PreparedStatement statement) throws MigrationException;} >>>>>>> >>>>>>> >>>>>>> >>>>>>> This will accept the PreparedStatement created for data retrieval >>>>>>> and returns the migrated API object. The implemenataion of the above >>>>>>> could >>>>>>> look as follows, >>>>>>> >>>>>>> public class APIMigratorImpl implements APIMigrator { >>>>>>> >>>>>>> public API migrate(PreparedStatement statement) throws >>>>>>> MigrationException { >>>>>>> API api = null; >>>>>>> try (ResultSet rs = statement.executeQuery()) { >>>>>>> while (rs.next()) { >>>>>>> String dataProductVersion = >>>>>>> rs.getString("PRODUCT_VERSION"); >>>>>>> >>>>>>> // Assume that currentProductVersion == "3.2.0" >>>>>>> >>>>>>> while >>>>>>> (!currentProductVersion.equals(dataProductVersion)) } >>>>>>> if ("3.0.0".equals(dataProductVersion)) { >>>>>>> // Logic to migrate data to next available >>>>>>> version 3.1.0 >>>>>>> // And update the PRODUCT_VERSION column of the >>>>>>> row to 3.1.0 >>>>>>> >>>>>>> dataProductVersion = "3.1.0"; >>>>>>> >>>>>>> } >>>>>>> >>>>>>> if ("3.1.0".equals(dataProductVersion)) { >>>>>>> // Logic to migrate data to next available >>>>>>> version 3.2.0 >>>>>>> // And update the PRODUCT_VERSION column of the >>>>>>> row to 3.2.0 >>>>>>> >>>>>>> dataProductVersion = "3.2.0"; >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> return api; >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> The above interface implementation will be called from within DOA >>>>>>> layer before returning an object instance. >>>>>>> >>>>>>> For the 3.0.0 release we just need to implement steps *1* and *2* >>>>>>> above. Step *3* can be done for all subsequent releases. >>>>>>> >>>>>>> >>>>>>> >>>>>>> [1] https://flywaydb.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Uvindra >>>>>>> >>>>>>> Mobile: 777733962 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Uvindra >>>>>> >>>>>> Mobile: 777733962 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Uvindra >>>> >>>> Mobile: 777733962 >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Director - WSO2, Inc. http://wso2.com >>> email : [email protected] >>> Phone : +94 777 775 729 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Regards, >> Uvindra >> >> Mobile: 777733962 >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > > > -- > Harsha Kumara > Associate Technical Lead, WSO2 Inc. > Mobile: +94775505618 > Blog:harshcreationz.blogspot.com > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Regards, Uvindra Mobile: 777733962
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
