Re: [Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
Hi,
Below is a sample message the tool will provide when it detects that
there's no change from the server.
*First time running the setup command:*
$ micro-gw setup pizzaproj -a PizzaShackAPI -v 1.0.0 -f
Enter Password for admin:
Setting up project pizzaproj successful.
*Second time:*
$ micro-gw setup pizzaproj -a PizzaShackAPI -v 1.0.0 -f
Enter Password for admin:
*No changes received from the server since the previous setup. If you
already have a built distribution, it can be reused.*
Setting up project pizzaproj successful.
The output message looks a bit long for me. Any suggestions for a better
and a short output message? Or is this message is fine?
Thanks!
On Tue, Jun 19, 2018 at 2:56 PM, Nuwan Dias wrote:
> When we do WUM updates to the distribution (Microgateway Toolkit), we may
> have to force build the runtime even if there aren't changes to the API and
> Policy definitions. In that case we may need a flag to force build.
>
> On Tue, Jun 19, 2018 at 2:41 PM, Isuru Haththotuwa
> wrote:
>
>> +1 for this approach.
>>
>> On Tue, Jun 19, 2018 at 1:47 PM, Malintha Amarasinghe > > wrote:
>>
>>> + IsuruH
>>>
>>> On Tue, Jun 19, 2018 at 12:41 PM, Malintha Amarasinghe <
>>> malint...@wso2.com> wrote:
>>>
List of fields planned to be added as of now; kindly let me know if any
field is missing.
*API*
name
context
version
apiDefinition
responseCaching
isDefaultVersion
type - (http vs ws)
transport - (http/https)
endpointConfig
endpointSecurity
corsConfiguration
authorizationHeader
*SubscriptionThrottlePolicy*
policyName
defaultLimit (throttling limits)
stopOnQuotaReach
*ApplicationThrottlePolicy*
policyName
defaultLimit (throttling limits)
Thanks!
Malintha
On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara
wrote:
>
>
> On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe <
> malint...@wso2.com> wrote:
>
>> Hi,
>>
>> Micro gateway CLI works completely separately to API manager;
>> whenever a new API is added for a label, whenever there is a change
>> happens
>> to an existing label there won't be any events published etc like
>> previously. The CLI needs to regenerate the source build it and push the
>> artifacts to the deployment and the full process needs to complete. In
>> most
>> occasions, the CLI can be configured to run periodically to generate
>> sources and do above job.
>>
>> But in this case, most of the time, the CLI will be uselessly
>> generating sources building it and pushing the artifacts to deployment.
>> Comparatively, building and pushing artifacts to deployment have a huge
>> overhead compared to generating sources.
>>
>> This effort is to avoid that as much as possible by change-detection;
>> i.e.
>>
>> 1. The CLI will check if any of the required resources has changed vs
>> the previous build and notify the user after a successful "setup" (source
>> generate) command using the command line output and the exit code of the
>> command.
>> 2. Using the exit code, a user can write a shell script etc to decide
>> whether he should proceed with "build" or not.
>>
>>
>> *Proposed implementation:*
>>
>> API Publisher APIs does not have ETag feature. Even if it is there,
>> the ETag will be generated for the whole resource. For code generation,
>> we
>> will be only using few attributes of the resource, hence using a global
>> ETag for a resource may lead to unnecessary changes for the ETag. Hence
>> the
>> proposed implementation will be using a CLI-side hash generation for
>> *used
>> attributes *of the resource (API/Policies) only.
>>
>> To mark the attributes which are used for generating the code, a
>> newly introduced annotation "@Hash" can be used.
>>
>> Ex:
>>
>> public class APIDetailedDTO extends APIInfoDTO {
>>
>> /**
>> * Swagger definition of the APIDetailedDTO which contains details
>> about URI templates and scopes\n
>> **/
>> *@Hash*
>> @JsonProperty("apiDefinition")
>> public String getApiDefinition() {
>> return apiDefinition;
>> }
>>
>> public void setApiDefinition(String apiDefinition) {
>> this.apiDefinition = apiDefinition;
>> }
>>
>>
>> /**
>> * WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
>> **/
>> @JsonProperty("wsdlUri")
>> public String getWsdlUri() {
>> return wsdlUri;
>> }
>>
>> public void setWsdlUri(String wsdlUri) {
>> this.wsdlUri = wsdlUri;
>> }
>>
>> *@Hash*
>> @JsonProperty("responseCaching")
>> public String
Re: [Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
When we do WUM updates to the distribution (Microgateway Toolkit), we may
have to force build the runtime even if there aren't changes to the API and
Policy definitions. In that case we may need a flag to force build.
On Tue, Jun 19, 2018 at 2:41 PM, Isuru Haththotuwa wrote:
> +1 for this approach.
>
> On Tue, Jun 19, 2018 at 1:47 PM, Malintha Amarasinghe
> wrote:
>
>> + IsuruH
>>
>> On Tue, Jun 19, 2018 at 12:41 PM, Malintha Amarasinghe <
>> malint...@wso2.com> wrote:
>>
>>> List of fields planned to be added as of now; kindly let me know if any
>>> field is missing.
>>>
>>> *API*
>>> name
>>> context
>>> version
>>> apiDefinition
>>> responseCaching
>>> isDefaultVersion
>>> type - (http vs ws)
>>> transport - (http/https)
>>> endpointConfig
>>> endpointSecurity
>>> corsConfiguration
>>> authorizationHeader
>>>
>>>
>>> *SubscriptionThrottlePolicy*
>>> policyName
>>> defaultLimit (throttling limits)
>>> stopOnQuotaReach
>>>
>>> *ApplicationThrottlePolicy*
>>> policyName
>>> defaultLimit (throttling limits)
>>>
>>>
>>> Thanks!
>>> Malintha
>>>
>>> On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara
>>> wrote:
>>>
On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe <
malint...@wso2.com> wrote:
> Hi,
>
> Micro gateway CLI works completely separately to API manager; whenever
> a new API is added for a label, whenever there is a change happens to an
> existing label there won't be any events published etc like previously.
> The
> CLI needs to regenerate the source build it and push the artifacts to the
> deployment and the full process needs to complete. In most occasions, the
> CLI can be configured to run periodically to generate sources and do above
> job.
>
> But in this case, most of the time, the CLI will be uselessly
> generating sources building it and pushing the artifacts to deployment.
> Comparatively, building and pushing artifacts to deployment have a huge
> overhead compared to generating sources.
>
> This effort is to avoid that as much as possible by change-detection;
> i.e.
>
> 1. The CLI will check if any of the required resources has changed vs
> the previous build and notify the user after a successful "setup" (source
> generate) command using the command line output and the exit code of the
> command.
> 2. Using the exit code, a user can write a shell script etc to decide
> whether he should proceed with "build" or not.
>
>
> *Proposed implementation:*
>
> API Publisher APIs does not have ETag feature. Even if it is there,
> the ETag will be generated for the whole resource. For code generation, we
> will be only using few attributes of the resource, hence using a global
> ETag for a resource may lead to unnecessary changes for the ETag. Hence
> the
> proposed implementation will be using a CLI-side hash generation for *used
> attributes *of the resource (API/Policies) only.
>
> To mark the attributes which are used for generating the code, a newly
> introduced annotation "@Hash" can be used.
>
> Ex:
>
> public class APIDetailedDTO extends APIInfoDTO {
>
> /**
> * Swagger definition of the APIDetailedDTO which contains details
> about URI templates and scopes\n
> **/
> *@Hash*
> @JsonProperty("apiDefinition")
> public String getApiDefinition() {
> return apiDefinition;
> }
>
> public void setApiDefinition(String apiDefinition) {
> this.apiDefinition = apiDefinition;
> }
>
>
> /**
> * WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
> **/
> @JsonProperty("wsdlUri")
> public String getWsdlUri() {
> return wsdlUri;
> }
>
> public void setWsdlUri(String wsdlUri) {
> this.wsdlUri = wsdlUri;
> }
>
> *@Hash*
> @JsonProperty("responseCaching")
> public String getResponseCaching() {
> return responseCaching;
> }
>
>
>
> The methods marked with *@Hash* will be automatically extracted from
> the code and will be used to generate the hashes for each resource.
>
> The generated hashes will be stored inside the CLI's temp folder
> against each resources' UUID, which will be used to compare the hash
> changes between next runs.
>
What are the fields which we have added to the hash?
>
>
> Highly appreciate your ideas on this.
>
> Thanks!
> Malintha
>
>
> --
> Malintha Amarasinghe
> *WSO2, Inc. - lean | enterprise | middleware*
> http://wso2.com/
>
> Mobile : +94 712383306
>
--
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com
>>>
>>>
>>>
Re: [Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
+1 for this approach.
On Tue, Jun 19, 2018 at 1:47 PM, Malintha Amarasinghe
wrote:
> + IsuruH
>
> On Tue, Jun 19, 2018 at 12:41 PM, Malintha Amarasinghe > wrote:
>
>> List of fields planned to be added as of now; kindly let me know if any
>> field is missing.
>>
>> *API*
>> name
>> context
>> version
>> apiDefinition
>> responseCaching
>> isDefaultVersion
>> type - (http vs ws)
>> transport - (http/https)
>> endpointConfig
>> endpointSecurity
>> corsConfiguration
>> authorizationHeader
>>
>>
>> *SubscriptionThrottlePolicy*
>> policyName
>> defaultLimit (throttling limits)
>> stopOnQuotaReach
>>
>> *ApplicationThrottlePolicy*
>> policyName
>> defaultLimit (throttling limits)
>>
>>
>> Thanks!
>> Malintha
>>
>> On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara wrote:
>>
>>>
>>>
>>> On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe <
>>> malint...@wso2.com> wrote:
>>>
Hi,
Micro gateway CLI works completely separately to API manager; whenever
a new API is added for a label, whenever there is a change happens to an
existing label there won't be any events published etc like previously. The
CLI needs to regenerate the source build it and push the artifacts to the
deployment and the full process needs to complete. In most occasions, the
CLI can be configured to run periodically to generate sources and do above
job.
But in this case, most of the time, the CLI will be uselessly
generating sources building it and pushing the artifacts to deployment.
Comparatively, building and pushing artifacts to deployment have a huge
overhead compared to generating sources.
This effort is to avoid that as much as possible by change-detection;
i.e.
1. The CLI will check if any of the required resources has changed vs
the previous build and notify the user after a successful "setup" (source
generate) command using the command line output and the exit code of the
command.
2. Using the exit code, a user can write a shell script etc to decide
whether he should proceed with "build" or not.
*Proposed implementation:*
API Publisher APIs does not have ETag feature. Even if it is there, the
ETag will be generated for the whole resource. For code generation, we will
be only using few attributes of the resource, hence using a global ETag for
a resource may lead to unnecessary changes for the ETag. Hence the proposed
implementation will be using a CLI-side hash generation for *used
attributes *of the resource (API/Policies) only.
To mark the attributes which are used for generating the code, a newly
introduced annotation "@Hash" can be used.
Ex:
public class APIDetailedDTO extends APIInfoDTO {
/**
* Swagger definition of the APIDetailedDTO which contains details
about URI templates and scopes\n
**/
*@Hash*
@JsonProperty("apiDefinition")
public String getApiDefinition() {
return apiDefinition;
}
public void setApiDefinition(String apiDefinition) {
this.apiDefinition = apiDefinition;
}
/**
* WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
**/
@JsonProperty("wsdlUri")
public String getWsdlUri() {
return wsdlUri;
}
public void setWsdlUri(String wsdlUri) {
this.wsdlUri = wsdlUri;
}
*@Hash*
@JsonProperty("responseCaching")
public String getResponseCaching() {
return responseCaching;
}
The methods marked with *@Hash* will be automatically extracted from
the code and will be used to generate the hashes for each resource.
The generated hashes will be stored inside the CLI's temp folder
against each resources' UUID, which will be used to compare the hash
changes between next runs.
>>> What are the fields which we have added to the hash?
>>>
Highly appreciate your ideas on this.
Thanks!
Malintha
--
Malintha Amarasinghe
*WSO2, Inc. - lean | enterprise | middleware*
http://wso2.com/
Mobile : +94 712383306
>>>
>>>
>>> --
>>> Harsha Kumara
>>> Associate Technical Lead, WSO2 Inc.
>>> Mobile: +94775505618
>>> Blog:harshcreationz.blogspot.com
>>>
>>
>>
>>
>> --
>> Malintha Amarasinghe
>> *WSO2, Inc. - lean | enterprise | middleware*
>> http://wso2.com/
>>
>> Mobile : +94 712383306
>>
>
>
>
> --
> Malintha Amarasinghe
> *WSO2, Inc. - lean | enterprise | middleware*
> http://wso2.com/
>
> Mobile : +94 712383306
>
--
Thanks and Regards,
Isuru H.
+94 716 358 048*
Re: [Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
List of fields planned to be added as of now; kindly let me know if any
field is missing.
*API*
name
context
version
apiDefinition
responseCaching
isDefaultVersion
type - (http vs ws)
transport - (http/https)
endpointConfig
endpointSecurity
corsConfiguration
authorizationHeader
*SubscriptionThrottlePolicy*
policyName
defaultLimit (throttling limits)
stopOnQuotaReach
*ApplicationThrottlePolicy*
policyName
defaultLimit (throttling limits)
Thanks!
Malintha
On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara wrote:
>
>
> On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe
> wrote:
>
>> Hi,
>>
>> Micro gateway CLI works completely separately to API manager; whenever a
>> new API is added for a label, whenever there is a change happens to an
>> existing label there won't be any events published etc like previously. The
>> CLI needs to regenerate the source build it and push the artifacts to the
>> deployment and the full process needs to complete. In most occasions, the
>> CLI can be configured to run periodically to generate sources and do above
>> job.
>>
>> But in this case, most of the time, the CLI will be uselessly generating
>> sources building it and pushing the artifacts to deployment. Comparatively,
>> building and pushing artifacts to deployment have a huge overhead compared
>> to generating sources.
>>
>> This effort is to avoid that as much as possible by change-detection;
>> i.e.
>>
>> 1. The CLI will check if any of the required resources has changed vs the
>> previous build and notify the user after a successful "setup" (source
>> generate) command using the command line output and the exit code of the
>> command.
>> 2. Using the exit code, a user can write a shell script etc to decide
>> whether he should proceed with "build" or not.
>>
>>
>> *Proposed implementation:*
>>
>> API Publisher APIs does not have ETag feature. Even if it is there, the
>> ETag will be generated for the whole resource. For code generation, we will
>> be only using few attributes of the resource, hence using a global ETag for
>> a resource may lead to unnecessary changes for the ETag. Hence the proposed
>> implementation will be using a CLI-side hash generation for *used
>> attributes *of the resource (API/Policies) only.
>>
>> To mark the attributes which are used for generating the code, a newly
>> introduced annotation "@Hash" can be used.
>>
>> Ex:
>>
>> public class APIDetailedDTO extends APIInfoDTO {
>>
>> /**
>> * Swagger definition of the APIDetailedDTO which contains details about
>> URI templates and scopes\n
>> **/
>> *@Hash*
>> @JsonProperty("apiDefinition")
>> public String getApiDefinition() {
>> return apiDefinition;
>> }
>>
>> public void setApiDefinition(String apiDefinition) {
>> this.apiDefinition = apiDefinition;
>> }
>>
>>
>> /**
>> * WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
>> **/
>> @JsonProperty("wsdlUri")
>> public String getWsdlUri() {
>> return wsdlUri;
>> }
>>
>> public void setWsdlUri(String wsdlUri) {
>> this.wsdlUri = wsdlUri;
>> }
>>
>> *@Hash*
>> @JsonProperty("responseCaching")
>> public String getResponseCaching() {
>> return responseCaching;
>> }
>>
>>
>>
>> The methods marked with *@Hash* will be automatically extracted from the
>> code and will be used to generate the hashes for each resource.
>>
>> The generated hashes will be stored inside the CLI's temp folder against
>> each resources' UUID, which will be used to compare the hash changes
>> between next runs.
>>
> What are the fields which we have added to the hash?
>
>>
>>
>> Highly appreciate your ideas on this.
>>
>> Thanks!
>> Malintha
>>
>>
>> --
>> Malintha Amarasinghe
>> *WSO2, Inc. - lean | enterprise | middleware*
>> http://wso2.com/
>>
>> Mobile : +94 712383306
>>
>
>
> --
> Harsha Kumara
> Associate Technical Lead, WSO2 Inc.
> Mobile: +94775505618
> Blog:harshcreationz.blogspot.com
>
--
Malintha Amarasinghe
*WSO2, Inc. - lean | enterprise | middleware*
http://wso2.com/
Mobile : +94 712383306
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe
wrote:
> Hi,
>
> Micro gateway CLI works completely separately to API manager; whenever a
> new API is added for a label, whenever there is a change happens to an
> existing label there won't be any events published etc like previously. The
> CLI needs to regenerate the source build it and push the artifacts to the
> deployment and the full process needs to complete. In most occasions, the
> CLI can be configured to run periodically to generate sources and do above
> job.
>
> But in this case, most of the time, the CLI will be uselessly generating
> sources building it and pushing the artifacts to deployment. Comparatively,
> building and pushing artifacts to deployment have a huge overhead compared
> to generating sources.
>
> This effort is to avoid that as much as possible by change-detection; i.e.
>
> 1. The CLI will check if any of the required resources has changed vs the
> previous build and notify the user after a successful "setup" (source
> generate) command using the command line output and the exit code of the
> command.
> 2. Using the exit code, a user can write a shell script etc to decide
> whether he should proceed with "build" or not.
>
>
> *Proposed implementation:*
>
> API Publisher APIs does not have ETag feature. Even if it is there, the
> ETag will be generated for the whole resource. For code generation, we will
> be only using few attributes of the resource, hence using a global ETag for
> a resource may lead to unnecessary changes for the ETag. Hence the proposed
> implementation will be using a CLI-side hash generation for *used
> attributes *of the resource (API/Policies) only.
>
> To mark the attributes which are used for generating the code, a newly
> introduced annotation "@Hash" can be used.
>
> Ex:
>
> public class APIDetailedDTO extends APIInfoDTO {
>
> /**
> * Swagger definition of the APIDetailedDTO which contains details about
> URI templates and scopes\n
> **/
> *@Hash*
> @JsonProperty("apiDefinition")
> public String getApiDefinition() {
> return apiDefinition;
> }
>
> public void setApiDefinition(String apiDefinition) {
> this.apiDefinition = apiDefinition;
> }
>
>
> /**
> * WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
> **/
> @JsonProperty("wsdlUri")
> public String getWsdlUri() {
> return wsdlUri;
> }
>
> public void setWsdlUri(String wsdlUri) {
> this.wsdlUri = wsdlUri;
> }
>
> *@Hash*
> @JsonProperty("responseCaching")
> public String getResponseCaching() {
> return responseCaching;
> }
>
>
>
> The methods marked with *@Hash* will be automatically extracted from the
> code and will be used to generate the hashes for each resource.
>
> The generated hashes will be stored inside the CLI's temp folder against
> each resources' UUID, which will be used to compare the hash changes
> between next runs.
>
What are the fields which we have added to the hash?
>
>
> Highly appreciate your ideas on this.
>
> Thanks!
> Malintha
>
>
> --
> Malintha Amarasinghe
> *WSO2, Inc. - lean | enterprise | middleware*
> http://wso2.com/
>
> Mobile : +94 712383306
>
--
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection
Hi,
Micro gateway CLI works completely separately to API manager; whenever a
new API is added for a label, whenever there is a change happens to an
existing label there won't be any events published etc like previously. The
CLI needs to regenerate the source build it and push the artifacts to the
deployment and the full process needs to complete. In most occasions, the
CLI can be configured to run periodically to generate sources and do above
job.
But in this case, most of the time, the CLI will be uselessly generating
sources building it and pushing the artifacts to deployment. Comparatively,
building and pushing artifacts to deployment have a huge overhead compared
to generating sources.
This effort is to avoid that as much as possible by change-detection; i.e.
1. The CLI will check if any of the required resources has changed vs the
previous build and notify the user after a successful "setup" (source
generate) command using the command line output and the exit code of the
command.
2. Using the exit code, a user can write a shell script etc to decide
whether he should proceed with "build" or not.
*Proposed implementation:*
API Publisher APIs does not have ETag feature. Even if it is there, the
ETag will be generated for the whole resource. For code generation, we will
be only using few attributes of the resource, hence using a global ETag for
a resource may lead to unnecessary changes for the ETag. Hence the proposed
implementation will be using a CLI-side hash generation for *used
attributes *of the resource (API/Policies) only.
To mark the attributes which are used for generating the code, a newly
introduced annotation "@Hash" can be used.
Ex:
public class APIDetailedDTO extends APIInfoDTO {
/**
* Swagger definition of the APIDetailedDTO which contains details
about URI templates and scopes\n
**/
*@Hash*
@JsonProperty("apiDefinition")
public String getApiDefinition() {
return apiDefinition;
}
public void setApiDefinition(String apiDefinition) {
this.apiDefinition = apiDefinition;
}
/**
* WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
**/
@JsonProperty("wsdlUri")
public String getWsdlUri() {
return wsdlUri;
}
public void setWsdlUri(String wsdlUri) {
this.wsdlUri = wsdlUri;
}
*@Hash*
@JsonProperty("responseCaching")
public String getResponseCaching() {
return responseCaching;
}
The methods marked with *@Hash* will be automatically extracted from the
code and will be used to generate the hashes for each resource.
The generated hashes will be stored inside the CLI's temp folder against
each resources' UUID, which will be used to compare the hash changes
between next runs.
Highly appreciate your ideas on this.
Thanks!
Malintha
--
Malintha Amarasinghe
*WSO2, Inc. - lean | enterprise | middleware*
http://wso2.com/
Mobile : +94 712383306
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
