Support for the property type 'null'
Hi, I tried using the property type 'null' in one of the properties of a node type. And I didn't pass any value to the property in the respective node template. When I tried storing the template, I get the following validation issues, Storing service template null_test... Validation issues: 2: field "type" in "aria_extension_tosca.simple_v1_0.definitions.PropertyDefinition" is not a valid "str": None @"null.yaml":9:15 ValueError: not a "unicode": None 4: "type" refers to an unknown data type in "my_null_param": 'None' @"null.yaml":9:15 Failed to parse service template I used the following service template, tosca_definitions_version: tosca_simple_yaml_1_0 node_types: tosca.nodes.null: derived_from: tosca.nodes.Root properties: my_null_param: required: True type: null my_str_param: required: True type: string topology_template: node_templates: test_node: type: tosca.nodes.null properties: my_null_param: my_str_param: hello I had a look at the source code and found that 'null' is regarded as a primitive datatype. As 'null' type also have a YAML specification that must be adhered, could that be handled as a separate datatype like 'timestamp' ? And also is this a known issue? Does a JIRA exist for this? Thanks, /Vaish
Node state updation in custom workflows
Hi, How does the node state updation during the execution of the custom workflows is handled? I have defined a new interface type with its own set of lifecycle operations. It does not involve the TOSCA defined lifecycle operations like create, configure, start etc. So when I execute my service using the custom workflow, the node state is in 'initial' state. I arrived at the following observations after looking through the source code, 1. Only the 'standard' interface type is checked and the node states are updated based on the lifecycle operation associated with that interface type. 2. Node states are not updated when the interface type is not 'standard' and it remains with the default value 'initial' 3. In the model class of the node, only the set of node states defined by TOSCA and the 'deleted' state is hardcoded and set as enum to the status column Do we have any plans to support the updation of the node states during the custom workflow execution? If we have, will the user be allowed to pass the node states in the custom workflows module with the transitional and the finished states? Thanks, /Vaish
Re: pip executable expected as part of plugin install.
Hi, In ARIA, the pip executable path is used to get the output of the 'pip freeze' command. This lists the installed packages in the local system. This list is written to a file and passed as a constraint to the 'pip install' command used by wagon. At times, the installation directory changes and so the pip executable path. This creates problem in our systems and we need to handle the installation of pip exclusively. So it would be good to use the pip library instead of the executable path. I tried using the pip library inplace of the 'pip executable path' to get the list of installed packages. The pip library for freeze does not return anything but rather prints the output to the 'stdout'. So the stdout needs to be redirected and stored in a variable. Is redirecting stdout recommended? With my current understanding on the utilities provided by pip, this is the available option to use the pip library instead of the pip executable. Thanks, /Vaish From: Vaishnavi K.R Sent: Friday, December 8, 2017 11:47:21 AM To: dev@ariatosca.incubator.apache.org Subject: Re: pip executable expected as part of plugin install. Hi, There is no change with regards to this in the latest code base. The following is the error that we get when the pip executable is not found. Validating plugin /root/testplugin-1.0-py27-none-any.wgn... Plugin validated successfully Installing plugin /root/testplugin-1.0-py27-none-any.wgn... Retrieving source... Extracting zip /root/testplugin-1.0-py27-none-any.wgn to /tmp/tmp3Q0BL2... Source is: /tmp/tmp3Q0BL2/testplugin [Errno 2] No such file or directory Thanks, /Vaish From: Thomas Nadeau Sent: Sunday, December 3, 2017 2:57:22 PM To: dev@ariatosca.incubator.apache.org Subject: Re: pip executable expected as part of plugin install. DJ, If when you take a look at this, you find that it fails, please send detailed log/output so we can figure this out. I don’t see any of that from the thread below. Thx, —Tom On Sun, Dec 3, 2017 at 11:22 AM, Tal Liron wrote: > Hi DJ, > > A lot has changed since August. :) I wonder if you can take a look at the > current state of master and see if things have improved with wagon > installs? > > On Fri, Dec 1, 2017 at 9:22 AM, D Jayachandran < > d.jayachand...@ericsson.com> > wrote: > > > Hi Ran, > > > > Sorry I had missed to answer this thread. Just to answer your question > > wagon also expects pip as a binary "/usr/bin/pip". The above path may > not > > be the same for al distros of linux and when the path varies we run into > > the issue/ > > As I already told we could probably fix this issue by using pip as > library > > instead of a 3PP. > > Please let me know if we can also apply the same fix with wagon as well. > > > > Regards, > > DJ > > -Original Message- > > From: Ran Ziv [mailto:r...@cloudify.co] > > Sent: Sunday, August 20, 2017 12:40 PM > > To: dev@ariatosca.incubator.apache.org > > Subject: Re: pip executable expected as part of plugin install. > > > > Can you try to explain again what's the issue you're seeing with the way > > Wagon works right now? > > We could create a pull request for Wagon as well, but I'm not sure I > > understand the problem at the moment. > > > > On Wed, Aug 16, 2017 at 6:04 PM, D Jayachandran < > > d.jayachand...@ericsson.com > > > wrote: > > > > > Even if we fix the issue in ARIA. Wagon library still uses the same > > > logic in finding the pip path and it is wrong. > > > Am not sure how to fix this with wagon. > > > > > > Regards, > > > DJ > > > -Original Message- > > > From: D Jayachandran [mailto:d.jayachand...@ericsson.com] > > > Sent: Thursday, August 03, 2017 5:00 PM > > > To: dev@ariatosca.incubator.apache.org > > > Subject: RE: pip executable expected as part of plugin install. > > > > > > Thanks Avia, I will open an issue. > > > > > > Regards, > > > DJ > > > > > > -Original Message- > > > From: Avia Efrat [mailto:a...@cloudify.co] > > > Sent: Thursday, August 03, 2017 4:01 PM > > > To: dev@ariatosca.incubator.apache.org > > > Subject: Re: pip executable expected as part of plugin install. > > > > > > Hi DJ, > > > It seems you are correct, I don't see a reason for not using the pip > > > library. > > > Maybe it was that way since we didn't want to add pip as a dependency > > > explicitly (this code is from the beginning of ARIA). > > > > > > Feel free to open an issue about that =) > > > > > > On Wed, Aug 2, 2017 at 10:19 AM, D Jayachandran < > > > d.jayachand...@ericsson.com > > > > wrote: > > > > > > > Hi, > > > > > > > > Am using a Ubuntu version of linux for my development and ARIA does > > > > not find the correct path of pip during the plugin install. > > > > To be precise this happens when pip freeze is executed. > > > > > > > > @staticmethod > > > > def _pip_freeze(): > > > > """Run pip freeze in current environment and return the > > output""" > > > >
Re: Loading custom workflows
Hi, Do we have any update on the design of the loading mechanism? Thanks, /Vaish From: Tal Liron Sent: Thursday, December 7, 2017 8:54:47 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Loading custom workflows We had some good discussions are are just about ready to make them public for soliciting feedback. I will handle it next week and let the mailing list know! On Thu, Dec 7, 2017 at 2:31 PM, Thomas Nadeau wrote: > Do you think it is a good idea to start a document and/or wiki entry for a > thumbnail sketch of how we want to do this? > > --Tom > > On Thu, Dec 7, 2017 at 2:04 PM, Tal Liron wrote: > > > Our current idea is to support multiple ways of installing extensions. > One > > of them will, of course, be by including them in the CSAR file (which can > > be done in various ways: Python source code, wheels, wagons, etc.). We > want > > to standardize all extensions with a single well-defined entry point. > > > > On Wed, Dec 6, 2017 at 4:58 PM, Vaishnavi K.R < > vaishnavi@ericsson.com> > > wrote: > > > > > Hi Tal, > > > > > > > > > Good that this has been considered. > > > > > > As all the three entities are mere python modules, it is good to have a > > > common loading mechanism. > > > > > > But will there be any major change in the existing way of loading > plugins > > > and using the same in the service template? > > > > > > > > > Thanks, > > > > > > /Vaish > > > > > > > > > From: Tal Liron > > > Sent: Wednesday, December 6, 2017 7:27:02 PM > > > To: dev@ariatosca.incubator.apache.org > > > Subject: Re: Loading custom workflows > > > > > > Great question, and it was asked very recently on this list ... > > > > > > There is a need for a unified way to dynamically load > > > extensions/plugins/workflows from CSAR files as well as other places. > We > > > are trying to come with a good, forward-looking architectural design > for > > > this loading mechanism. I will provide an update on this in the next > few > > > days. > > > > > > On Wed, Dec 6, 2017 at 3:51 PM, Vaishnavi K.R < > > vaishnavi@ericsson.com> > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > I tried using the custom workflow support provided by ARIA. > > > > > > > > In the current ARIA, the custom workflows are directly imported as > > python > > > > modules. > > > > > > > > It looks like it is loading the python modules that are bundled along > > > with > > > > the service template in CSAR. > > > > > > > > > > > > Also I could see that you have plans to load it as how the plugins > are. > > > > > > > > Is anyone working on this? > > > > > > > > Will the workflows be packaged as wagon archives and installed? > > > > > > > > Will they be referred in the service template with the same > convention > > as > > > > plugins? > > > > > > > > > > > > Thanks, > > > > > > > > /Vaish > > > > > > > > > >
Re: pip executable expected as part of plugin install.
Hi,
There is no change with regards to this in the latest code base.
The following is the error that we get when the pip executable is not found.
Validating plugin /root/testplugin-1.0-py27-none-any.wgn...
Plugin validated successfully
Installing plugin /root/testplugin-1.0-py27-none-any.wgn...
Retrieving source...
Extracting zip /root/testplugin-1.0-py27-none-any.wgn to /tmp/tmp3Q0BL2...
Source is: /tmp/tmp3Q0BL2/testplugin
[Errno 2] No such file or directory
Thanks,
/Vaish
From: Thomas Nadeau
Sent: Sunday, December 3, 2017 2:57:22 PM
To: dev@ariatosca.incubator.apache.org
Subject: Re: pip executable expected as part of plugin install.
DJ,
If when you take a look at this, you find that it fails, please send
detailed log/output
so we can figure this out. I don’t see any of that from the thread below.
Thx,
—Tom
On Sun, Dec 3, 2017 at 11:22 AM, Tal Liron wrote:
> Hi DJ,
>
> A lot has changed since August. :) I wonder if you can take a look at the
> current state of master and see if things have improved with wagon
> installs?
>
> On Fri, Dec 1, 2017 at 9:22 AM, D Jayachandran <
> d.jayachand...@ericsson.com>
> wrote:
>
> > Hi Ran,
> >
> > Sorry I had missed to answer this thread. Just to answer your question
> > wagon also expects pip as a binary "/usr/bin/pip". The above path may
> not
> > be the same for al distros of linux and when the path varies we run into
> > the issue/
> > As I already told we could probably fix this issue by using pip as
> library
> > instead of a 3PP.
> > Please let me know if we can also apply the same fix with wagon as well.
> >
> > Regards,
> > DJ
> > -Original Message-
> > From: Ran Ziv [mailto:r...@cloudify.co]
> > Sent: Sunday, August 20, 2017 12:40 PM
> > To: dev@ariatosca.incubator.apache.org
> > Subject: Re: pip executable expected as part of plugin install.
> >
> > Can you try to explain again what's the issue you're seeing with the way
> > Wagon works right now?
> > We could create a pull request for Wagon as well, but I'm not sure I
> > understand the problem at the moment.
> >
> > On Wed, Aug 16, 2017 at 6:04 PM, D Jayachandran <
> > d.jayachand...@ericsson.com
> > > wrote:
> >
> > > Even if we fix the issue in ARIA. Wagon library still uses the same
> > > logic in finding the pip path and it is wrong.
> > > Am not sure how to fix this with wagon.
> > >
> > > Regards,
> > > DJ
> > > -Original Message-
> > > From: D Jayachandran [mailto:d.jayachand...@ericsson.com]
> > > Sent: Thursday, August 03, 2017 5:00 PM
> > > To: dev@ariatosca.incubator.apache.org
> > > Subject: RE: pip executable expected as part of plugin install.
> > >
> > > Thanks Avia, I will open an issue.
> > >
> > > Regards,
> > > DJ
> > >
> > > -Original Message-
> > > From: Avia Efrat [mailto:a...@cloudify.co]
> > > Sent: Thursday, August 03, 2017 4:01 PM
> > > To: dev@ariatosca.incubator.apache.org
> > > Subject: Re: pip executable expected as part of plugin install.
> > >
> > > Hi DJ,
> > > It seems you are correct, I don't see a reason for not using the pip
> > > library.
> > > Maybe it was that way since we didn't want to add pip as a dependency
> > > explicitly (this code is from the beginning of ARIA).
> > >
> > > Feel free to open an issue about that =)
> > >
> > > On Wed, Aug 2, 2017 at 10:19 AM, D Jayachandran <
> > > d.jayachand...@ericsson.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Am using a Ubuntu version of linux for my development and ARIA does
> > > > not find the correct path of pip during the plugin install.
> > > > To be precise this happens when pip freeze is executed.
> > > >
> > > > @staticmethod
> > > > def _pip_freeze():
> > > > """Run pip freeze in current environment and return the
> > output"""
> > > > bin_dir = 'Scripts' if os.name == 'nt' else 'bin'
> > > > pip_path = os.path.join(sys.prefix, bin_dir,
> > > > 'pip{0}'.format('.exe' if os.name ==
> > > 'nt'
> > > > else ''))
> > > > pip_freeze = subprocess.Popen([pip_path, 'freeze'],
> > > > stdout=subprocess.PIPE)
> > > > pip_freeze_output, _ = pip_freeze.communicate()
> > > > assert not pip_freeze.poll()
> > > > return pip_freeze_output
> > > >
> > > > Now the question is why are we executing a pip command directly and
> > > > not using pip as a library.
> > > >
> > > >
> > > > Regards,
> > > > DJ
> > > >
> > >
> >
>
Re: Loading Custom Workflows from CSAR
Hi Tal, The loading of the python modules with python functions for workflows happens when the module is in PYTHONPATH. But I could also see that, the absolute path of the resource storage with the service template ID is added to sys.path to make it available for python. This facilitates the loading of modules from CSAR, provided the modules are present in the root directory of the CSAR. Am I right? Thanks, /Vaish From: Tal Liron Sent: Thursday, December 7, 2017 8:51:49 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Loading Custom Workflows from CSAR Actually, the current implementation does not use the CSAR: it expects that the Python function (decorated by @workflow) would be somewhere in the Python path, and we are currently missing a CSAR code-loading mechanism. We do not expect the @workflow function structure to change in any way, so I think you can feel confident about writing custom workflows in this matter. What might change slightly in the near future is how to package the .py file. On Thu, Dec 7, 2017 at 2:57 PM, Steve Baillargeon < steve.baillarg...@ericsson.com> wrote: > Hi > While the investigation is on-going, I need to double check what can be > done today. > But I guess how it is done today might change as well (?) > Based on our analysis of the latest code base, the loading of the custom > workflows today is from CSAR only. > Please advise. > > -Steve > > -Original Message- > From: Tal Liron [mailto:t...@cloudify.co] > Sent: Tuesday, November 28, 2017 2:06 PM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Loading Custom Workflows from CSAR > > Thanks, Steve. We don't have a JIRA for this right now, but we do intend > to have a way to include "plugins" in a CSAR and have them automatically > installed. A "plugin" is basically a Python extension to ARIA that can be > loaded at runtime and contained per service. > > This is part of our current general work trying to find a good design for > plugins generally, and will likely result in a JIRA epic with several > issues. At this time, I don't think it's worth created this epic if we > don't have a plan. > > The design will likely be inspired by the plugin design in Cloudify, from > which we grabbed our seed code. However, there is room for some > re-imagination especially as pertains TOSCA-specific issues. > > On Tue, Nov 28, 2017 at 1:01 PM, Steve Baillargeon < > steve.baillarg...@ericsson.com> wrote: > > > Hi > > As far as I know the implementation string associated with the > > aria.Workflow type must call or execute a Python function (using the > > dot > > notation) that is stored in ARIA, like a plugin. > > When will it be possible to refer to a .py file stored in the CSAR > > instead? > > Do you have any Jira for this enhancement? > > > > Regards > > Steve B > > >
Re: Loading custom workflows
Hi Tal, Good that this has been considered. As all the three entities are mere python modules, it is good to have a common loading mechanism. But will there be any major change in the existing way of loading plugins and using the same in the service template? Thanks, /Vaish From: Tal Liron Sent: Wednesday, December 6, 2017 7:27:02 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Loading custom workflows Great question, and it was asked very recently on this list ... There is a need for a unified way to dynamically load extensions/plugins/workflows from CSAR files as well as other places. We are trying to come with a good, forward-looking architectural design for this loading mechanism. I will provide an update on this in the next few days. On Wed, Dec 6, 2017 at 3:51 PM, Vaishnavi K.R wrote: > Hi, > > > I tried using the custom workflow support provided by ARIA. > > In the current ARIA, the custom workflows are directly imported as python > modules. > > It looks like it is loading the python modules that are bundled along with > the service template in CSAR. > > > Also I could see that you have plans to load it as how the plugins are. > > Is anyone working on this? > > Will the workflows be packaged as wagon archives and installed? > > Will they be referred in the service template with the same convention as > plugins? > > > Thanks, > > /Vaish >
Loading custom workflows
Hi, I tried using the custom workflow support provided by ARIA. In the current ARIA, the custom workflows are directly imported as python modules. It looks like it is loading the python modules that are bundled along with the service template in CSAR. Also I could see that you have plans to load it as how the plugins are. Is anyone working on this? Will the workflows be packaged as wagon archives and installed? Will they be referred in the service template with the same convention as plugins? Thanks, /Vaish
Re: decoupling type definitions from CSAR
Hi, I agree with your points. It is possible to load the type definitions or any files using the URL. But in a situation where 1. it is not recommended to access the external repositories 2. for the user to be sure of what is being used it would be good to provide an option for the user to import the type definitions locally and reuse it across service templates. Thanks, /Vaish From: Tal Liron Sent: Friday, November 10, 2017 10:40:14 PM To: dev@ariatosca.incubator.apache.org Subject: Re: decoupling type definitions from CSAR I'll just add that you can either use a repository or just a complete URL in the "file" section of "imports" (or short-form, which is just the URL). Also, obviously there is a risk in using external imports: external changes could break existing imports. But considering the complexities of cloud deployments, I imagine there are a lot of dependencies on external factors, anyway... On Fri, Nov 10, 2017 at 10:12 AM, Steve Baillargeon < steve.baillarg...@ericsson.com> wrote: > Hi Vaish > I agree. > > We should also consider the same idea for NFV and ONAP types (as opposed > to make them built-in). > I suspect those types may get updated without the need to update the ARIA > orchestrator version. > > Is it just a matter of allowing the admin user to load/delete type > definitions files into the ARIA local host and specifying the repository > option in the import definition? > Similar to plugins? > > For instance: > > For individually managed plugins: > -file : plugin.yaml > repository: plugins > > For individually managed type definitions files: > -file : typedefintion.yaml > repository: definitions > > > Regards > Steve B > > > > > -Original Message- > From: Vaishnavi K.R [mailto:vaishnavi@ericsson.com] > Sent: Friday, November 10, 2017 6:30 AM > To: dev@ariatosca.incubator.apache.org > Subject: decoupling type definitions from CSAR > > Hi, > > > With the current ARIA, in order to use the type definitions that are user > defined, it must be imported in the Service Template. The definitions files > are imported by specifying the relative paths in the 'imports' section. > These type definition files can rather reside in the localhost running ARIA > or can be made available by bundling along with the service templates in a > CSAR. For remote executions, the latter holds good. > > > However if the type definitions are so common and can be used across > Service Templates, it becomes mandatory to include the same file in > different CSARs. > > > In order to reuse the type definitions that are common across service > templates, it would be better to maintain the user defined type definitions > separately and every service template can use it irrespective of their > CSARs. > > > This makes the user defined type definitions to be loaded once and the > service templates could import it. > > > I would like to know your view on this. > > > Thanks, > > /Vaish >
decoupling type definitions from CSAR
Hi, With the current ARIA, in order to use the type definitions that are user defined, it must be imported in the Service Template. The definitions files are imported by specifying the relative paths in the 'imports' section. These type definition files can rather reside in the localhost running ARIA or can be made available by bundling along with the service templates in a CSAR. For remote executions, the latter holds good. However if the type definitions are so common and can be used across Service Templates, it becomes mandatory to include the same file in different CSARs. In order to reuse the type definitions that are common across service templates, it would be better to maintain the user defined type definitions separately and every service template can use it irrespective of their CSARs. This makes the user defined type definitions to be loaded once and the service templates could import it. I would like to know your view on this. Thanks, /Vaish
Re: Unique identification of an instance element across services
Thanks Tal. I think many of us are not convinced with the inclusion of the UUID for element identification. As of now, we have ID for the unique identification. So why should we restrict the users from giving duplicate names for the service templates. I wish to confirm if anyone has a second opinion in allowing duplicate names. If not, I can raise a JIRA issue and fix it. Looking forward to your comments. Thanks, /Vaish From: Tal Liron Sent: Tuesday, September 26, 2017 2:04:50 AM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services DeWayne, I think you missed parts of this discussion where we answered some these issues: 1) ARIA *does* allow you configure you choice of ID generation, and I agree it can be an integration requirement. (We have a JIRA open to give this configuration a CLI.) 2) ARIA has a choice of UUID formats beyond the very long 36-character hex. Base57 is 22 characters and designed for human readability. 3) All the costs you mention seem very negligible to me. ARIA's database storage is tiny. UUID generation happens only when new nodes are created, and is many orders of magnitude faster than storing anything in a database. It doesn't easy to resolve this issue, as there are two camps here. But I'm very convinced that Vaish and I are correct here. :) Node (and service) names in the real world are used for other systems beyond ARIA once the nodes are installed: names become domain names linked to IP addresses, names of operating system services, registration IDs for message queues, analytics IDs, etc. For all of these a collision is disastrous, and Vaish is right that if it's set up initially by ARIA there is no need to do anything else. The only possible user discomfort is in using the "aria nodes show", which frankly is a command that I have never even used myself. As for logs, in any install that has more than one service you will be filtering by workflow ID or service ID anyway. Also, in case there was any confusion: we are talking about node names here, not database IDs. The database IDs are entirely handled within the database and are obviously unique only per table and per installation of ARIA. The "name" is an extra column that is uses out in the real world. If this will have to come to a vote among committers, I will absolutely advocate UUIDs by default, and a preference for base57 format. I would consider any other kind of default to be broken for real world cloud orchestration, and I would be very worried if ARIA is to be broken out of the box. On Thu, Sep 21, 2017 at 11:09 AM, DeWayne Filppi wrote: > IMHO, obviously UUIDs "work", in the sense that they are "universally > unique" and therefore make collisions unlikely. On the other hand, they > are "universally unique", which includes time and space. There is a cost > to that, and it is the ridiculous number of bits used (IOW they are > insanely inefficient). That has a cost both in storage and readability.. > Also, unless there is a way to mathematically map the UUID to the table > index it refers to, the UUID will have to be in the database, and therefore > the database will be exposed to the user. Besides bulk, the UUID gets > exposed in logs ( and occasionally in the CLI ), which just creates a mess > and eats storage. So UUIDs work, but are a last resort, IMO. Has anyone > put any thought into a structured ID? Structured IDs are far more useful > and user friendly (readable) and potentially compact. I think it would be > good to at least exhaust alternatives before resorting to UUIDs. Another > option is just to punt, make user exposed ID generation pluggable, and > provide a default implementation (or two). This would allow consumers to > use their own ID formats, which might be an integration requirement. > > --DeWayne > > On Thu, Sep 21, 2017 at 4:25 AM, Vaishnavi K.R > > wrote: > > > Hi, > > > > > > Yes. You are correct. IDs remain unique across the table. > > > > Usually the IDs in the database are used for the internal operations. > > > > In general, they need not be exposed to the user. It is more used by the > > application itself. > > > > > > That's why it would be better to have an UUID which is specially meant to > > be used by the user. And also in the large scale environments, where huge > > number of service templates and instances pour in, they could have > uniform > > identification IDs rather than incremental numbers. > > > > > > And about allowing duplicate names for the service templates and service > > instances, it is MUST to have it. In multi user and multi tenant > > applications, the probability of getting the duplicate names is high. So > > its better to handle it in the initial phase itself. > > > > > > So I would like to know your suggestion and comment on the following > three > > items, > > > > > > 1. Allowing duplicate names for the service templates and service > > i
Re: Unique identification of an instance element across services
Hi, Yes. You are correct. IDs remain unique across the table. Usually the IDs in the database are used for the internal operations. In general, they need not be exposed to the user. It is more used by the application itself. That's why it would be better to have an UUID which is specially meant to be used by the user. And also in the large scale environments, where huge number of service templates and instances pour in, they could have uniform identification IDs rather than incremental numbers. And about allowing duplicate names for the service templates and service instances, it is MUST to have it. In multi user and multi tenant applications, the probability of getting the duplicate names is high. So its better to handle it in the initial phase itself. So I would like to know your suggestion and comment on the following three items, 1. Allowing duplicate names for the service templates and service instances 2. Appending UUID to the node instances 3. Identifying the service templates and the service instances by UUIDs (not appended to their names, because that might confuse the user when a list of items are scrolled on) Thanks, /Vaish From: Ran Ziv Sent: Monday, September 18, 2017 4:25:57 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services The service name is optional - it may be auto-generated according to the service-template name. The service-template name can also be made optional (see this jira issue: https://issues.apache.org/jira/browse/ARIA-221 ). Regarding the scenario of non-CLI interaction - for any non-human usage, IDs should be used, as they're guaranteed to be unique. I don't see why UUIDs are necessary in this case. On Mon, Sep 18, 2017 at 9:22 AM, Vaishnavi K.R wrote: > Hi All, > > > In addition to the node instance name, I am concerned about the service > template name and the service instance names. In a wider perspective, there > is high chance for these names to be the same. > > > And as I have already mentioned in previous discussion, its an overhead > for an user to change the name again and again when he encounters the > 'already exist' error. > > > And also when ARIA is used as a TOSCA Orchestration service provider, > manual interaction via CLI won't happen. All operations could be performed > over the HTTP calls. In such scenerio, it would be great and very much > useful, if elements are queried or identified using the UUID. > > > So I see the uniqueness should prevail across the elements like service > templates, service instances and node instances. > > > Thanks, > > /Vaish > > > From: Ran Ziv > Sent: Saturday, September 16, 2017 12:12:22 AM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Unique identification of an instance element across services > > I can't seem to be able to access our JIRA at the moment, but generally > speaking, the CLI currently supports "static completion" only, i.e. it > auto-completes CLI commands but not object names. > We tried implementing dynamic completion (e.g. tab on "-s" would > auto-complete service names from the storage), but we ran into some issues > with the underlying Click framework. > I'm not sure if an issue for trying to implement this further is currently > open on our JIRA. > > Regarding a "partial hash" concept, I don't really find this to be useful > in this case. IMO, as Tal's mentioned, the cases when you need to actually > use these auto-generated long names are rare, and when that happens, > dynamic completion can take care of it well, if we can get it done. > > > > On Fri, Sep 15, 2017 at 9:04 PM, Thomas Nadeau > wrote: > > > > > > On Sep 15, 2017, at 1:53 PM, Tal Liron wrote: > > > > > > When do you actually have to ender node names? Probably only in "aria > > nodes > > > show". And in those cases you would be copy-pasting from a list. We > could > > > also improve our CLI completion code to properly complete node IDs. > > > > That sounds like a very useful enhancement. Do we have a Jira > for > > this yet? *) > > > > > I think the serial numbers are more confusing than helpful. Let's say > you > > > currently have 20 difference services running, and they are of various > > > different service templates. But let's say a few service templates have > > > node templates with the same name, "database". You could potentially > > > "database_1" in the list and "database_2", but each one of these nodes > > > would be of a different node template of a different service template. > > The > > > serial number gives the false sense that these two nodes are somehow > > > together. Anyway, we discussed this in much detail already: we all > agree > > > that the serial system is totally broken if you're using more than ARIA > > > install, or even if a few different ARIA users are using the same cloud > > > accounts (each ARIA install could create its
Re: Unique identification of an instance element across services
Hi All, In addition to the node instance name, I am concerned about the service template name and the service instance names. In a wider perspective, there is high chance for these names to be the same. And as I have already mentioned in previous discussion, its an overhead for an user to change the name again and again when he encounters the 'already exist' error. And also when ARIA is used as a TOSCA Orchestration service provider, manual interaction via CLI won't happen. All operations could be performed over the HTTP calls. In such scenerio, it would be great and very much useful, if elements are queried or identified using the UUID. So I see the uniqueness should prevail across the elements like service templates, service instances and node instances. Thanks, /Vaish From: Ran Ziv Sent: Saturday, September 16, 2017 12:12:22 AM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services I can't seem to be able to access our JIRA at the moment, but generally speaking, the CLI currently supports "static completion" only, i.e. it auto-completes CLI commands but not object names. We tried implementing dynamic completion (e.g. tab on "-s" would auto-complete service names from the storage), but we ran into some issues with the underlying Click framework. I'm not sure if an issue for trying to implement this further is currently open on our JIRA. Regarding a "partial hash" concept, I don't really find this to be useful in this case. IMO, as Tal's mentioned, the cases when you need to actually use these auto-generated long names are rare, and when that happens, dynamic completion can take care of it well, if we can get it done. On Fri, Sep 15, 2017 at 9:04 PM, Thomas Nadeau wrote: > > > On Sep 15, 2017, at 1:53 PM, Tal Liron wrote: > > > > When do you actually have to ender node names? Probably only in "aria > nodes > > show". And in those cases you would be copy-pasting from a list. We could > > also improve our CLI completion code to properly complete node IDs. > > That sounds like a very useful enhancement. Do we have a Jira for > this yet? *) > > > I think the serial numbers are more confusing than helpful. Let's say you > > currently have 20 difference services running, and they are of various > > different service templates. But let's say a few service templates have > > node templates with the same name, "database". You could potentially > > "database_1" in the list and "database_2", but each one of these nodes > > would be of a different node template of a different service template. > The > > serial number gives the false sense that these two nodes are somehow > > together. Anyway, we discussed this in much detail already: we all agree > > that the serial system is totally broken if you're using more than ARIA > > install, or even if a few different ARIA users are using the same cloud > > accounts (each ARIA install could create its own "database_1" -- what if > > you have two hosts with that same DNS name?). > > I was just going to say the point you made above about DNS name > overlap. > It sounds like we need to sit down and re-visit the serial number > management? > > —Tom > > > > > > > > On Fri, Sep 15, 2017 at 12:35 PM, DeWayne Filppi > > wrote: > > > >> I get the feeling that you are more gifted typist than most. Or are you > >> saying nobody will ever be required to type in one those IDs? > >> > >> On Fri, Sep 15, 2017 at 9:27 AM, Tal Liron wrote: > >> > >>> Before we allow users to configure this, we have another JIRA to > resolve: > >>> actually, we don't have a mechanism for storing configuration yet! Here > >> is > >>> the open JIRA: > >>> > >>> https://issues.apache.org/jira/browse/ARIA-229 > >>> > >>> As for what to configure in this case, our practice until now was that > >> the > >>> UUID would be added as an underscored postfix of the object's name. So > if > >>> you have a node template named "database" then node instances could be, > >>> assuming longest form of UUID (alphanumeric, 36 characters): > >>> > >>> database_2d79fd86-0877-49ca-81d8-cd2dc9f7b0e2 > >>> database_2819972e-3b07-4923-be94-43e95985155f > >>> database_45b9faf5-8bf4-482a-a570-d1c058270424 > >>> > >>> This guarantees names that are universally unique and yet still > >> meaningful: > >>> you would be able to tell at a glance what kind of node this is: a > >>> database. Note that we also have a mechanism in place to warn you if > the > >>> final name is more than 63 characters, because such names can't be used > >> as > >>> DNS hostnames (a common usage for node names in the cloud). This should > >>> also be configurable. > >>> > >>> I don't see why this needs to be abstracted from the user. If you are > >> using > >>> the CLI and see the list of nodes, you can refer to the node you are > >>> examining with the full name as seen above. I think having a separate > UI > >>> name s
Property type 'null' validation issue
Hi, I tried using the 'null' property type as mentioned in the TOSCA simple YAML 1.0 specification. I get the validation issue saying that the datatype specified in the service template is an unknown datatype. I just surfed through the code and found that 'null' is considered as one of the primitive datatypes. But when it is validated if its a primitive datatype, a class object of Null() is passed as a value. The validation fails as it expects a string value of 'null'. Is this a known issue? Do we have a JIRA item opened for this? Thanks, /Vaish
Re: Unique identification of an instance element across services
Hi, Thanks for the update. With current ARIA, the utility module to generate the UUID is available. But the UUID support will also mandate the following changes if my understanding is right, 1. the inclusion of the UUID column in the mapper classes of sqlalchemy 2. the model object created should set the value for the UUID and send it to database For an use case in our product, we badly need this UUID based element identification. So I look forward to your comments on the following, 1. We contribute the UUID support to ARIA without affecting the current CLI module i.e. CLI will continue to use the name option. The UUID will be completely abstracted from the user. 2. Configurable option to use UUID or name based identification. By default, it will work with the name based identification Also I need clarification on the UUID generation. Currently ARIA supports four variants. Do we have any standard on how this UUID should be and also on what aspect these four variants are concluded on? Thanks, /Vaish From: Tal Liron Sent: Tuesday, September 5, 2017 8:24:41 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services We already have utility code to generate all kinds of UUIDs, so it's trivial to make the change. I guess it's just a matter of making a decision... On Tue, Sep 5, 2017 at 3:57 AM, Vaishnavi K.R wrote: > > Hi, > > I agree that with the CLI based usage in ARIA, the requirement of the UUID > based identification of the node and service instance elements is an > overhead. > > From the discussions so far, it seems like UUID is important in handling > the multi-user and multi-tenant scenarios. > > Do you have any update on when UUID will be considered in the roadmap? > If its not too far, we would like to make our contribution to ARIA on UUID. > > Looking forward to your response. > > > Thanks, > > /Vaish > > > From: Avia Efrat > Sent: Sunday, July 30, 2017 10:35:45 PM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Unique identification of an instance element across services > > First, good arguments from both 'sides'. > > I am for at least adding a uuid as an option, as ARIA is intended to be > used at scale as well. > But currently, I am for the simple ids to be used as default (and not > uuids). Like it or not, right now ARIA is more at a 'TOSCA playground' > stage, and I think that's perfectly fine =) > And at this stage, I think simple ids will be better, as they easier to use > via the CLI, but more importantly, don't clog the logs with long > meaningless strings. As ARIA matures, we could switch the default to UUIDs. > > And BTW, as our log format is configurable, there could be other ways than > UUIDs to distinguish between nodes with the 'same id' in a central logging > system, e.g using the user name as another indicator. > > On Thu, Jul 27, 2017 at 10:20 AM, Vaishnavi K.R < > vaishnavi@ericsson.com> > wrote: > > > Hi, > > > > > > Thanks for the active discussion. > > > > > > Having UUID at the node instance level will just make the nodes unique. > > > > And these names will not be used by the user directly as no operations > are > > happening on the node instance name. > > > > But at the service template and the service level, UUID will be of great > > help considering the multi user and multi tenancy situations. > > > > > > So in a large scale perspective, the node names and the service template > > names have high probability of being same. > > > > When these enter into the automated world, it will create more problem > > when name conflicts occur and its adds overhead to make changes every > time > > to overcome the conflict. > > > > > > UUID at service template and the service level: will be of much use in > the > > above scenario and operations by user on these are less > > > > UUID at node instance level: makes the node much unique and no operation > > happens on it > > > > > > Thanks, > > > > /Vaish > > > > > > From: Tal Liron > > Sent: Wednesday, July 26, 2017 8:48:40 PM > > To: dev@ariatosca.incubator.apache.org > > Subject: Re: Unique identification of an instance element across services > > > > I just don't see users having to deal much with node IDs outside of > simple > > hello-world style tutorials, and I'd hate for the first impressions that > > users get out of ARIA is that it's just a playground for TOSCA. It should > > be ready out-of-the-box for the real world. > > > > On Wed, Jul 26, 2017 at 9:13 AM, DeWayne Filppi > > wrote: > > > > > Such is their strength. I'm just advocating using them as a last > resort > > > because they are user unfriendly and gigantic. > > > > > > On Tue, Jul 25, 2017 at 12:55 PM, Tal Liron wrote: > > > > > > > Let's consider a mass deployment: thousands of service instances of > the > > > > same service template, created by many different
Re: Unique identification of an instance element across services
Hi, I agree that with the CLI based usage in ARIA, the requirement of the UUID based identification of the node and service instance elements is an overhead. >From the discussions so far, it seems like UUID is important in handling the >multi-user and multi-tenant scenarios. Do you have any update on when UUID will be considered in the roadmap? If its not too far, we would like to make our contribution to ARIA on UUID. Looking forward to your response. Thanks, /Vaish From: Avia Efrat Sent: Sunday, July 30, 2017 10:35:45 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services First, good arguments from both 'sides'. I am for at least adding a uuid as an option, as ARIA is intended to be used at scale as well. But currently, I am for the simple ids to be used as default (and not uuids). Like it or not, right now ARIA is more at a 'TOSCA playground' stage, and I think that's perfectly fine =) And at this stage, I think simple ids will be better, as they easier to use via the CLI, but more importantly, don't clog the logs with long meaningless strings. As ARIA matures, we could switch the default to UUIDs. And BTW, as our log format is configurable, there could be other ways than UUIDs to distinguish between nodes with the 'same id' in a central logging system, e.g using the user name as another indicator. On Thu, Jul 27, 2017 at 10:20 AM, Vaishnavi K.R wrote: > Hi, > > > Thanks for the active discussion. > > > Having UUID at the node instance level will just make the nodes unique. > > And these names will not be used by the user directly as no operations are > happening on the node instance name. > > But at the service template and the service level, UUID will be of great > help considering the multi user and multi tenancy situations. > > > So in a large scale perspective, the node names and the service template > names have high probability of being same. > > When these enter into the automated world, it will create more problem > when name conflicts occur and its adds overhead to make changes every time > to overcome the conflict. > > > UUID at service template and the service level: will be of much use in the > above scenario and operations by user on these are less > > UUID at node instance level: makes the node much unique and no operation > happens on it > > > Thanks, > > /Vaish > > > From: Tal Liron > Sent: Wednesday, July 26, 2017 8:48:40 PM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Unique identification of an instance element across services > > I just don't see users having to deal much with node IDs outside of simple > hello-world style tutorials, and I'd hate for the first impressions that > users get out of ARIA is that it's just a playground for TOSCA. It should > be ready out-of-the-box for the real world. > > On Wed, Jul 26, 2017 at 9:13 AM, DeWayne Filppi > wrote: > > > Such is their strength. I'm just advocating using them as a last resort > > because they are user unfriendly and gigantic. > > > > On Tue, Jul 25, 2017 at 12:55 PM, Tal Liron wrote: > > > > > Let's consider a mass deployment: thousands of service instances of the > > > same service template, created by many different users with their own > > ARIA > > > installations (and databases). In that case, assuming we use sequential > > > IDs, you would have the same node ID appear many times. You would have > to > > > identify it via the particular user and service instance. If you're > > > centralizing logs, this can quickly be cumbersome. A UUID will identify > > it > > > globally and avoid any confusion. > > > > > > I think the default should be something that avoids such problems. For > > > users who insist on shorter IDs, we can allow them to configure it. > > > > > > On Tue, Jul 25, 2017 at 2:42 PM, DeWayne Filppi > > > wrote: > > > > > > > True uuids are seductive, because of their simplicity. But they are > > > huge, > > > > overkill, and meaningless. Imho a structured id is superior if it > can > > be > > > > made to work without a global locking scheme. > > > > > > > > - DeWayne > > > > > > > > On Jul 25, 2017 12:11 PM, "Tal Liron" wrote: > > > > > > > > > It's not an issue of thread safety -- it could be entirely > different > > > > > processes, on different machines, accessing the same db. It can be > > > solved > > > > > via a SQL transaction, but I feel the whole issue can be avoided by > > > using > > > > > UUIDs. > > > > > > > > > > Using the CLI to access specific nodes is not something I see > > > happening a > > > > > lot outside of debugging. And when you do debug, you'll probably be > > > > copying > > > > > and pasting a node ID from the logs, so shorter names do not add > much > > > > ease > > > > > of use. > > > > > > > > > > Again, I would be personally happiest if this was configurable (and > > > > > personally think UUIDs should be the reasonable
Re: TOSCA spec compliance on finding target node
Hi, I tried the following combinations in my service template, 1. Type definition with capability type alone but node template having any of the following, * capability type alone * capability name alone * node type alone * node name alone * capability name and node name * capability name and node type * capability type and node type * capability type and node type 2. Type definition with capability type and node type * capability type alone * capability name alone * node type alone * node name alone * capability name and node name * capability name and node type * capability type and node type * capability type and node type As per the TOSCA specification, the above are valid combinations. Will ARIA support all the above ?? If so, we wish to contribute. Looking forward to your comment. Thanks, /Vaish From: Tal Liron Sent: Tuesday, July 25, 2017 10:03:18 PM To: dev@ariatosca.incubator.apache.org Subject: Re: TOSCA spec compliance on finding target node It indeed should *not* be required. I just verified that it you are correct, and a match is not made if only the capability is specified without a node type/template. This is a regression, because it used to work correctly. There is currently work in progress to refactor that mechanism, so I will add a test case to make sure the regression is fixed. See my test case and follow progress here: https://issues.apache.org/jira/browse/ARIA-174 On Tue, Jul 25, 2017 at 3:28 AM, Vaishnavi K.R wrote: > Hi ARIA folks, > > > I had a look at the source code of ARIA on how the target node is > identified based on the requirement and capability information furnished in > the node template and its corresponding node type. But I find that only few > of the combinations are supported i.e., as per the TOSCA spec, in the > requirement section of a node template, the 'node' option is not mandatory, > but ARIA expects that to be present. > > > In my use-case, my node template has a requirement on a node which has a > particular capability. So I just specify the capability type in my node > template under the requirement section. As ARIA expects the 'node' option > to be present, this use-case fails. > > > So I wish to get clarified is there any specific reason for mandating the > 'node' option or if TOSCA spec compliance on this target identification > based on the capability name or type will be supported in the future > versions? > > > Thanks, > > /Vaish >
Re: Unique identification of an instance element across services
Hi, Thanks for the active discussion. Having UUID at the node instance level will just make the nodes unique. And these names will not be used by the user directly as no operations are happening on the node instance name. But at the service template and the service level, UUID will be of great help considering the multi user and multi tenancy situations. So in a large scale perspective, the node names and the service template names have high probability of being same. When these enter into the automated world, it will create more problem when name conflicts occur and its adds overhead to make changes every time to overcome the conflict. UUID at service template and the service level: will be of much use in the above scenario and operations by user on these are less UUID at node instance level: makes the node much unique and no operation happens on it Thanks, /Vaish From: Tal Liron Sent: Wednesday, July 26, 2017 8:48:40 PM To: dev@ariatosca.incubator.apache.org Subject: Re: Unique identification of an instance element across services I just don't see users having to deal much with node IDs outside of simple hello-world style tutorials, and I'd hate for the first impressions that users get out of ARIA is that it's just a playground for TOSCA. It should be ready out-of-the-box for the real world. On Wed, Jul 26, 2017 at 9:13 AM, DeWayne Filppi wrote: > Such is their strength. I'm just advocating using them as a last resort > because they are user unfriendly and gigantic. > > On Tue, Jul 25, 2017 at 12:55 PM, Tal Liron wrote: > > > Let's consider a mass deployment: thousands of service instances of the > > same service template, created by many different users with their own > ARIA > > installations (and databases). In that case, assuming we use sequential > > IDs, you would have the same node ID appear many times. You would have to > > identify it via the particular user and service instance. If you're > > centralizing logs, this can quickly be cumbersome. A UUID will identify > it > > globally and avoid any confusion. > > > > I think the default should be something that avoids such problems. For > > users who insist on shorter IDs, we can allow them to configure it. > > > > On Tue, Jul 25, 2017 at 2:42 PM, DeWayne Filppi > > wrote: > > > > > True uuids are seductive, because of their simplicity. But they are > > huge, > > > overkill, and meaningless. Imho a structured id is superior if it can > be > > > made to work without a global locking scheme. > > > > > > - DeWayne > > > > > > On Jul 25, 2017 12:11 PM, "Tal Liron" wrote: > > > > > > > It's not an issue of thread safety -- it could be entirely different > > > > processes, on different machines, accessing the same db. It can be > > solved > > > > via a SQL transaction, but I feel the whole issue can be avoided by > > using > > > > UUIDs. > > > > > > > > Using the CLI to access specific nodes is not something I see > > happening a > > > > lot outside of debugging. And when you do debug, you'll probably be > > > copying > > > > and pasting a node ID from the logs, so shorter names do not add much > > > ease > > > > of use. > > > > > > > > Again, I would be personally happiest if this was configurable (and > > > > personally think UUIDs should be the reasonable default). > > > > > > > > On Tue, Jul 25, 2017 at 2:01 PM, Maxim Orlov > > wrote: > > > > > > > > > Technically we have no issue with implementing this via uuid or a > > > > > threadsafe solution for the current index implementation. > > > > > > > > > > Getting node data via the cli feels more intuitive using the index > > > based > > > > > ID, rather than the uuid based ID in my opionion. > > > > > > > > > > On Jul 25, 2017 9:49 PM, "Tal Liron" wrote: > > > > > > > > > > Our code for determining the next index is not concurrently safe > (no > > > > atomic > > > > > transaction) so I can see it breaking in concurrent use cases > > (running > > > > two > > > > > ARIA commands at the same time). > > > > > > > > > > What is to gain here in terms of human readability? In my opinion > it > > > adds > > > > > confusion because it gives a false sense of predictability. > > > > > > > > > > In my opinion the best compromise is to use base57-encoded UUIDs. > > These > > > > are > > > > > true UUIDs, but use a mix of upper and lowercase alphanumerics > > ensuring > > > > no > > > > > visually ambiguous characters. We have the code for this in > > > > utils/uuid.py. > > > > > > > > > > See also: https://github.com/wyattisimo/base57-ruby > > > > > > > > > > On Tue, Jul 25, 2017 at 1:28 PM, Maxim Orlov > > > wrote: > > > > > > > > > > > Actually the refactoring was made so the id would be more user > > > > readable. > > > > > > The index is determined according to the used indices (it's not > > just > > > a > > > > > > running number). If indeed this poses an issue (or if indeed a > uuid > > > is > > > > > > easier to recognize, or even use in a
Unique identification of an instance element across services
Hi, With my understanding in current ARIA, the node instances are made unique by prefixing the node name with the 'id of the service' (i.e. the primary key of the service table) as the instances are specific to the service. What will be the name of the node instances if the default instances for the node template is '3' and how this will hold good during scale in and out? Could UUID be of great help in handling such cases by including that as a column in the database tables of the service and the node? This will wipe out the naming confusions and querying can be made easy with the UUIDs. Looking forward to your suggestion. Thanks, /Vaish
TOSCA spec compliance on finding target node
Hi ARIA folks, I had a look at the source code of ARIA on how the target node is identified based on the requirement and capability information furnished in the node template and its corresponding node type. But I find that only few of the combinations are supported i.e., as per the TOSCA spec, in the requirement section of a node template, the 'node' option is not mandatory, but ARIA expects that to be present. In my use-case, my node template has a requirement on a node which has a particular capability. So I just specify the capability type in my node template under the requirement section. As ARIA expects the 'node' option to be present, this use-case fails. So I wish to get clarified is there any specific reason for mandating the 'node' option or if TOSCA spec compliance on this target identification based on the capability name or type will be supported in the future versions? Thanks, /Vaish
