hi Ignasi, that's an interesting issue. Re. your last paragraph, that could certainly be a good approach, but it still won't address the question of onboarding existing infrastructure, right? i.e. where there are rules that weren't written according to those conventions - or of upgrading to the new jclouds version (with GCE security groups) to continue to manage infrastructure that _was_ created and provisioned by jclouds.
I'll certainly need to have a think about it. As you say we could refine and be explicit about how the proposal would work - perhaps come up with a use-case that illustrates the scenarios, then we could more easily see if there are gaps to fill. Geoff On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <[email protected]> wrote: > I understand the motivation behind ignoring existing stuff, but I have > some concerns. > > Onboarding existing cloud infrastructure is a valid use case for > jclouds users, and that can already be done (limited by the bounds of > the Compute abstraction) with the current jclouds implementation. It > is fair to expect that if the SecurityGroupExtension is present, and > there is a method called "listSecurityGroupsForNode" you should be > able to onboard the firewall rules of your existing node. As a user, > I'd have these expectations so I don't think we can discard the > onboarding use case just because we've found it difficult to map in a > consistent way. > > This said, if we went that path, I think we still need to refine the > proposal, or at least be explicit and enumerate all affected points > and how we plan to fill the gaps. > > Say you create a node in jclouds with "options.inboundPorts(22, 80)". > With the actual code, the node would be behind a couple firewall > rules, but the security group extension won't be able to return any > security group for the node. That is inconsistent, as something you > created using the abstraction, cannot be retrieved with the same > information using the abstraction extension. It will give users the > impression that the node has no firewall rules applied. > > This case could be as simple as creating the firewall rules for the > "inboundPorts" following our SG naming and tag convention. I just want > to illustrate that the problem is not exclusive of the resources > created "outside" jclouds. > > On 15 June 2017 at 11:09, Svetoslav Neykov > <[email protected]> wrote: > >> Svet you don't mention GCE "tags" or the question of subnets in the > above: > > > > That's right, tags are an important part of the convention and I didn't > expand on that. Your description nicely captures the idea. > > I believe the security groups will work across subnets in a single vpc > network. > > > > Svet. > > > > > >> On 15.06.2017 г., at 11:57, Geoff Macartney < > [email protected]> wrote: > >> > >> I like the sound of this proposal, I think it's certainly worth > >> investigating. > >> > >> Ignasi those are good questions you've asked. Here's my thoughts: > >> > >> We can regard 'security groups' on GCE as a jclouds "overlay", and be > >> explicit about that in documentation. What I mean here is that we don't > >> need to do a "reverse mapping" of existing stuff back into jclouds > >> concepts. So if there are existing firewall rules they simply don't > show up > >> in the results of 'listSecurityGroups()", > "listSecurityGroupsInLocation()" > >> or "listSecurityGroupsForNode()". That is, not all rules participate > in > >> "security groups" from the jclouds point of view, only ones that jclouds > >> added for security group purposes. > >> > >> Svet you don't mention GCE "tags" or the question of subnets in the > above: > >> > >> 1. What do you think about the idea of using tags to identify security > >> groups - each security group would correspond to a tag. These tags would > >> correspond to groupIds in the IPPermission object, i.e. you could > create an > >> IPPermission using a groupId [1] which would relate to a tag in the GCE > >> platform. Then any nodes with that tag (= in that security group) would > be > >> permitted access. > >> 2. Are there any restrictions/implications relating to subnets in your > >> proposal, e.g. would you need to use them to achieve what you want, or > can > >> it work irrespective of any subnets on the network? > >> > >> Geoff > >> > >> [1] > >> > https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String) > >> > >> > >> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <[email protected]> wrote: > >> > >>> Thanks for starting this discussion Svet! > >>> > >>> It would be great to come up with a solution we are happy with. > >>> > >>> The main issue I see here is that the primary entity of the extension, > >>> the SecurityGroup, does not have a mapping in GCE (which was one of > >>> the reasons we decided to remove the old implementation). > >>> > >>> I am OK applying conventions to implement the extension somehow, but > >>> conventions are tricky when it comes to deal with existing stuff in > >>> the provider. With this proposal, what are your plans to implement the > >>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location > >>> location)" methods? There is also a method to list the security groups > >>> from a node. Given that the ComputeService is able to "listNodes" that > >>> weren't created by jclouds, in case of a returned Node that already > >>> has firewall rules assigned but that don't fit in our naming > >>> conventions, how would the "listSecurityGroupsForNode(String id)" > >>> behave? > >>> > >>> I. > >>> > >>> On 15 June 2017 at 09:03, Svetoslav Neykov > >>> <[email protected]> wrote: > >>>> I'd like to try implement the SecurityGroupExtension interface for > GCE. > >>> Looking at the documentation it seems that the combination of firewall > >>> rules and node tags is flexible enough to allow us implement the > >>> functionality. > >>>> It's been tried before but the implementation's been removed (see > [1]). > >>> It's main drawback was that for each security group the code creates a > new > >>> network. > >>>> Currently the biggest mismatch between the jclouds abstraction and the > >>> GCE functionality is that firewall rules must be attached to a network. > >>>> Here's my suggested approach: > >>>> IpPermission roughly corresponds to a firewall rule > >>>> SecurityGroup is just a collection of firewall rules (there's no cloud > >>> resource that corresponds to it). The firewall rules of a security > group > >>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They > all > >>> belong to the same network. > >>>> They key bit: createSecurityGroup accepts a Location with a scope of > >>> Network, returning a custom implementation of SecurityGroup which > keeps a > >>> reference to the network, so all IpPermission objects added > subsequently > >>> will be on it. > >>>> While the suggested approach fits into the SecurityGroupExtension > >>> interface it's different enough from the other implementations that it > >>> might not be worth the trouble of implementing and supporting (even be > >>> harmful as users might be surprised by the different behaviour). > >>>> Would be interested in hearing other opinions on the approach. I've > >>> created a Jira to track the effort at [2]. > >>>> > >>>> Svet. > >>>> > >>>> > >>>> [1] > >>> > https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c > >>> < > >>> > https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c > >>>> > >>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 < > >>> https://issues.apache.org/jira/browse/JCLOUDS-1309> > >>>> > >>> > > >
