Using the Location parameter to create a security group in a Network might do the trick, and perhaps we find the way to document or model it so it is easy to write portable code. We just need to think a bit more about it and explore all the implications and possible alternatives.
Here are some additional thoughts to take into account if we go that path: If we use the location parameter we should also think about its relationship with the collection returned by the ComputeService.listAvailableLocations method [1]. It returns the list of locations that can be used to spawn nodes, but I see it as a general method of the locations made available to be managed by the Compute abstraction. * Should we return a location with Network scope for each existing network? * Should we keep that method as-is and ask users to build a proper network-scoped Location object themselves? * How should we refine the semantics of that method so it is crystal clear to users how to use it and what to expect? I. [1] https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/ComputeService.java#L108-L118 P.S. Thanks to you Svet for bringing this discussion to the list! On 19 June 2017 at 10:24, Geoff Macartney <geoff.macart...@cloudsoftcorp.com> wrote: > Ok Svet; > > It would certainly be a shame to have to write > > if (provider.equals("GCE")) { > createSecurityGroupOneWay(); > } else { > createSecurityGroupAnotherWay(); > } > > I guess I agree it's probably better, when the provider doesn't support a > concept, to avoid having an implementation that's "nearly but not quite", > otherwise you'd end up with lots of code like the above. > > G > > On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov < > svetoslav.ney...@cloudsoftcorp.com> wrote: > >> > if that entity does not exist in GCE and >> > users won't be able to use the extension as in other clouds, is there >> > a real use case for it that justifies its implementation? >> >> >> My personal drive for the implementation is to reuse existing code >> managing the ingress access to (freshly spun up) nodes managed by jclouds. >> So it would work nicely for my case :). >> >> What still irks me about the proposal is the different scope of the >> location required to create the security group. It essentially means the >> implementation: >> * is incompatible with existing code out in the wild - it will break it >> when "securityGroupExtension.isPresent()" returns true, but the >> "createSecurityGroup" subsequently fails due to incorrect location argument >> * requires special casing in the code to pass the correct location scope >> - doing template.getLocation() will not do it for GCE which is what I use >> to pass as an argument to "createSecurityGroup" >> >> I took some time to think this through, that's why the delay in response, >> but don't see a nice solution to ^^^. That's why I'm backing off from the >> proposal. Still think it's been a useful discussion to have. Appreciate the >> involvement Ignasi and Geoff. >> >> >> Svet. >> >> >> >> >> >> >> > On 16.06.2017 г., at 10:01, Ignasi Barrera <n...@apache.org> wrote: >> > >> > I agree it is a grey area where we can only do our best. >> > >> > I'd say it is ok to skip egress rules and to only consider the rules >> > created by jclouds, that is something we could assume given the fact >> > that the concept of security groups does not exist in GCE, which >> > brings me to the question: if that entity does not exist in GCE and >> > users won't be able to use the extension as in other clouds, is there >> > a real use case for it that justifies its implementation? >> > >> > If the answer is yes, then let's move forward with your PoC. But I'd >> > like to make sure that we have identified all conflicting points (I >> > think we do). We can assume we're not going to support existing >> > resources not related to jclouds, but as Geoff mentioned too I think >> > we should try to support onboarding existing resources that were >> > created with previous versions of jclouds with the current >> > "inboundPorts" implementation. >> > >> > On 15 June 2017 at 15:59, Svetoslav Neykov >> > <svetoslav.ney...@cloudsoftcorp.com> wrote: >> >> The onboarding experience takes a best effort approach and never >> represents the cloud state exactly. It's a gray area of compromises. >> >> For example GCE supports ingress/egress; allow/deny rules. Of those >> only ingress+allow rules can be represented using IpPermission. What do we >> do about the rest? Just ignore, like we do for other clouds? Where do we >> draw the line and say that it doesn't fit the abstraction so it's not >> presented to the user? Taking this to the extreme we could decide that the >> only rules that fit the abstraction are those created by jclouds. >> >> >> >> It would be fairly easy to come up with a solution which transforms the >> firewall rules into a SecurityGroup (read SGs). What I'm worried about is >> updating onboarded SGs. It could result in some unexpected changes. >> >> >> >> Re the security group created for inboundPorts (or anything managed by >> jclouds) - it's something we expect and know what it looks like, how it's >> used, so should be straightforward to accommodate in the design. It's the >> unknown that's scary :). >> >> >> >> If we decide to represent existing firewall rules, here's how it could >> work: >> >> * skip egress; deny rules >> >> * rules having a list of target tags >> >> - create a security group per tag per network >> >> - share a rule between security groups - one per tag per network >> >> - deleting an IpPermission is removing the tag from the firewall >> rule (and deleting it if last one) >> >> - adding a rule is straightforward - create a permission with the >> same destination tag as the security group ID >> >> - adding a SG to a node is adding the ID for the SG to the tags of >> the node >> >> * rules having an "All instances in the network" target >> >> - group into a single SG (per network) >> >> - consider it a network SG and deny requests to attach it to a node >> >> >> >> I still think the "surprise factor" with the GCE implementation will be >> the requirement to pass a network scope location to the >> "createSecurityGroup" call. It's something that's different from the other >> implementations and means it can't "just work" with existing users. If we >> agree on a solution here, then I'm sure we'll make everything else work. >> >> >> >> Svet. >> >> >> >> >> >> >> >> >> >>> On 15.06.2017 г., at 13:26, Geoff Macartney < >> geoff.macart...@cloudsoftcorp.com> wrote: >> >>> >> >>> hi Ignasi, >> >>> >> >>> that's an interesting issue. Re. your last paragraph, that could >> certainly >> >>> be a good approach, but it still won't address the question of >> onboarding >> >>> existing infrastructure, right? i.e. where there are rules that >> weren't >> >>> written according to those conventions - or of upgrading to the new >> jclouds >> >>> version (with GCE security groups) to continue to manage infrastructure >> >>> that _was_ created and provisioned by jclouds. >> >>> >> >>> I'll certainly need to have a think about it. As you say we could >> refine >> >>> and be explicit about how the proposal would work - perhaps come up >> with a >> >>> use-case that illustrates the scenarios, then we could more easily see >> if >> >>> there are gaps to fill. >> >>> >> >>> Geoff >> >>> >> >>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <n...@apache.org> wrote: >> >>> >> >>>> I understand the motivation behind ignoring existing stuff, but I have >> >>>> some concerns. >> >>>> >> >>>> Onboarding existing cloud infrastructure is a valid use case for >> >>>> jclouds users, and that can already be done (limited by the bounds of >> >>>> the Compute abstraction) with the current jclouds implementation. It >> >>>> is fair to expect that if the SecurityGroupExtension is present, and >> >>>> there is a method called "listSecurityGroupsForNode" you should be >> >>>> able to onboard the firewall rules of your existing node. As a user, >> >>>> I'd have these expectations so I don't think we can discard the >> >>>> onboarding use case just because we've found it difficult to map in a >> >>>> consistent way. >> >>>> >> >>>> This said, if we went that path, I think we still need to refine the >> >>>> proposal, or at least be explicit and enumerate all affected points >> >>>> and how we plan to fill the gaps. >> >>>> >> >>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)". >> >>>> With the actual code, the node would be behind a couple firewall >> >>>> rules, but the security group extension won't be able to return any >> >>>> security group for the node. That is inconsistent, as something you >> >>>> created using the abstraction, cannot be retrieved with the same >> >>>> information using the abstraction extension. It will give users the >> >>>> impression that the node has no firewall rules applied. >> >>>> >> >>>> This case could be as simple as creating the firewall rules for the >> >>>> "inboundPorts" following our SG naming and tag convention. I just want >> >>>> to illustrate that the problem is not exclusive of the resources >> >>>> created "outside" jclouds. >> >>>> >> >>>> On 15 June 2017 at 11:09, Svetoslav Neykov >> >>>> <svetoslav.ney...@cloudsoftcorp.com> wrote: >> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the >> >>>> above: >> >>>>> >> >>>>> That's right, tags are an important part of the convention and I >> didn't >> >>>> expand on that. Your description nicely captures the idea. >> >>>>> I believe the security groups will work across subnets in a single >> vpc >> >>>> network. >> >>>>> >> >>>>> Svet. >> >>>>> >> >>>>> >> >>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney < >> >>>> geoff.macart...@cloudsoftcorp.com> wrote: >> >>>>>> >> >>>>>> I like the sound of this proposal, I think it's certainly worth >> >>>>>> investigating. >> >>>>>> >> >>>>>> Ignasi those are good questions you've asked. Here's my thoughts: >> >>>>>> >> >>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and >> be >> >>>>>> explicit about that in documentation. What I mean here is that we >> don't >> >>>>>> need to do a "reverse mapping" of existing stuff back into jclouds >> >>>>>> concepts. So if there are existing firewall rules they simply don't >> >>>> show up >> >>>>>> in the results of 'listSecurityGroups()", >> >>>> "listSecurityGroupsInLocation()" >> >>>>>> or "listSecurityGroupsForNode()". That is, not all rules >> participate >> >>>> in >> >>>>>> "security groups" from the jclouds point of view, only ones that >> jclouds >> >>>>>> added for security group purposes. >> >>>>>> >> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the >> >>>> above: >> >>>>>> >> >>>>>> 1. What do you think about the idea of using tags to identify >> security >> >>>>>> groups - each security group would correspond to a tag. These tags >> would >> >>>>>> correspond to groupIds in the IPPermission object, i.e. you could >> >>>> create an >> >>>>>> IPPermission using a groupId [1] which would relate to a tag in the >> GCE >> >>>>>> platform. Then any nodes with that tag (= in that security group) >> would >> >>>> be >> >>>>>> permitted access. >> >>>>>> 2. Are there any restrictions/implications relating to subnets in >> your >> >>>>>> proposal, e.g. would you need to use them to achieve what you want, >> or >> >>>> can >> >>>>>> it work irrespective of any subnets on the network? >> >>>>>> >> >>>>>> Geoff >> >>>>>> >> >>>>>> [1] >> >>>>>> >> >>>> >> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String) >> >>>>>> >> >>>>>> >> >>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <n...@apache.org> >> wrote: >> >>>>>> >> >>>>>>> Thanks for starting this discussion Svet! >> >>>>>>> >> >>>>>>> It would be great to come up with a solution we are happy with. >> >>>>>>> >> >>>>>>> The main issue I see here is that the primary entity of the >> extension, >> >>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of >> >>>>>>> the reasons we decided to remove the old implementation). >> >>>>>>> >> >>>>>>> I am OK applying conventions to implement the extension somehow, >> but >> >>>>>>> conventions are tricky when it comes to deal with existing stuff in >> >>>>>>> the provider. With this proposal, what are your plans to implement >> the >> >>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location >> >>>>>>> location)" methods? There is also a method to list the security >> groups >> >>>>>>> from a node. Given that the ComputeService is able to "listNodes" >> that >> >>>>>>> weren't created by jclouds, in case of a returned Node that already >> >>>>>>> has firewall rules assigned but that don't fit in our naming >> >>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)" >> >>>>>>> behave? >> >>>>>>> >> >>>>>>> I. >> >>>>>>> >> >>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov >> >>>>>>> <svetoslav.ney...@cloudsoftcorp.com> wrote: >> >>>>>>>> I'd like to try implement the SecurityGroupExtension interface for >> >>>> GCE. >> >>>>>>> Looking at the documentation it seems that the combination of >> firewall >> >>>>>>> rules and node tags is flexible enough to allow us implement the >> >>>>>>> functionality. >> >>>>>>>> It's been tried before but the implementation's been removed (see >> >>>> [1]). >> >>>>>>> It's main drawback was that for each security group the code >> creates a >> >>>> new >> >>>>>>> network. >> >>>>>>>> Currently the biggest mismatch between the jclouds abstraction >> and the >> >>>>>>> GCE functionality is that firewall rules must be attached to a >> network. >> >>>>>>>> Here's my suggested approach: >> >>>>>>>> IpPermission roughly corresponds to a firewall rule >> >>>>>>>> SecurityGroup is just a collection of firewall rules (there's no >> cloud >> >>>>>>> resource that corresponds to it). The firewall rules of a security >> >>>> group >> >>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. >> They >> >>>> all >> >>>>>>> belong to the same network. >> >>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope >> of >> >>>>>>> Network, returning a custom implementation of SecurityGroup which >> >>>> keeps a >> >>>>>>> reference to the network, so all IpPermission objects added >> >>>> subsequently >> >>>>>>> will be on it. >> >>>>>>>> While the suggested approach fits into the SecurityGroupExtension >> >>>>>>> interface it's different enough from the other implementations >> that it >> >>>>>>> might not be worth the trouble of implementing and supporting >> (even be >> >>>>>>> harmful as users might be surprised by the different behaviour). >> >>>>>>>> Would be interested in hearing other opinions on the approach. >> I've >> >>>>>>> created a Jira to track the effort at [2]. >> >>>>>>>> >> >>>>>>>> Svet. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> [1] >> >>>>>>> >> >>>> >> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c >> >>>>>>> < >> >>>>>>> >> >>>> >> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c >> >>>>>>>> >> >>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 < >> >>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >> >> >>
