The onboarding experience takes a best effort approach and never represents the
cloud state exactly. It's a gray area of compromises.
For example GCE supports ingress/egress; allow/deny rules. Of those only
ingress+allow rules can be represented using IpPermission. What do we do about
the rest? Just ignore, like we do for other clouds? Where do we draw the line
and say that it doesn't fit the abstraction so it's not presented to the user?
Taking this to the extreme we could decide that the only rules that fit the
abstraction are those created by jclouds.
It would be fairly easy to come up with a solution which transforms the
firewall rules into a SecurityGroup (read SGs). What I'm worried about is
updating onboarded SGs. It could result in some unexpected changes.
Re the security group created for inboundPorts (or anything managed by jclouds)
- it's something we expect and know what it looks like, how it's used, so
should be straightforward to accommodate in the design. It's the unknown that's
scary :).
If we decide to represent existing firewall rules, here's how it could work:
* skip egress; deny rules
* rules having a list of target tags
- create a security group per tag per network
- share a rule between security groups - one per tag per network
- deleting an IpPermission is removing the tag from the firewall rule (and
deleting it if last one)
- adding a rule is straightforward - create a permission with the same
destination tag as the security group ID
- adding a SG to a node is adding the ID for the SG to the tags of the node
* rules having an "All instances in the network" target
- group into a single SG (per network)
- consider it a network SG and deny requests to attach it to a node
I still think the "surprise factor" with the GCE implementation will be the
requirement to pass a network scope location to the "createSecurityGroup" call.
It's something that's different from the other implementations and means it
can't "just work" with existing users. If we agree on a solution here, then I'm
sure we'll make everything else work.
Svet.
> On 15.06.2017 г., at 13:26, Geoff Macartney
> <geoff.macart...@cloudsoftcorp.com> wrote:
>
> hi Ignasi,
>
> that's an interesting issue. Re. your last paragraph, that could certainly
> be a good approach, but it still won't address the question of onboarding
> existing infrastructure, right? i.e. where there are rules that weren't
> written according to those conventions - or of upgrading to the new jclouds
> version (with GCE security groups) to continue to manage infrastructure
> that _was_ created and provisioned by jclouds.
>
> I'll certainly need to have a think about it. As you say we could refine
> and be explicit about how the proposal would work - perhaps come up with a
> use-case that illustrates the scenarios, then we could more easily see if
> there are gaps to fill.
>
> Geoff
>
> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <n...@apache.org> wrote:
>
>> I understand the motivation behind ignoring existing stuff, but I have
>> some concerns.
>>
>> Onboarding existing cloud infrastructure is a valid use case for
>> jclouds users, and that can already be done (limited by the bounds of
>> the Compute abstraction) with the current jclouds implementation. It
>> is fair to expect that if the SecurityGroupExtension is present, and
>> there is a method called "listSecurityGroupsForNode" you should be
>> able to onboard the firewall rules of your existing node. As a user,
>> I'd have these expectations so I don't think we can discard the
>> onboarding use case just because we've found it difficult to map in a
>> consistent way.
>>
>> This said, if we went that path, I think we still need to refine the
>> proposal, or at least be explicit and enumerate all affected points
>> and how we plan to fill the gaps.
>>
>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>> With the actual code, the node would be behind a couple firewall
>> rules, but the security group extension won't be able to return any
>> security group for the node. That is inconsistent, as something you
>> created using the abstraction, cannot be retrieved with the same
>> information using the abstraction extension. It will give users the
>> impression that the node has no firewall rules applied.
>>
>> This case could be as simple as creating the firewall rules for the
>> "inboundPorts" following our SG naming and tag convention. I just want
>> to illustrate that the problem is not exclusive of the resources
>> created "outside" jclouds.
>>
>> On 15 June 2017 at 11:09, Svetoslav Neykov
>> <svetoslav.ney...@cloudsoftcorp.com> wrote:
>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> above:
>>>
>>> That's right, tags are an important part of the convention and I didn't
>> expand on that. Your description nicely captures the idea.
>>> I believe the security groups will work across subnets in a single vpc
>> network.
>>>
>>> Svet.
>>>
>>>
>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>> geoff.macart...@cloudsoftcorp.com> wrote:
>>>>
>>>> I like the sound of this proposal, I think it's certainly worth
>>>> investigating.
>>>>
>>>> Ignasi those are good questions you've asked. Here's my thoughts:
>>>>
>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and be
>>>> explicit about that in documentation. What I mean here is that we don't
>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>> concepts. So if there are existing firewall rules they simply don't
>> show up
>>>> in the results of 'listSecurityGroups()",
>> "listSecurityGroupsInLocation()"
>>>> or "listSecurityGroupsForNode()". That is, not all rules participate
>> in
>>>> "security groups" from the jclouds point of view, only ones that jclouds
>>>> added for security group purposes.
>>>>
>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> above:
>>>>
>>>> 1. What do you think about the idea of using tags to identify security
>>>> groups - each security group would correspond to a tag. These tags would
>>>> correspond to groupIds in the IPPermission object, i.e. you could
>> create an
>>>> IPPermission using a groupId [1] which would relate to a tag in the GCE
>>>> platform. Then any nodes with that tag (= in that security group) would
>> be
>>>> permitted access.
>>>> 2. Are there any restrictions/implications relating to subnets in your
>>>> proposal, e.g. would you need to use them to achieve what you want, or
>> can
>>>> it work irrespective of any subnets on the network?
>>>>
>>>> Geoff
>>>>
>>>> [1]
>>>>
>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>>
>>>>
>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <n...@apache.org> wrote:
>>>>
>>>>> Thanks for starting this discussion Svet!
>>>>>
>>>>> It would be great to come up with a solution we are happy with.
>>>>>
>>>>> The main issue I see here is that the primary entity of the extension,
>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>> the reasons we decided to remove the old implementation).
>>>>>
>>>>> I am OK applying conventions to implement the extension somehow, but
>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>> the provider. With this proposal, what are your plans to implement the
>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>> location)" methods? There is also a method to list the security groups
>>>>> from a node. Given that the ComputeService is able to "listNodes" that
>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>> has firewall rules assigned but that don't fit in our naming
>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>> behave?
>>>>>
>>>>> I.
>>>>>
>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>> <svetoslav.ney...@cloudsoftcorp.com> wrote:
>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>> GCE.
>>>>> Looking at the documentation it seems that the combination of firewall
>>>>> rules and node tags is flexible enough to allow us implement the
>>>>> functionality.
>>>>>> It's been tried before but the implementation's been removed (see
>> [1]).
>>>>> It's main drawback was that for each security group the code creates a
>> new
>>>>> network.
>>>>>> Currently the biggest mismatch between the jclouds abstraction and the
>>>>> GCE functionality is that firewall rules must be attached to a network.
>>>>>> Here's my suggested approach:
>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>>>>> resource that corresponds to it). The firewall rules of a security
>> group
>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They
>> all
>>>>> belong to the same network.
>>>>>> They key bit: createSecurityGroup accepts a Location with a scope of
>>>>> Network, returning a custom implementation of SecurityGroup which
>> keeps a
>>>>> reference to the network, so all IpPermission objects added
>> subsequently
>>>>> will be on it.
>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>> interface it's different enough from the other implementations that it
>>>>> might not be worth the trouble of implementing and supporting (even be
>>>>> harmful as users might be surprised by the different behaviour).
>>>>>> Would be interested in hearing other opinions on the approach. I've
>>>>> created a Jira to track the effort at [2].
>>>>>>
>>>>>> Svet.
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>> <
>>>>>
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>
>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>>
>>>>>
>>>
>>
