Ok Svet;
It would certainly be a shame to have to write
if (provider.equals("GCE")) {
createSecurityGroupOneWay();
} else {
createSecurityGroupAnotherWay();
}
I guess I agree it's probably better, when the provider doesn't support a
concept, to avoid having an implementation that's "nearly but not quite",
otherwise you'd end up with lots of code like the above.
G
On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov <
svetoslav.ney...@cloudsoftcorp.com> wrote:
> > if that entity does not exist in GCE and
> > users won't be able to use the extension as in other clouds, is there
> > a real use case for it that justifies its implementation?
>
>
> My personal drive for the implementation is to reuse existing code
> managing the ingress access to (freshly spun up) nodes managed by jclouds.
> So it would work nicely for my case :).
>
> What still irks me about the proposal is the different scope of the
> location required to create the security group. It essentially means the
> implementation:
> * is incompatible with existing code out in the wild - it will break it
> when "securityGroupExtension.isPresent()" returns true, but the
> "createSecurityGroup" subsequently fails due to incorrect location argument
> * requires special casing in the code to pass the correct location scope
> - doing template.getLocation() will not do it for GCE which is what I use
> to pass as an argument to "createSecurityGroup"
>
> I took some time to think this through, that's why the delay in response,
> but don't see a nice solution to ^^^. That's why I'm backing off from the
> proposal. Still think it's been a useful discussion to have. Appreciate the
> involvement Ignasi and Geoff.
>
>
> Svet.
>
>
>
>
>
>
> > On 16.06.2017 г., at 10:01, Ignasi Barrera <n...@apache.org> wrote:
> >
> > I agree it is a grey area where we can only do our best.
> >
> > I'd say it is ok to skip egress rules and to only consider the rules
> > created by jclouds, that is something we could assume given the fact
> > that the concept of security groups does not exist in GCE, which
> > brings me to the question: if that entity does not exist in GCE and
> > users won't be able to use the extension as in other clouds, is there
> > a real use case for it that justifies its implementation?
> >
> > If the answer is yes, then let's move forward with your PoC. But I'd
> > like to make sure that we have identified all conflicting points (I
> > think we do). We can assume we're not going to support existing
> > resources not related to jclouds, but as Geoff mentioned too I think
> > we should try to support onboarding existing resources that were
> > created with previous versions of jclouds with the current
> > "inboundPorts" implementation.
> >
> > On 15 June 2017 at 15:59, Svetoslav Neykov
> > <svetoslav.ney...@cloudsoftcorp.com> wrote:
> >> The onboarding experience takes a best effort approach and never
> represents the cloud state exactly. It's a gray area of compromises.
> >> For example GCE supports ingress/egress; allow/deny rules. Of those
> only ingress+allow rules can be represented using IpPermission. What do we
> do about the rest? Just ignore, like we do for other clouds? Where do we
> draw the line and say that it doesn't fit the abstraction so it's not
> presented to the user? Taking this to the extreme we could decide that the
> only rules that fit the abstraction are those created by jclouds.
> >>
> >> It would be fairly easy to come up with a solution which transforms the
> firewall rules into a SecurityGroup (read SGs). What I'm worried about is
> updating onboarded SGs. It could result in some unexpected changes.
> >>
> >> Re the security group created for inboundPorts (or anything managed by
> jclouds) - it's something we expect and know what it looks like, how it's
> used, so should be straightforward to accommodate in the design. It's the
> unknown that's scary :).
> >>
> >> If we decide to represent existing firewall rules, here's how it could
> work:
> >> * skip egress; deny rules
> >> * rules having a list of target tags
> >> - create a security group per tag per network
> >> - share a rule between security groups - one per tag per network
> >> - deleting an IpPermission is removing the tag from the firewall
> rule (and deleting it if last one)
> >> - adding a rule is straightforward - create a permission with the
> same destination tag as the security group ID
> >> - adding a SG to a node is adding the ID for the SG to the tags of
> the node
> >> * rules having an "All instances in the network" target
> >> - group into a single SG (per network)
> >> - consider it a network SG and deny requests to attach it to a node
> >>
> >> I still think the "surprise factor" with the GCE implementation will be
> the requirement to pass a network scope location to the
> "createSecurityGroup" call. It's something that's different from the other
> implementations and means it can't "just work" with existing users. If we
> agree on a solution here, then I'm sure we'll make everything else work.
> >>
> >> Svet.
> >>
> >>
> >>
> >>
> >>> On 15.06.2017 г., at 13:26, Geoff Macartney <
> geoff.macart...@cloudsoftcorp.com> wrote:
> >>>
> >>> hi Ignasi,
> >>>
> >>> that's an interesting issue. Re. your last paragraph, that could
> certainly
> >>> be a good approach, but it still won't address the question of
> onboarding
> >>> existing infrastructure, right? i.e. where there are rules that
> weren't
> >>> written according to those conventions - or of upgrading to the new
> jclouds
> >>> version (with GCE security groups) to continue to manage infrastructure
> >>> that _was_ created and provisioned by jclouds.
> >>>
> >>> I'll certainly need to have a think about it. As you say we could
> refine
> >>> and be explicit about how the proposal would work - perhaps come up
> with a
> >>> use-case that illustrates the scenarios, then we could more easily see
> if
> >>> there are gaps to fill.
> >>>
> >>> Geoff
> >>>
> >>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <n...@apache.org> wrote:
> >>>
> >>>> I understand the motivation behind ignoring existing stuff, but I have
> >>>> some concerns.
> >>>>
> >>>> Onboarding existing cloud infrastructure is a valid use case for
> >>>> jclouds users, and that can already be done (limited by the bounds of
> >>>> the Compute abstraction) with the current jclouds implementation. It
> >>>> is fair to expect that if the SecurityGroupExtension is present, and
> >>>> there is a method called "listSecurityGroupsForNode" you should be
> >>>> able to onboard the firewall rules of your existing node. As a user,
> >>>> I'd have these expectations so I don't think we can discard the
> >>>> onboarding use case just because we've found it difficult to map in a
> >>>> consistent way.
> >>>>
> >>>> This said, if we went that path, I think we still need to refine the
> >>>> proposal, or at least be explicit and enumerate all affected points
> >>>> and how we plan to fill the gaps.
> >>>>
> >>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
> >>>> With the actual code, the node would be behind a couple firewall
> >>>> rules, but the security group extension won't be able to return any
> >>>> security group for the node. That is inconsistent, as something you
> >>>> created using the abstraction, cannot be retrieved with the same
> >>>> information using the abstraction extension. It will give users the
> >>>> impression that the node has no firewall rules applied.
> >>>>
> >>>> This case could be as simple as creating the firewall rules for the
> >>>> "inboundPorts" following our SG naming and tag convention. I just want
> >>>> to illustrate that the problem is not exclusive of the resources
> >>>> created "outside" jclouds.
> >>>>
> >>>> On 15 June 2017 at 11:09, Svetoslav Neykov
> >>>> <svetoslav.ney...@cloudsoftcorp.com> wrote:
> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
> >>>> above:
> >>>>>
> >>>>> That's right, tags are an important part of the convention and I
> didn't
> >>>> expand on that. Your description nicely captures the idea.
> >>>>> I believe the security groups will work across subnets in a single
> vpc
> >>>> network.
> >>>>>
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
> >>>> geoff.macart...@cloudsoftcorp.com> wrote:
> >>>>>>
> >>>>>> I like the sound of this proposal, I think it's certainly worth
> >>>>>> investigating.
> >>>>>>
> >>>>>> Ignasi those are good questions you've asked. Here's my thoughts:
> >>>>>>
> >>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and
> be
> >>>>>> explicit about that in documentation. What I mean here is that we
> don't
> >>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
> >>>>>> concepts. So if there are existing firewall rules they simply don't
> >>>> show up
> >>>>>> in the results of 'listSecurityGroups()",
> >>>> "listSecurityGroupsInLocation()"
> >>>>>> or "listSecurityGroupsForNode()". That is, not all rules
> participate
> >>>> in
> >>>>>> "security groups" from the jclouds point of view, only ones that
> jclouds
> >>>>>> added for security group purposes.
> >>>>>>
> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
> >>>> above:
> >>>>>>
> >>>>>> 1. What do you think about the idea of using tags to identify
> security
> >>>>>> groups - each security group would correspond to a tag. These tags
> would
> >>>>>> correspond to groupIds in the IPPermission object, i.e. you could
> >>>> create an
> >>>>>> IPPermission using a groupId [1] which would relate to a tag in the
> GCE
> >>>>>> platform. Then any nodes with that tag (= in that security group)
> would
> >>>> be
> >>>>>> permitted access.
> >>>>>> 2. Are there any restrictions/implications relating to subnets in
> your
> >>>>>> proposal, e.g. would you need to use them to achieve what you want,
> or
> >>>> can
> >>>>>> it work irrespective of any subnets on the network?
> >>>>>>
> >>>>>> Geoff
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <n...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> Thanks for starting this discussion Svet!
> >>>>>>>
> >>>>>>> It would be great to come up with a solution we are happy with.
> >>>>>>>
> >>>>>>> The main issue I see here is that the primary entity of the
> extension,
> >>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
> >>>>>>> the reasons we decided to remove the old implementation).
> >>>>>>>
> >>>>>>> I am OK applying conventions to implement the extension somehow,
> but
> >>>>>>> conventions are tricky when it comes to deal with existing stuff in
> >>>>>>> the provider. With this proposal, what are your plans to implement
> the
> >>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
> >>>>>>> location)" methods? There is also a method to list the security
> groups
> >>>>>>> from a node. Given that the ComputeService is able to "listNodes"
> that
> >>>>>>> weren't created by jclouds, in case of a returned Node that already
> >>>>>>> has firewall rules assigned but that don't fit in our naming
> >>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
> >>>>>>> behave?
> >>>>>>>
> >>>>>>> I.
> >>>>>>>
> >>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
> >>>>>>> <svetoslav.ney...@cloudsoftcorp.com> wrote:
> >>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
> >>>> GCE.
> >>>>>>> Looking at the documentation it seems that the combination of
> firewall
> >>>>>>> rules and node tags is flexible enough to allow us implement the
> >>>>>>> functionality.
> >>>>>>>> It's been tried before but the implementation's been removed (see
> >>>> [1]).
> >>>>>>> It's main drawback was that for each security group the code
> creates a
> >>>> new
> >>>>>>> network.
> >>>>>>>> Currently the biggest mismatch between the jclouds abstraction
> and the
> >>>>>>> GCE functionality is that firewall rules must be attached to a
> network.
> >>>>>>>> Here's my suggested approach:
> >>>>>>>> IpPermission roughly corresponds to a firewall rule
> >>>>>>>> SecurityGroup is just a collection of firewall rules (there's no
> cloud
> >>>>>>> resource that corresponds to it). The firewall rules of a security
> >>>> group
> >>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>.
> They
> >>>> all
> >>>>>>> belong to the same network.
> >>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope
> of
> >>>>>>> Network, returning a custom implementation of SecurityGroup which
> >>>> keeps a
> >>>>>>> reference to the network, so all IpPermission objects added
> >>>> subsequently
> >>>>>>> will be on it.
> >>>>>>>> While the suggested approach fits into the SecurityGroupExtension
> >>>>>>> interface it's different enough from the other implementations
> that it
> >>>>>>> might not be worth the trouble of implementing and supporting
> (even be
> >>>>>>> harmful as users might be surprised by the different behaviour).
> >>>>>>>> Would be interested in hearing other opinions on the approach.
> I've
> >>>>>>> created a Jira to track the effort at [2].
> >>>>>>>>
> >>>>>>>> Svet.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>
> >>>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>>>>>> <
> >>>>>>>
> >>>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>>>>>>>
> >>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
> >>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
>
>
