Re: [Spacewalk-devel] [PATCH] SSH Push Feature Proposal
On 24/04/13 14:38, Johannes Renner wrote: Can the logic you propose to be put to taskomatic be put to osa-dispatcher, to throttle the number of clients which get invited to rhn_check? Doesn't osad work by having the clients connect via XMPP _to_ the server? -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] [PATCH] SSH Push Feature Proposal
On Tue, Apr 30, 2013 at 04:08:54PM +0200, Duncan Mac-Vicar P. wrote: On 24/04/13 14:38, Johannes Renner wrote: Can the logic you propose to be put to taskomatic be put to osa-dispatcher, to throttle the number of clients which get invited to rhn_check? Doesn't osad work by having the clients connect via XMPP _to_ the server? Sure. But they just connect and then wait for osa-dispatcher to tell them to run rhn_check. -- Jan Pazdziora Principal Software Engineer, Satellite Engineering, Red Hat ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] [PATCH] SSH Push Feature Proposal
On 24/04/13 14:38, Johannes Renner wrote: Frankly, the first scenario does not sound that interesting to me. Access to and from DMZ is typically closed from/to all other networks as well and only opened in a very targeted fashion. The IT of that organization would still need to allow access _to_ the DMZ to sshd ports on those machines. You can always have Spacewalk Proxy in If you put a system in a public cloud with ssh access and want to manage it from your internal Spacewalk server you are already in that scenario. -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] [PATCH] SSH Push Feature Proposal
On 04/12/2013 01:24 PM, Jan Pazdziora wrote: Johannes, let me summarize some big picture impressions, without commenting on every detail. You propose to address two situations: 1) clients in DMZ that cannot reach the server, with server able to reach the clients; 2) overloading Spacewalk when multiple actions get (auto)scheduled for many clients and they get woken up by osa-dispatcher/jabberd/osad all at the same time. Frankly, the first scenario does not sound that interesting to me. Access to and from DMZ is typically closed from/to all other networks as well and only opened in a very targeted fashion. The IT of that organization would still need to allow access _to_ the DMZ to sshd ports on those machines. You can always have Spacewalk Proxy in DMZ2, having client talk to the proxy and that proxy to the Spacewalk, if your IT does not want to open the ports in the DMZ configuration directly. In both cases, the HTTP requests run by rhn_check / yum will end up on that Spacewalk server and if there is a way to compromise that server that way, it will happen. I would still need to check the patches in details to see how you solve the problem of the client IP addresses as seen by the Spacewalk server being 127.0.0.1 for all those requests which is hardly something you'd like to see in production. You propose new scheduling service in taskomatic to initiate the SSH Push for client ... but we already have such a functionality, it's osa-dispatcher. So either we should get rid of osa-dispatcher and do even the jabber/osad based notifications from taskomatic, or we should stick with osa-dispatcher and not create very similar solution in Java. I agree that duplicating such functionality is not a very good thing to do and that it would make more sense to do jabber as well as ssh based push notifications from within the same component of the software. On the other hand I wonder why osa-dispatcher has been written in Python in the first place? If there is something like taskomatic, shouldn't we put such functionality in there rather than having yet another separate component? The second scenario is however much more important and interesting -- yes, the server will get overloaded if you use many osad-enabled clients, and we had Spacewalk users complaining about this on the mailing list in the past. However, is the cure really to allow ssh access from server to the clients? How about clients that are behind NAT, roaming, or in general unavailable? I am not saying that SSH Push should be the cure for anything. Rather I would consider it as just another contact method that can optionally be used if it is suitable to anybody's needs. OSAD should IMO stay alive and be available as a valid alternative. It should be the customer's choice. The cure for the client complaints should rather be to make the existing software more robust and stable in the first place, i.e. mitigating issues with scalability. Taskomatic even offers some generic classes that can be used to do threading and queuing (TaskQueue, QueueDriver and QueueWorker), so isn't it actually a real option to port that osa-dispatcher functionality over there? I would assume that the majority of the Spacewalk (and downstream products') installations has server accessible from clients because otherwise things would currently not work. If you completely reverse the style of operation, it will cause the disruption in our users' setups. And still, clients for which you won't be able or willing to enable the SSH Push functionality will not get the improvement in timely actions that don't put the server to its knees. I don't think this would cause any disruptions. Actually it would even enable users to administrate systems using Spacewalk where it was not possible before. This means more users might even be attracted and also existing users might use Spacewalk to maintain more systems than before. You can even use ssh push to manage systems located in some public cloud overseas, as long as you can login using ssh from the server. I would very much love to see improvements to the second problem which could be used by all *existing* clients of Spacewalk, even those that are behind NAT, independed from the SSH Push feature. Can the logic you propose to be put to taskomatic be put to osa-dispatcher, to throttle the number of clients which get invited to rhn_check? Well, it's currently implemented in Java of course, and the first layer of throttling is only queuing and threading, as mentioned before: There is a (configurable) number of threads which specifies the number of concurrent ssh connections allowed at a time. If there is actions to be performed on more clients, the oldest ones in the schedule are treated first, while the rest will be queued and revisited as soon as there is free threads available. This was all done using existing classes from the taskomatic framework. The logic that was
Re: [Spacewalk-devel] [PATCH] SSH Push Feature Proposal
On Fri, Mar 08, 2013 at 05:14:35PM +0100, Johannes Renner wrote:
Hello,
Here is a proposal for a new Spacewalk feature that we would like to merge
into
master. An initial set of patches is attached, but let me explain 'quickly'
first:
It happens that there are systems that should be be managed, but they are
located
in a DMZ or some other subnet with restrictive access to the internal company
network. Such systems might technically not be allowed to call back to the
server
in order to ask for scheduled actions and such. One way of getting around this
problem is to open the connection from the server instead, and call back to
the
server ('rhn_check') using a secured tunnel.
Therefore we would in the first place propose to allow different contact
methods
to be configured for a registered system. Instead of the traditional default
Pull we would offer SSH Push via Tunnel as well as SSH Push (which is
the
same just without the tunnel). The contact method can be chosen on the
activation
key level, which means that all systems registered with a certain activation
key
will inherit the respective contact method (this is all in patches #1 and #2).
Further there is a job in taskomatic which runs once every minute to find
systems
that are configured for SSH push and with actions being scheduled to be
executed
in this moment. These will be contacted via
ssh -R high_port:server:443 client rhn_check
All scheduled actions will be fetched to the client, while the connection is
established by the server. To make this work it is however necessary to
reconfigure
the client in two places:
- /etc/hosts needs to contain server in the localhost line
- /etc/sysconfig/rhn/up2date needs to point to server:high_port instead of
only server
This reconfiguration is currently done during system registration. Since
registration of such a system needs to be done from the server as well (via
tunnel),
we provide a dedicated script, namely 'spacewalk-push-register', see patch #4.
Using this script, a client can be registered from the server's commandline:
spacewalk-push-register client path_to_bootstrap_script
Further we would want to prevent a system from not checking in, just in case
there
is no actions scheduled for a certain period of time. Therefore we should
contact
such systems before the respective threshold of inactiveness is reached
(default is
1 day). In order to prevent from many systems re-checking in at the same time
again,
randomly generated thresholds are used to determine if a system should
checkin or
not. The result is that all inactive systems will eventually checkin between
12 and
24 hours of inactiveness, but you do not know when exactly it will happen.
I can explain the implementation in more detail if you want, but you could
also run
the included unit test, which actually performs a simulation with 10
clients
to record their checkin times using the implemented algorithm.
In comparison to the existing method of pushing actions using osad, the
proposed
SSH Push should be more reliable in general and could therefore serve as a
valid
alternative (even without the tunneling). Further it scales better to a high
number
of client systems, since the number of threads opening connections to clients
at
the same time can be configured and therefore limited (default is 2). This is
however not the case with osad. All clients will be pinged and will call back
to
the server at the same time, which might cause a server to break down under
circumstances.
Things to be improved:
- Client registration: enable/disable a client for either SSH push with or
without
tunnel.
- UI integration for reconfiguring clients when the contact method is changed
for
a system.
- Push via osad could be another contact method or at least we should
somehow
integrate with the push status indication in the UI.
Your feedback and comments are more than welcome!
Johannes,
let me summarize some big picture impressions, without commenting on
every detail.
You propose to address two situations:
1) clients in DMZ that cannot reach the server, with server able to
reach the clients;
2) overloading Spacewalk when multiple actions get (auto)scheduled for
many clients and they get woken up by osa-dispatcher/jabberd/osad
all at the same time.
Frankly, the first scenario does not sound that interesting to me.
Access to and from DMZ is typically closed from/to all other networks
as well and only opened in a very targeted fashion. The IT of that
organization would still need to allow access _to_ the DMZ to sshd
ports on those machines. You can always have Spacewalk Proxy in
DMZ2, having client talk to the proxy and that proxy to the Spacewalk,
if your IT does not want to open the ports in the DMZ configuration
directly. In both cases, the HTTP requests run by rhn_check / yum will
end up on that Spacewalk server
