Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

  a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

  b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

  c) ipa-server-install, ipa-server-uninstall, 
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers. 
They are causing us grief in API versioning and I don't see why it 
would be any different here.
However you must keep the version because of schema and data upgrade, so 
why not to execute update as one batch instead of doing config upgrade 
all the time, and then data upgrade only if required.


Some configuration upgrades, like adding new DNS related services, 
requires new schema, how we can handle this?

Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there 
was only one or two occurences of such bug in the past 3 years (I 
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have 
to be dumb like this.



Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously 
considering this the right optimization in a process that is 
magnitudes slower?
Ok the speed is not so important, but I still do not like the idea of 
executing the code which is not needed to be executed, because I know 
the version is the same as was before last restart, so nothing changed.




My personal opinion is, application should not try to fix itself every
restart.


One could say that application should not try to upgrade itself every 
restart, but here we are, doing it anyway...

I want to do upgrade only if needed not every restart.





In the past, I do not recall
ipa-upgradeconfig as being really fast, especially certmonger/Dogtag
related
updates were really slow thank to service restarts, etc.


Correct, but I was talking about configuration file updates, not
(re)starts, which have to always be done in ipactl anyway.




3)


* Prevent user to use ipa-upgradeconfig and ipa-ldap-updater


Even without arguments? Is ipactl now the only right place to
trigger manual
update?

Sorry, I will add more details there, if this is not clear.
ipa-upgrateconfig will be removed
ipa-ldap-updater will not be able to do overall update, you will need to
specify options and update file.


4)


Plugins are called from update files, using new directive
update-plugin:plugin-name


Why update-plugin and not just plugin? Do you expect other kinds
of plugins
to be called from update files in the future? (I certainly don't.)


I have no strong feelings on this one, but IMO it is always better to
have some
plan B if we choose to indeed implement some yet 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 10:55 Martin Kosek napsal(a):

On 03/03/2015 09:55 AM, Martin Basti wrote:

On 03/03/15 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
* ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what features
are actually available. A state file is tons better than a single version
number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?

at least ipa-dnskeysyncd service, requires updated schema for keys metadata.
This service is mandratory for DNS, so it is newly configured during upgrade.
Now it works because schema update is the first, so during configuration
upgrade we have actual schema.



Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?

The clever optimizations worked in past, but IPA grown and now contains
constraints/requirements which nobody expected.  What if we will need some
update which needs to execute time-consuming system check during every upgrade
in future?
User can always run the upgrade manually, with --skip-version-check, and then
configuration plugins will decide if the upgrade is needed.


I tend to agree with Martin, I would prefer to be on the safe side and not run
config upgrades every time, unless we are explicitly asked to or 

Re: [Freeipa-devel] [PATCHES 399-401] Allow multiple API instances

2015-03-03 Thread Tomas Babej


On 03/03/2015 04:01 PM, Martin Kosek wrote:

On 03/03/2015 03:49 PM, Jan Cholasta wrote:

Hi,

the attached patches provide an attempt to fix
https://fedorahosted.org/freeipa/ticket/3090.

Patch 401 serves as an example and modifies ipa-advise to use its own API
instance for Advice plugins.

Honza

Thanks. At least patches 399 and 400 look reasonable short for 4.2.

So with these patches, could we also get rid of temporary_ldap2_connection we
have in ipa-replica-install? Petr3 may have other examples he met in the past...

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


401 seems reasonable enough to me too, the bulk of the code is mostly 
just moving the code around and renaming variables.


Plus we have a very extensive (100%) coverage for the advise tool, so I 
wouldn't exclude it from the patchset.


Tomas

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Simo Sorce
On Tue, 2015-03-03 at 10:04 -0500, Rob Crittenden wrote:
 Martin Basti wrote:
  On 03/03/15 15:33, Martin Kosek wrote:
  On 03/03/2015 03:16 PM, Simo Sorce wrote:
  On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
  On 02/03/15 18:28, Martin Kosek wrote:
  On 03/02/2015 06:12 PM, Martin Basti wrote:
  On 02/03/15 15:43, Rob Crittenden wrote:
  Martin Basti wrote:
  ...
  But you haven't explained any case why LDAPI would fail. If LDAPI
  fails
  then you've got more serious problems that I'm not sure binding
  as DM is
  going to solve.
 
  The only case where DM would be handy IMHO is either some worst case
  scenario upgrade where 389-ds is up but not binding to LDAPI or
  if you
  want to allow running LDAP updates as non-root.
  I don't know cases when LDAPI would failed, except the case LDAPI is
  misconfigured by user, or disabled by user.
  Wasn't LDAPI needed for the DM password less upgrade so that
  upgrader could
  simply bind as root with EXTERNAL auth?
  We can do upgrade in both way, using LDAPI or using DM password,
  preferred is LDAPI.
  Question is, what is the use case for using DM password instead of
  LDAPI
  during upgrade.
  There is no use case for using the DM password.
  +1, so we will only use LDAPI and ditch DM password options and
  querying that
  we now have with ipa-ldap-updater?
 
  It is not big effort to keep both DM binding and LDAPI in code.  A
  user can
  always found som unexpected use case for LDAP update with DM
  password.
 
  On ipactl, would it be overkill if there is a tty to prompt the
  user to
  upgrade? In a non-container world it might be surprising to
  have an
  upgrade happen esp since upgrades take a while.
  In non-container enviroment, we can still use upgrade during RPM
  transaction.
 
  So you suggest  not to do update automaticaly, just write Error
  the IPA
  upgrade is required?
  People do all sorts of strange things. Installing the packages with
  --no-script isn't in the range of impossible. A prompt, and I'm not
  saying it's a great idea, is 2 lines of code.
 
  I guess it just makes me nervous.
  So lets summarize this:
  * DO upgrade if possible during RPM transaction
  Umm, I thought we want to get rid of running upgrade during RPM
  transaction. It
  is extremely difficult to debug upgrade stuck during RPM
  transaction, it also
  makes RPM upgrade run longer than needed. It also makes admins
  nervous when
  their rpm upgrade is suddenly waiting right before the end. I even
  see the
  fingers slowly reaching to CTRL+C combo... (You can see the
  consequences)
  People are used to have IPA upgraded and ready after RPM upgrade.
  They may be shocked if IPA services will be in shutdown state after RPM
  transaction.
  This is true, stopping IPA and requiring manual intervention is not ok.
  What is the plan then? Keep upgrades done during RPM transaction? Note
  that RPM
  transaction was already stuck several times because IPA, or rather DS,
  deadlocked.
 
  We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
  original plan was to do the upgrade during ipactl start, this would
  fix this
  ticket. Alternatively, should we remove the upgrade from RPM scriptlet
  and only
  call asynchronous systemctl restart ipa.service that would trigger the
  upgrade in separate process and log results in ipa.service?
 
  Martin
  The plan is do upgrade during RPM transaction if possible. If not then
  ipactl start, will show warning for user to do manual upgrade (Rob
  wanted it in this way, not doing auto upgrade by ipactl)
 
 Only if there is a tty which means no asking during package update,
 which I thought was the idea. I just think it is rather unexpected to
 update a package during a restart.
 
  So the fedup case is: RPM upgrade failed, ipactl start will detect
  version mismatch, show error and prompt user to run ipa-server-upgrade
 
 I'm beginning to have my own doubts about version, recognizing that
 there isn't exactly another obvious solution. Running the updates every
 time ipactl is run isn't great. The updates are not fast by any stretch,
 29s on one VM, and we need to log whenever an update is done. My
 ipaupgrade log is 48M from 20 updates. How many times does one run
 ipactl restart when diagnosing a problem?
 
 My biggest concern with version is who keeps count and where? This is
 particularly problematic in packaged servers where changes are made
 without rebasing (Fedora and RHEL). Somewhere the version would need to
 be bumped with each release? Or only when updates are added? Or only
 when someone remembers? It just seems fragile and prone to human error
 unless you have some automatic version incrementor that takes this into
 consideration.
 
 If fallible version or slow updates are the only option then I'd have to
 go with slow updates if only to avoid a lot of support issues. And I
 really hate the idea of updates during service restart.

Yet updates at start are the only option with container 

Re: [Freeipa-devel] [PATCHES 399-401] Allow multiple API instances

2015-03-03 Thread Martin Kosek
On 03/03/2015 04:09 PM, Jan Cholasta wrote:
 Dne 3.3.2015 v 16:04 Tomas Babej napsal(a):

 On 03/03/2015 04:01 PM, Martin Kosek wrote:
 On 03/03/2015 03:49 PM, Jan Cholasta wrote:
 Hi,

 the attached patches provide an attempt to fix
 https://fedorahosted.org/freeipa/ticket/3090.

 Patch 401 serves as an example and modifies ipa-advise to use its own
 API
 instance for Advice plugins.

 Honza
 Thanks. At least patches 399 and 400 look reasonable short for 4.2.

 So with these patches, could we also get rid of
 temporary_ldap2_connection we
 have in ipa-replica-install? Petr3 may have other examples he met in
 the past...
 
 I think we can. Shall I prepare a patch?

If it is reasonable simple, I would go for it. It would be another selling
point for your patches.

 

 Martin

 401 seems reasonable enough to me too, the bulk of the code is mostly
 just moving the code around and renaming variables.
 
 Right.
 

 Plus we have a very extensive (100%) coverage for the advise tool, so I
 wouldn't exclude it from the patchset.
 
 +1

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Rob Crittenden
Martin Basti wrote:
 On 03/03/15 15:33, Martin Kosek wrote:
 On 03/03/2015 03:16 PM, Simo Sorce wrote:
 On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
 On 03/02/2015 06:12 PM, Martin Basti wrote:
 On 02/03/15 15:43, Rob Crittenden wrote:
 Martin Basti wrote:
 ...
 But you haven't explained any case why LDAPI would fail. If LDAPI
 fails
 then you've got more serious problems that I'm not sure binding
 as DM is
 going to solve.

 The only case where DM would be handy IMHO is either some worst case
 scenario upgrade where 389-ds is up but not binding to LDAPI or
 if you
 want to allow running LDAP updates as non-root.
 I don't know cases when LDAPI would failed, except the case LDAPI is
 misconfigured by user, or disabled by user.
 Wasn't LDAPI needed for the DM password less upgrade so that
 upgrader could
 simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password,
 preferred is LDAPI.
 Question is, what is the use case for using DM password instead of
 LDAPI
 during upgrade.
 There is no use case for using the DM password.
 +1, so we will only use LDAPI and ditch DM password options and
 querying that
 we now have with ipa-ldap-updater?

 It is not big effort to keep both DM binding and LDAPI in code.  A
 user can
 always found som unexpected use case for LDAP update with DM
 password.

 On ipactl, would it be overkill if there is a tty to prompt the
 user to
 upgrade? In a non-container world it might be surprising to
 have an
 upgrade happen esp since upgrades take a while.
 In non-container enviroment, we can still use upgrade during RPM
 transaction.

 So you suggest  not to do update automaticaly, just write Error
 the IPA
 upgrade is required?
 People do all sorts of strange things. Installing the packages with
 --no-script isn't in the range of impossible. A prompt, and I'm not
 saying it's a great idea, is 2 lines of code.

 I guess it just makes me nervous.
 So lets summarize this:
 * DO upgrade if possible during RPM transaction
 Umm, I thought we want to get rid of running upgrade during RPM
 transaction. It
 is extremely difficult to debug upgrade stuck during RPM
 transaction, it also
 makes RPM upgrade run longer than needed. It also makes admins
 nervous when
 their rpm upgrade is suddenly waiting right before the end. I even
 see the
 fingers slowly reaching to CTRL+C combo... (You can see the
 consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM
 transaction.
 This is true, stopping IPA and requiring manual intervention is not ok.
 What is the plan then? Keep upgrades done during RPM transaction? Note
 that RPM
 transaction was already stuck several times because IPA, or rather DS,
 deadlocked.

 We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
 original plan was to do the upgrade during ipactl start, this would
 fix this
 ticket. Alternatively, should we remove the upgrade from RPM scriptlet
 and only
 call asynchronous systemctl restart ipa.service that would trigger the
 upgrade in separate process and log results in ipa.service?

 Martin
 The plan is do upgrade during RPM transaction if possible. If not then
 ipactl start, will show warning for user to do manual upgrade (Rob
 wanted it in this way, not doing auto upgrade by ipactl)

Only if there is a tty which means no asking during package update,
which I thought was the idea. I just think it is rather unexpected to
update a package during a restart.

 So the fedup case is: RPM upgrade failed, ipactl start will detect
 version mismatch, show error and prompt user to run ipa-server-upgrade

I'm beginning to have my own doubts about version, recognizing that
there isn't exactly another obvious solution. Running the updates every
time ipactl is run isn't great. The updates are not fast by any stretch,
29s on one VM, and we need to log whenever an update is done. My
ipaupgrade log is 48M from 20 updates. How many times does one run
ipactl restart when diagnosing a problem?

My biggest concern with version is who keeps count and where? This is
particularly problematic in packaged servers where changes are made
without rebasing (Fedora and RHEL). Somewhere the version would need to
be bumped with each release? Or only when updates are added? Or only
when someone remembers? It just seems fragile and prone to human error
unless you have some automatic version incrementor that takes this into
consideration.

If fallible version or slow updates are the only option then I'd have to
go with slow updates if only to avoid a lot of support issues. And I
really hate the idea of updates during service restart.

rob

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] [PATCHES 399-401] Allow multiple API instances

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 16:04 Tomas Babej napsal(a):


On 03/03/2015 04:01 PM, Martin Kosek wrote:

On 03/03/2015 03:49 PM, Jan Cholasta wrote:

Hi,

the attached patches provide an attempt to fix
https://fedorahosted.org/freeipa/ticket/3090.

Patch 401 serves as an example and modifies ipa-advise to use its own
API
instance for Advice plugins.

Honza

Thanks. At least patches 399 and 400 look reasonable short for 4.2.

So with these patches, could we also get rid of
temporary_ldap2_connection we
have in ipa-replica-install? Petr3 may have other examples he met in
the past...


I think we can. Shall I prepare a patch?



Martin


401 seems reasonable enough to me too, the bulk of the code is mostly
just moving the code around and renaming variables.


Right.



Plus we have a very extensive (100%) coverage for the advise tool, so I
wouldn't exclude it from the patchset.


+1



Tomas


--
Jan Cholasta

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 16:11, Simo Sorce wrote:

On Tue, 2015-03-03 at 10:04 -0500, Rob Crittenden wrote:

Martin Basti wrote:

On 03/03/15 15:33, Martin Kosek wrote:

On 03/03/2015 03:16 PM, Simo Sorce wrote:

On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:

On 02/03/15 18:28, Martin Kosek wrote:

On 03/02/2015 06:12 PM, Martin Basti wrote:

On 02/03/15 15:43, Rob Crittenden wrote:

Martin Basti wrote:

...

But you haven't explained any case why LDAPI would fail. If LDAPI
fails
then you've got more serious problems that I'm not sure binding
as DM is
going to solve.

The only case where DM would be handy IMHO is either some worst case
scenario upgrade where 389-ds is up but not binding to LDAPI or
if you
want to allow running LDAP updates as non-root.

I don't know cases when LDAPI would failed, except the case LDAPI is
misconfigured by user, or disabled by user.

Wasn't LDAPI needed for the DM password less upgrade so that
upgrader could
simply bind as root with EXTERNAL auth?

We can do upgrade in both way, using LDAPI or using DM password,
preferred is LDAPI.
Question is, what is the use case for using DM password instead of
LDAPI
during upgrade.

There is no use case for using the DM password.

+1, so we will only use LDAPI and ditch DM password options and
querying that
we now have with ipa-ldap-updater?


It is not big effort to keep both DM binding and LDAPI in code.  A
user can
always found som unexpected use case for LDAP update with DM
password.


On ipactl, would it be overkill if there is a tty to prompt the
user to
upgrade? In a non-container world it might be surprising to
have an
upgrade happen esp since upgrades take a while.

In non-container enviroment, we can still use upgrade during RPM
transaction.

So you suggest  not to do update automaticaly, just write Error
the IPA
upgrade is required?

People do all sorts of strange things. Installing the packages with
--no-script isn't in the range of impossible. A prompt, and I'm not
saying it's a great idea, is 2 lines of code.

I guess it just makes me nervous.

So lets summarize this:
* DO upgrade if possible during RPM transaction

Umm, I thought we want to get rid of running upgrade during RPM
transaction. It
is extremely difficult to debug upgrade stuck during RPM
transaction, it also
makes RPM upgrade run longer than needed. It also makes admins
nervous when
their rpm upgrade is suddenly waiting right before the end. I even
see the
fingers slowly reaching to CTRL+C combo... (You can see the
consequences)

People are used to have IPA upgraded and ready after RPM upgrade.
They may be shocked if IPA services will be in shutdown state after RPM
transaction.

This is true, stopping IPA and requiring manual intervention is not ok.

What is the plan then? Keep upgrades done during RPM transaction? Note
that RPM
transaction was already stuck several times because IPA, or rather DS,
deadlocked.

We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
original plan was to do the upgrade during ipactl start, this would
fix this
ticket. Alternatively, should we remove the upgrade from RPM scriptlet
and only
call asynchronous systemctl restart ipa.service that would trigger the
upgrade in separate process and log results in ipa.service?

Martin

The plan is do upgrade during RPM transaction if possible. If not then
ipactl start, will show warning for user to do manual upgrade (Rob
wanted it in this way, not doing auto upgrade by ipactl)

Only if there is a tty which means no asking during package update,
which I thought was the idea. I just think it is rather unexpected to
update a package during a restart.


So the fedup case is: RPM upgrade failed, ipactl start will detect
version mismatch, show error and prompt user to run ipa-server-upgrade

I'm beginning to have my own doubts about version, recognizing that
there isn't exactly another obvious solution. Running the updates every
time ipactl is run isn't great. The updates are not fast by any stretch,
29s on one VM, and we need to log whenever an update is done. My
ipaupgrade log is 48M from 20 updates. How many times does one run
ipactl restart when diagnosing a problem?

My biggest concern with version is who keeps count and where? This is
particularly problematic in packaged servers where changes are made
without rebasing (Fedora and RHEL). Somewhere the version would need to
be bumped with each release? Or only when updates are added? Or only
when someone remembers? It just seems fragile and prone to human error
unless you have some automatic version incrementor that takes this into
consideration.

If fallible version or slow updates are the only option then I'd have to
go with slow updates if only to avoid a lot of support issues. And I
really hate the idea of updates during service restart.
I do not like idea of auto upgrading during (re)start as well, it is not 
expected by user.
At least, upgrade takes 5-10 minutes, user may not to know the upgrade 
is happening.


We have 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 04:43 PM, Martin Basti wrote:
 On 03/03/15 16:11, Simo Sorce wrote:
 On Tue, 2015-03-03 at 10:04 -0500, Rob Crittenden wrote:
 Martin Basti wrote:
 On 03/03/15 15:33, Martin Kosek wrote:
 On 03/03/2015 03:16 PM, Simo Sorce wrote:
 On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
 On 03/02/2015 06:12 PM, Martin Basti wrote:
 On 02/03/15 15:43, Rob Crittenden wrote:
 Martin Basti wrote:
 ...
 But you haven't explained any case why LDAPI would fail. If LDAPI
 fails
 then you've got more serious problems that I'm not sure binding
 as DM is
 going to solve.

 The only case where DM would be handy IMHO is either some worst case
 scenario upgrade where 389-ds is up but not binding to LDAPI or
 if you
 want to allow running LDAP updates as non-root.
 I don't know cases when LDAPI would failed, except the case LDAPI is
 misconfigured by user, or disabled by user.
 Wasn't LDAPI needed for the DM password less upgrade so that
 upgrader could
 simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password,
 preferred is LDAPI.
 Question is, what is the use case for using DM password instead of
 LDAPI
 during upgrade.
 There is no use case for using the DM password.
 +1, so we will only use LDAPI and ditch DM password options and
 querying that
 we now have with ipa-ldap-updater?

 It is not big effort to keep both DM binding and LDAPI in code.  A
 user can
 always found som unexpected use case for LDAP update with DM
 password.

 On ipactl, would it be overkill if there is a tty to prompt the
 user to
 upgrade? In a non-container world it might be surprising to
 have an
 upgrade happen esp since upgrades take a while.
 In non-container enviroment, we can still use upgrade during RPM
 transaction.

 So you suggest  not to do update automaticaly, just write Error
 the IPA
 upgrade is required?
 People do all sorts of strange things. Installing the packages with
 --no-script isn't in the range of impossible. A prompt, and I'm not
 saying it's a great idea, is 2 lines of code.

 I guess it just makes me nervous.
 So lets summarize this:
 * DO upgrade if possible during RPM transaction
 Umm, I thought we want to get rid of running upgrade during RPM
 transaction. It
 is extremely difficult to debug upgrade stuck during RPM
 transaction, it also
 makes RPM upgrade run longer than needed. It also makes admins
 nervous when
 their rpm upgrade is suddenly waiting right before the end. I even
 see the
 fingers slowly reaching to CTRL+C combo... (You can see the
 consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM
 transaction.
 This is true, stopping IPA and requiring manual intervention is not ok.
 What is the plan then? Keep upgrades done during RPM transaction? Note
 that RPM
 transaction was already stuck several times because IPA, or rather DS,
 deadlocked.

 We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
 original plan was to do the upgrade during ipactl start, this would
 fix this
 ticket. Alternatively, should we remove the upgrade from RPM scriptlet
 and only
 call asynchronous systemctl restart ipa.service that would trigger the
 upgrade in separate process and log results in ipa.service?

 Martin
 The plan is do upgrade during RPM transaction if possible. If not then
 ipactl start, will show warning for user to do manual upgrade (Rob
 wanted it in this way, not doing auto upgrade by ipactl)
 Only if there is a tty which means no asking during package update,
 which I thought was the idea. I just think it is rather unexpected to
 update a package during a restart.

 So the fedup case is: RPM upgrade failed, ipactl start will detect
 version mismatch, show error and prompt user to run ipa-server-upgrade
 I'm beginning to have my own doubts about version, recognizing that
 there isn't exactly another obvious solution. Running the updates every
 time ipactl is run isn't great. The updates are not fast by any stretch,
 29s on one VM, and we need to log whenever an update is done. My
 ipaupgrade log is 48M from 20 updates. How many times does one run
 ipactl restart when diagnosing a problem?

 My biggest concern with version is who keeps count and where? This is
 particularly problematic in packaged servers where changes are made
 without rebasing (Fedora and RHEL). Somewhere the version would need to
 be bumped with each release? Or only when updates are added? Or only
 when someone remembers? It just seems fragile and prone to human error
 unless you have some automatic version incrementor that takes this into
 consideration.

 If fallible version or slow updates are the only option then I'd have to
 go with slow updates if only to avoid a lot of support issues. And I
 really hate the idea of updates during service restart.
 I do not like idea of auto upgrading during (re)start as well, it is 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 16:47, Martin Kosek wrote:

On 03/03/2015 04:43 PM, Martin Basti wrote:

On 03/03/15 16:11, Simo Sorce wrote:

On Tue, 2015-03-03 at 10:04 -0500, Rob Crittenden wrote:

Martin Basti wrote:

On 03/03/15 15:33, Martin Kosek wrote:

On 03/03/2015 03:16 PM, Simo Sorce wrote:

On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:

On 02/03/15 18:28, Martin Kosek wrote:

On 03/02/2015 06:12 PM, Martin Basti wrote:

On 02/03/15 15:43, Rob Crittenden wrote:

Martin Basti wrote:

...

But you haven't explained any case why LDAPI would fail. If LDAPI
fails
then you've got more serious problems that I'm not sure binding
as DM is
going to solve.

The only case where DM would be handy IMHO is either some worst case
scenario upgrade where 389-ds is up but not binding to LDAPI or
if you
want to allow running LDAP updates as non-root.

I don't know cases when LDAPI would failed, except the case LDAPI is
misconfigured by user, or disabled by user.

Wasn't LDAPI needed for the DM password less upgrade so that
upgrader could
simply bind as root with EXTERNAL auth?

We can do upgrade in both way, using LDAPI or using DM password,
preferred is LDAPI.
Question is, what is the use case for using DM password instead of
LDAPI
during upgrade.

There is no use case for using the DM password.

+1, so we will only use LDAPI and ditch DM password options and
querying that
we now have with ipa-ldap-updater?


It is not big effort to keep both DM binding and LDAPI in code.  A
user can
always found som unexpected use case for LDAP update with DM
password.


On ipactl, would it be overkill if there is a tty to prompt the
user to
upgrade? In a non-container world it might be surprising to
have an
upgrade happen esp since upgrades take a while.

In non-container enviroment, we can still use upgrade during RPM
transaction.

So you suggest  not to do update automaticaly, just write Error
the IPA
upgrade is required?

People do all sorts of strange things. Installing the packages with
--no-script isn't in the range of impossible. A prompt, and I'm not
saying it's a great idea, is 2 lines of code.

I guess it just makes me nervous.

So lets summarize this:
* DO upgrade if possible during RPM transaction

Umm, I thought we want to get rid of running upgrade during RPM
transaction. It
is extremely difficult to debug upgrade stuck during RPM
transaction, it also
makes RPM upgrade run longer than needed. It also makes admins
nervous when
their rpm upgrade is suddenly waiting right before the end. I even
see the
fingers slowly reaching to CTRL+C combo... (You can see the
consequences)

People are used to have IPA upgraded and ready after RPM upgrade.
They may be shocked if IPA services will be in shutdown state after RPM
transaction.

This is true, stopping IPA and requiring manual intervention is not ok.

What is the plan then? Keep upgrades done during RPM transaction? Note
that RPM
transaction was already stuck several times because IPA, or rather DS,
deadlocked.

We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
original plan was to do the upgrade during ipactl start, this would
fix this
ticket. Alternatively, should we remove the upgrade from RPM scriptlet
and only
call asynchronous systemctl restart ipa.service that would trigger the
upgrade in separate process and log results in ipa.service?

Martin

The plan is do upgrade during RPM transaction if possible. If not then
ipactl start, will show warning for user to do manual upgrade (Rob
wanted it in this way, not doing auto upgrade by ipactl)

Only if there is a tty which means no asking during package update,
which I thought was the idea. I just think it is rather unexpected to
update a package during a restart.


So the fedup case is: RPM upgrade failed, ipactl start will detect
version mismatch, show error and prompt user to run ipa-server-upgrade

I'm beginning to have my own doubts about version, recognizing that
there isn't exactly another obvious solution. Running the updates every
time ipactl is run isn't great. The updates are not fast by any stretch,
29s on one VM, and we need to log whenever an update is done. My
ipaupgrade log is 48M from 20 updates. How many times does one run
ipactl restart when diagnosing a problem?

My biggest concern with version is who keeps count and where? This is
particularly problematic in packaged servers where changes are made
without rebasing (Fedora and RHEL). Somewhere the version would need to
be bumped with each release? Or only when updates are added? Or only
when someone remembers? It just seems fragile and prone to human error
unless you have some automatic version incrementor that takes this into
consideration.

If fallible version or slow updates are the only option then I'd have to
go with slow updates if only to avoid a lot of support issues. And I
really hate the idea of updates during service restart.

I do not like idea of auto upgrading during (re)start as well, it is not
expected by user.
At least, 

Re: [Freeipa-devel] [PATCHES 399-401] Allow multiple API instances

2015-03-03 Thread Martin Kosek
On 03/03/2015 03:49 PM, Jan Cholasta wrote:
 Hi,
 
 the attached patches provide an attempt to fix
 https://fedorahosted.org/freeipa/ticket/3090.
 
 Patch 401 serves as an example and modifies ipa-advise to use its own API
 instance for Advice plugins.
 
 Honza

Thanks. At least patches 399 and 400 look reasonable short for 4.2.

So with these patches, could we also get rid of temporary_ldap2_connection we
have in ipa-replica-install? Petr3 may have other examples he met in the past...

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 16:10, Martin Kosek wrote:

On 03/03/2015 04:04 PM, Rob Crittenden wrote:

Martin Basti wrote:

On 03/03/15 15:33, Martin Kosek wrote:

On 03/03/2015 03:16 PM, Simo Sorce wrote:

On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:

On 02/03/15 18:28, Martin Kosek wrote:

On 03/02/2015 06:12 PM, Martin Basti wrote:

On 02/03/15 15:43, Rob Crittenden wrote:

Martin Basti wrote:

...

But you haven't explained any case why LDAPI would fail. If LDAPI
fails
then you've got more serious problems that I'm not sure binding
as DM is
going to solve.

The only case where DM would be handy IMHO is either some worst case
scenario upgrade where 389-ds is up but not binding to LDAPI or
if you
want to allow running LDAP updates as non-root.

I don't know cases when LDAPI would failed, except the case LDAPI is
misconfigured by user, or disabled by user.

Wasn't LDAPI needed for the DM password less upgrade so that
upgrader could
simply bind as root with EXTERNAL auth?

We can do upgrade in both way, using LDAPI or using DM password,
preferred is LDAPI.
Question is, what is the use case for using DM password instead of
LDAPI
during upgrade.

There is no use case for using the DM password.

+1, so we will only use LDAPI and ditch DM password options and
querying that
we now have with ipa-ldap-updater?


It is not big effort to keep both DM binding and LDAPI in code.  A
user can
always found som unexpected use case for LDAP update with DM
password.


On ipactl, would it be overkill if there is a tty to prompt the
user to
upgrade? In a non-container world it might be surprising to
have an
upgrade happen esp since upgrades take a while.

In non-container enviroment, we can still use upgrade during RPM
transaction.

So you suggest  not to do update automaticaly, just write Error
the IPA
upgrade is required?

People do all sorts of strange things. Installing the packages with
--no-script isn't in the range of impossible. A prompt, and I'm not
saying it's a great idea, is 2 lines of code.

I guess it just makes me nervous.

So lets summarize this:
* DO upgrade if possible during RPM transaction

Umm, I thought we want to get rid of running upgrade during RPM
transaction. It
is extremely difficult to debug upgrade stuck during RPM
transaction, it also
makes RPM upgrade run longer than needed. It also makes admins
nervous when
their rpm upgrade is suddenly waiting right before the end. I even
see the
fingers slowly reaching to CTRL+C combo... (You can see the
consequences)

People are used to have IPA upgraded and ready after RPM upgrade.
They may be shocked if IPA services will be in shutdown state after RPM
transaction.

This is true, stopping IPA and requiring manual intervention is not ok.

What is the plan then? Keep upgrades done during RPM transaction? Note
that RPM
transaction was already stuck several times because IPA, or rather DS,
deadlocked.

We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
original plan was to do the upgrade during ipactl start, this would
fix this
ticket. Alternatively, should we remove the upgrade from RPM scriptlet
and only
call asynchronous systemctl restart ipa.service that would trigger the
upgrade in separate process and log results in ipa.service?

Martin

The plan is do upgrade during RPM transaction if possible. If not then
ipactl start, will show warning for user to do manual upgrade (Rob
wanted it in this way, not doing auto upgrade by ipactl)

Only if there is a tty which means no asking during package update,
which I thought was the idea. I just think it is rather unexpected to
update a package during a restart.


So the fedup case is: RPM upgrade failed, ipactl start will detect
version mismatch, show error and prompt user to run ipa-server-upgrade

I'm beginning to have my own doubts about version, recognizing that
there isn't exactly another obvious solution. Running the updates every
time ipactl is run isn't great. The updates are not fast by any stretch,
29s on one VM, and we need to log whenever an update is done. My
ipaupgrade log is 48M from 20 updates. How many times does one run
ipactl restart when diagnosing a problem?

My biggest concern with version is who keeps count and where? This is
particularly problematic in packaged servers where changes are made
without rebasing (Fedora and RHEL). Somewhere the version would need to
be bumped with each release? Or only when updates are added? Or only
when someone remembers? It just seems fragile and prone to human error
unless you have some automatic version incrementor that takes this into
consideration.

If fallible version or slow updates are the only option then I'd have to
go with slow updates if only to avoid a lot of support issues. And I
really hate the idea of updates during service restart.

rob

Storing the version is something we just have to do, we need it for missing
upgrade detection in FedUp case. We also need for container use case, to make
sure that we can check whether we 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Petr Spacek
On 3.3.2015 10:58, Martin Kosek wrote:
 On 03/03/2015 09:36 AM, Petr Spacek wrote:
 On 3.3.2015 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
 ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
* ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.

 Because there is no exact mapping between version number and what features 
 are
 actually available. A state file is tons better than a single version 
 number.


 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?

 This does not sound right. Could you be more specific?

 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.

 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?

 I very much agree with Honza. We should always start with something
 stupidly-simply and enhance it later, when it is clear if it is really 
 necessary.

 Do not over-engineer it from the very beginning.
 
 I completely agree with starting stupid and simply and improving in time.
 However, are we sure that what Honza proposed is the simple and stupid way?
 
 Doing config upgrade only when needed and thus not depending on the efficiency
 and idempotency of the config upgraders seems to me as *the* stupid and simple
 way for upgrade refactoring.

Maybe I'm missing something but

if not state.get('dns_forward_zones_supported'):
# upgrade to forward zones here

seems less error-prone than

if version  4.0:
# 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 11:04 Petr Spacek napsal(a):

On 3.3.2015 10:58, Martin Kosek wrote:

On 03/03/2015 09:36 AM, Petr Spacek wrote:

On 3.3.2015 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
* ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what features are
actually available. A state file is tons better than a single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?


Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?


I very much agree with Honza. We should always start with something
stupidly-simply and enhance it later, when it is clear if it is really 
necessary.

Do not over-engineer it from the very beginning.


I completely agree with starting stupid and simply and improving in time.
However, are we sure that what Honza proposed is the simple and stupid way?

Doing config upgrade only when needed and thus not depending on the efficiency
and idempotency of the config upgraders seems to me as *the* stupid and simple
way for upgrade refactoring.


Maybe I'm missing something but

if not state.get('dns_forward_zones_supported'):
# upgrade to forward zones here

seems less error-prone than

if version  4.0:
# upgrade to forward 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 11:06 AM, Jan Cholasta wrote:
 Dne 3.3.2015 v 11:04 Petr Spacek napsal(a):
 On 3.3.2015 10:58, Martin Kosek wrote:
 On 03/03/2015 09:36 AM, Petr Spacek wrote:
 On 3.3.2015 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

b) ipa-server-install [install], ipa-server-install 
 uninstall,
 ipa-server-install upgrade

c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
 * ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one 
 should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.

 Because there is no exact mapping between version number and what features
 are
 actually available. A state file is tons better than a single version 
 number.


 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?

 This does not sound right. Could you be more specific?

 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not 
 work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration 
 which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.

 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?

 I very much agree with Honza. We should always start with something
 stupidly-simply and enhance it later, when it is clear if it is really
 necessary.

 Do not over-engineer it from the very beginning.

 I completely agree with starting stupid and simply and improving in time.
 However, are we sure that what Honza proposed is the simple and stupid way?

 Doing config upgrade only when needed and thus not depending on the 
 efficiency
 and idempotency of the config upgraders seems to me as *the* stupid and 
 simple
 way for upgrade refactoring.

 Maybe I'm missing something but

 if not 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 11:04, Petr Spacek wrote:

On 3.3.2015 10:58, Martin Kosek wrote:

On 03/03/2015 09:36 AM, Petr Spacek wrote:

On 3.3.2015 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade

Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
* ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.

I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.

In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.

Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.

Because there is no exact mapping between version number and what features are
actually available. A state file is tons better than a single version number.


Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?

This does not sound right. Could you be more specific?


Running schema upgrade every time?

What if a service changes in a way, the IPA configuration will not work?

Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok

The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.

Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.

How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.

Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?

I very much agree with Honza. We should always start with something
stupidly-simply and enhance it later, when it is clear if it is really 
necessary.

Do not over-engineer it from the very beginning.

I completely agree with starting stupid and simply and improving in time.
However, are we sure that what Honza proposed is the simple and stupid way?

Doing config upgrade only when needed and thus not depending on the efficiency
and idempotency of the config upgraders seems to me as *the* stupid and simple
way for upgrade refactoring.

Maybe I'm missing something but

if not state.get('dns_forward_zones_supported'):
# upgrade to forward zones here

seems less error-prone than

if version  4.0:
# upgrade to forward zones here
# make me 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 09:36 AM, Petr Spacek wrote:
 On 3.3.2015 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
 ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
* ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.

 Because there is no exact mapping between version number and what features 
 are
 actually available. A state file is tons better than a single version number.


 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?

 This does not sound right. Could you be more specific?

 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.

 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?
 
 I very much agree with Honza. We should always start with something
 stupidly-simply and enhance it later, when it is clear if it is really 
 necessary.
 
 Do not over-engineer it from the very beginning.

I completely agree with starting stupid and simply and improving in time.
However, are we sure that what Honza proposed is the simple and stupid way?

Doing config upgrade only when needed and thus not depending on the efficiency
and idempotency of the config upgraders seems to me as *the* stupid and simple
way for upgrade refactoring.

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 12:08 Martin Kosek napsal(a):

On 03/03/2015 11:06 AM, Jan Cholasta wrote:

Dne 3.3.2015 v 11:04 Petr Spacek napsal(a):

On 3.3.2015 10:58, Martin Kosek wrote:

On 03/03/2015 09:36 AM, Petr Spacek wrote:

On 3.3.2015 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
 * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what features
are
actually available. A state file is tons better than a single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?


Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?


I very much agree with Honza. We should always start with something
stupidly-simply and enhance it later, when it is clear if it is really
necessary.

Do not over-engineer it from the very beginning.


I completely agree with starting stupid and simply and improving in time.
However, are we sure that what Honza proposed is the simple and stupid way?

Doing config upgrade only when needed and thus not depending on the efficiency
and idempotency of the config upgraders seems to me as *the* stupid and simple
way for upgrade refactoring.


Maybe I'm missing something but

if not state.get('dns_forward_zones_supported'):
 # upgrade to forward 

Re: [Freeipa-devel] [PATCH 0001] ipa-client-install: attempt to get host TGT several times before aborting client installation

2015-03-03 Thread Rob Crittenden
Martin Babinsky wrote:
 On 03/02/2015 04:28 PM, Rob Crittenden wrote:
 Petr Vobornik wrote:
 On 01/12/2015 05:45 PM, Martin Babinsky wrote:
 related to ticket https://fedorahosted.org/freeipa/ticket/4808

 this patch seems to be a bit forgotten.

 It works, looks fine.

 One minor issue: trailing whitespaces in the man page.

 I also wonder if it shouldn't be used in other tools which call kinit
 with keytab:
 * ipa-client-automount:434
 * ipa-client-install:2591 (this usage should be fine since it's used for
 server installation)
 * dcerpc.py:545
 * rpcserver.py: 971, 981 (armor for web ui forms base auth)

 Most importantly the ipa-client-automount because it's called from
 ipa-client-install (if location is specified) and therefore it might
 fail during client installation.

 Or also, kinit call with admin creadentials worked for the user but I
 wonder if it was just a coincidence and may break under slightly
 different but similar conditions.

 I think that's a fine idea. In fact there is already a function that
 could be extended, kinit_hostprincipal().

 rob

 
 So in principle we could add multiple TGT retries to
 kinit_hostprincipal() and then plug this function to all the places
 Petr mentioned in order to provide this functionality each time TGT is
 requested using keytab.
 
 Do I understand it correctly?
 

Honestly I think I'd only do the retries on client installation.  I
don't know that the other uses would really benefit or need this.

But this is an opportunity to consolidate some code, so I guess the
approach I'd take is to add an option to kinit_hostprincipal of
retries=0 so that only a single kinit is done. The client installers
would pass in some value.

This change is quite a bit more invasive but it's also early in the
release cycle so the risk will be spread out.

rob

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Simo Sorce
On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
  On 03/02/2015 06:12 PM, Martin Basti wrote:
  On 02/03/15 15:43, Rob Crittenden wrote:
  Martin Basti wrote:
  ...
  But you haven't explained any case why LDAPI would fail. If LDAPI fails
  then you've got more serious problems that I'm not sure binding as DM is
  going to solve.
 
  The only case where DM would be handy IMHO is either some worst case
  scenario upgrade where 389-ds is up but not binding to LDAPI or if you
  want to allow running LDAP updates as non-root.
  I don't know cases when LDAPI would failed, except the case LDAPI is
  misconfigured by user, or disabled by user.
  Wasn't LDAPI needed for the DM password less upgrade so that upgrader could
  simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password, 
 preferred is LDAPI.
 Question is, what is the use case for using DM password instead of LDAPI 
 during upgrade.

There is no use case for using the DM password.

  It is not big effort to keep both DM binding and LDAPI in code.  A user can
  always found som unexpected use case for LDAP update with DM password.
 
  On ipactl, would it be overkill if there is a tty to prompt the user to
  upgrade? In a non-container world it might be surprising to have an
  upgrade happen esp since upgrades take a while.
  In non-container enviroment, we can still use upgrade during RPM
  transaction.
 
  So you suggest  not to do update automaticaly, just write Error the IPA
  upgrade is required?
  People do all sorts of strange things. Installing the packages with
  --no-script isn't in the range of impossible. A prompt, and I'm not
  saying it's a great idea, is 2 lines of code.
 
  I guess it just makes me nervous.
  So lets summarize this:
  * DO upgrade if possible during RPM transaction
  Umm, I thought we want to get rid of running upgrade during RPM 
  transaction. It
  is extremely difficult to debug upgrade stuck during RPM transaction, it 
  also
  makes RPM upgrade run longer than needed. It also makes admins nervous when
  their rpm upgrade is suddenly waiting right before the end. I even see the
  fingers slowly reaching to CTRL+C combo... (You can see the consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM 
 transaction.

This is true, stopping IPA and requiring manual intervention is not ok.

 I have no more objections.
 
 
  * ipactl will NOT run upgrade, just print Error: 'please upgrade '
  * User has to run ipa-server-upgrade manually
 
  Does I understand it correctly?
  With --skip-version-check what sorts of problems can we foresee? I
  assume a big warning will be added to at least the man page, if not
  the cli?
  For this big warning everywhere.
  The main problem is try to run older IPA with newer data. In containers
  the problem is to run different platform specific versions which differ
  in functionality/bugfixes etc..
  Ok, pretty much the things I was thinking as well. A scary warning
  definitely seems warranted.
 
  Where does platform come from? I'm wondering how Debian will handle 
  this.
  platform is derived from ipaplatform file which is used with the
  particular build. Debian should have own platform file.
  Ok, I'd add that detail to the design.
 
  rob
 
 
 


-- 
Simo Sorce * Red Hat, Inc * New York

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Petr Spacek
On 3.3.2015 11:01, Jan Cholasta wrote:
 I would very much prefer to do it the other way around, so that most bugs in
 the code are caught early, instead of hidden behind the version comparison.

+1

-- 
Petr^2 Spacek

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] [PATCH 0001] ipa-client-install: attempt to get host TGT several times before aborting client installation

2015-03-03 Thread Martin Babinsky

On 03/03/2015 02:32 PM, Rob Crittenden wrote:

Martin Babinsky wrote:

On 03/02/2015 04:28 PM, Rob Crittenden wrote:

Petr Vobornik wrote:

On 01/12/2015 05:45 PM, Martin Babinsky wrote:

related to ticket https://fedorahosted.org/freeipa/ticket/4808


this patch seems to be a bit forgotten.

It works, looks fine.

One minor issue: trailing whitespaces in the man page.

I also wonder if it shouldn't be used in other tools which call kinit
with keytab:
* ipa-client-automount:434
* ipa-client-install:2591 (this usage should be fine since it's used for
server installation)
* dcerpc.py:545
* rpcserver.py: 971, 981 (armor for web ui forms base auth)

Most importantly the ipa-client-automount because it's called from
ipa-client-install (if location is specified) and therefore it might
fail during client installation.

Or also, kinit call with admin creadentials worked for the user but I
wonder if it was just a coincidence and may break under slightly
different but similar conditions.


I think that's a fine idea. In fact there is already a function that
could be extended, kinit_hostprincipal().

rob



So in principle we could add multiple TGT retries to
kinit_hostprincipal() and then plug this function to all the places
Petr mentioned in order to provide this functionality each time TGT is
requested using keytab.

Do I understand it correctly?



Honestly I think I'd only do the retries on client installation.  I
don't know that the other uses would really benefit or need this.

But this is an opportunity to consolidate some code, so I guess the
approach I'd take is to add an option to kinit_hostprincipal of
retries=0 so that only a single kinit is done. The client installers
would pass in some value.

This change is quite a bit more invasive but it's also early in the
release cycle so the risk will be spread out.

rob

Ok I will try to implement these changes and submit them as separate 
patchset.


--
Martin^3 Babinsky

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

  a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

  b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

  c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what 
features are actually available. A state file is tons better than a 
single version number.




Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?


Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole 
refactoring bussiness in the first place?






My personal opinion is, application should not try to fix itself every
restart.


One could say that application should not try to upgrade itself every
restart, but here we are, doing it anyway...

I want to do upgrade only if needed not every restart.


If there is nothing to upgrade, nothing will be upgraded. The effect is 
the same.







In the past, I do not recall
ipa-upgradeconfig as being really fast, especially certmonger/Dogtag
related
updates were really slow thank to service restarts, etc.


Correct, but I was talking about configuration file updates, not
(re)starts, which have to always be done in ipactl anyway.




3)


* Prevent user to use ipa-upgradeconfig and ipa-ldap-updater


Even without arguments? Is ipactl now the only right place to
trigger manual
update?

Sorry, I will add more details there, if this is not clear.
ipa-upgrateconfig will be removed
ipa-ldap-updater will 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions 
appreciated

http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one 
of:


  a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

  b) ipa-server-install [install], ipa-server-install 
uninstall,

ipa-server-install upgrade

  c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We 
kind

of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.
I doesn't like this idea, if user wants to fix something, the one 
should

use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what 
features are actually available. A state file is tons better than a 
single version number.




Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?

at least ipa-dnskeysyncd service, requires updated schema for keys metadata.
This service is mandratory for DNS, so it is newly configured during 
upgrade.
Now it works because schema update is the first, so during configuration 
upgrade we have actual schema.



Running schema upgrade every time?


What if a service changes in a way, the IPA configuration will not 
work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration 
which

will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.

Yes, we have upgrade state file, but then the comparing of one 
value is

faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole 
refactoring bussiness in the first place?
The clever optimizations worked in past, but IPA grown and now 
contains constraints/requirements which nobody expected.  What if we 
will need some update which needs to execute time-consuming system check 
during every upgrade in future?
User can always run the upgrade manually, with --skip-version-check, and 
then configuration plugins will decide if the upgrade is needed.








My personal opinion is, application should not try to fix itself every
restart.


One could say that application should not try to upgrade itself every
restart, but here we are, doing it anyway...

I want to do 

Re: [Freeipa-devel] One-way trust design

2015-03-03 Thread Alexander Bokovoy

On Tue, 03 Mar 2015, Jan Pazdziora wrote:

On Mon, Feb 23, 2015 at 06:02:53PM +0200, Alexander Bokovoy wrote:

trust-related  functionality would be limited to IPA admins or TDO
object in LDAP would have to be more  accessible. Given that TDO
credentials can be used to compromise access to our  domain, it is not


Could you clarify which domain is the our domain?
From SMB perspective whole IPA realm is a single domain.





advisable to give a wider access to them.

As a side-effect of reducing exposure of TDO credentials, FreeIPA lost
ability to establish and use one-way trust to Active Directory. The


Lost ability might be confusing -- was removed in 3.1 (?) might be
better.

We never had it as a feature so support for that wasn't removed. Rather,
we lost ability to add that support.



purpose of this feature is to regain the one-way trust support, yet
without giving an elevated access to TDO credentials.


You might also want to either add a note or a link, explaining why
one-way trust is harder than two-way, IOW, why we lost the one-way
ability when we have the two-way one.

I think current text covers it clearly. If you have concrete
suggestions, feel free to edit the wiki, it is not locked down. :)

--
/ Alexander Bokovoy

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 11:05 AM, Jan Cholasta wrote:
 Dne 3.3.2015 v 10:58 Martin Kosek napsal(a):
 On 03/03/2015 09:36 AM, Petr Spacek wrote:
 On 3.3.2015 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

b) ipa-server-install [install], ipa-server-install uninstall,
 ipa-server-install upgrade

c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
 * ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.

 Because there is no exact mapping between version number and what features 
 are
 actually available. A state file is tons better than a single version 
 number.


 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?

 This does not sound right. Could you be more specific?

 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.

 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?

 I very much agree with Honza. We should always start with something
 stupidly-simply and enhance it later, when it is clear if it is really
 necessary.

 Do not over-engineer it from the very beginning.

 I completely agree with starting stupid and simply and improving in time.
 However, are we sure that what Honza proposed is the simple and stupid way?

 Doing config upgrade only when needed and thus not depending on the 
 efficiency
 and idempotency of the config upgraders seems to me as *the* stupid and 
 simple
 way for upgrade refactoring.
 
 How exactly is adding another logic layer on top of the configuration update
 more simple and stupid?

Maybe it isn't. I 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Petr Spacek
On 3.3.2015 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
 ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
* ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.
 
 Because there is no exact mapping between version number and what features are
 actually available. A state file is tons better than a single version number.
 

 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?
 
 This does not sound right. Could you be more specific?
 
 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.
 
 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?

I very much agree with Honza. We should always start with something
stupidly-simply and enhance it later, when it is clear if it is really 
necessary.

Do not over-engineer it from the very beginning.

Petr^2 Spacek

 My personal opinion is, application should not try to fix itself every
 restart.

 One could say that application should not try to upgrade itself every
 restart, but here we are, doing it anyway...
 I want to do upgrade only if needed not every restart.
 
 If there is nothing to upgrade, nothing will be upgraded. The effect is the 
 same.
 


 In the past, I do not recall
 ipa-upgradeconfig as being really fast, especially certmonger/Dogtag
 related
 updates were really slow thank to service restarts, etc.

 Correct, but I was talking about configuration file updates, not
 (re)starts, which 

Re: [Freeipa-devel] One-way trust design

2015-03-03 Thread Jan Pazdziora
On Mon, Feb 23, 2015 at 06:02:53PM +0200, Alexander Bokovoy wrote:
 trust-related  functionality would be limited to IPA admins or TDO
 object in LDAP would have to be more  accessible. Given that TDO
 credentials can be used to compromise access to our  domain, it is not

Could you clarify which domain is the our domain?

 advisable to give a wider access to them.
 
 As a side-effect of reducing exposure of TDO credentials, FreeIPA lost
 ability to establish and use one-way trust to Active Directory. The

Lost ability might be confusing -- was removed in 3.1 (?) might be
better.

 purpose of this feature is to regain the one-way trust support, yet
 without giving an elevated access to TDO credentials.

You might also want to either add a note or a link, explaining why
one-way trust is harder than two-way, IOW, why we lost the one-way
ability when we have the two-way one.

-- 
Jan Pazdziora
Principal Software Engineer, Identity Management Engineering, Red Hat

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 10:58 Martin Kosek napsal(a):

On 03/03/2015 09:36 AM, Petr Spacek wrote:

On 3.3.2015 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
* ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what features are
actually available. A state file is tons better than a single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?


Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?


I very much agree with Honza. We should always start with something
stupidly-simply and enhance it later, when it is clear if it is really 
necessary.

Do not over-engineer it from the very beginning.


I completely agree with starting stupid and simply and improving in time.
However, are we sure that what Honza proposed is the simple and stupid way?

Doing config upgrade only when needed and thus not depending on the efficiency
and idempotency of the config upgraders seems to me as *the* stupid and simple
way for upgrade refactoring.


How exactly is adding another logic layer on top of the configuration 
update more simple and stupid?




Martin


--
Jan Cholasta

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 11:00 Martin Basti napsal(a):

On 03/03/15 10:55, Jan Cholasta wrote:

Dne 3.3.2015 v 09:55 Martin Basti napsal(a):

On 03/03/15 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

 ...
2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like
data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it
always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade
would
be re-running the installer and we should aim to do just that. We
kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100%
sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one
should
use --skip-version-check option, and the IPA upgrade will be
executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data
upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what
features are actually available. A state file is tons better than a
single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?

at least ipa-dnskeysyncd service, requires updated schema for keys
metadata.
This service is mandratory for DNS, so it is newly configured during
upgrade.
Now it works because schema update is the first, so during configuration
upgrade we have actual schema.


Right, but what's your point? We are not discussing order of updates
here, I'm perfectly fine with schema updates being done before
configuration updates.

So you want to run schema update before configuration upgrade every
restart?
OR you want to run schema update if needed based on version, and then
run configuration upgrade?


I don't really care, as long as the schema is up-to-date, but I guess 
there is no harm in always running schema update either.


--
Jan Cholasta

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 09:55 AM, Martin Basti wrote:
 On 03/03/15 09:33, Jan Cholasta wrote:
 Dne 3.3.2015 v 09:06 Martin Basti napsal(a):
 On 03/03/15 07:31, Jan Cholasta wrote:
 Dne 2.3.2015 v 13:51 Martin Basti napsal(a):
 On 02/03/15 13:12, Jan Cholasta wrote:
 Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):
 On 03/02/2015 07:49 AM, Jan Cholasta wrote:
 Hi,

 Dne 24.2.2015 v 19:10 Martin Basti napsal(a):
 Hello all,

 please read the design page, any objections/suggestions appreciated
 http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring


 1)

 
 * Merge server update commands into the one command
 (ipa-server-upgrade)
 

 So there is ipa-server-install to install the server,
 ipa-server-install
 --uninstall to uninstall the server and ipa-server-upgrade to
 upgrade the
 server. Maybe we should bring some consistency here and have one of:

   a) ipa-server-install [--install], ipa-server-install
 --uninstall,
 ipa-server-install --upgrade

   b) ipa-server-install [install], ipa-server-install uninstall,
 ipa-server-install upgrade

   c) ipa-server-install, ipa-server-uninstall,
 ipa-server-upgrade

 Long term, I think we want C. Besides other advantages, it will let
 us have
 independent sets of options, based on what you want to do.

 2)

 
 * Prevent to run IPA service, if code version and configuration
 version does
 not match
* ipactl should execute ipa-server-upgrade if needed
 

 There should be no configuration version, configuration update
 should be run
 always. It's fast and hence does not need to be optimized like data
 update by
 using a monolithic version number, which brings more than a few
 problems on its
 own.

 I do not agree in this section. Why would you like to run it always,
 even if it
 was fast? No run is still faster than fast run.

 In the ideal case the installer would be idempotent and upgrade would
 be re-running the installer and we should aim to do just that. We kind
 of do that already, but there is a lot of code duplication in
 installers and ipa-upgradeconfig (I would like to fix that when
 refactoring installers). IMO it's better to always make 100% sure the
 configuration is correct rather than to save a second or two.
 I doesn't like this idea, if user wants to fix something, the one should
 use --skip-version-check option, and the IPA upgrade will be executed.

 Well, what I don't like is dealing with meaningless version numbers.
 They are causing us grief in API versioning and I don't see why it
 would be any different here.
 However you must keep the version because of schema and data upgrade, so
 why not to execute update as one batch instead of doing config upgrade
 all the time, and then data upgrade only if required.

 Because there is no exact mapping between version number and what features
 are actually available. A state file is tons better than a single version
 number.


 Some configuration upgrades, like adding new DNS related services,
 requires new schema, how we can handle this?

 This does not sound right. Could you be more specific?
 at least ipa-dnskeysyncd service, requires updated schema for keys metadata.
 This service is mandratory for DNS, so it is newly configured during upgrade.
 Now it works because schema update is the first, so during configuration
 upgrade we have actual schema.

 Running schema upgrade every time?

 What if a service changes in a way, the IPA configuration will not work?

 Then it's a bug and needs to be fixed, like any other bug. IIRC there
 was only one or two occurences of such bug in the past 3 years (I
 remember sshd_config), so I don't think you have a strong case here.
 Ok

 The user will need to change it manually, but after each restart,
 upgrade will change the value back into IPA required configuration which
 will not work.

 Says who? It's our code, we can do whatever we want, it doesn't have
 to be dumb like this.

 Yes, we have upgrade state file, but then the comparing of one value is
 faster then checking each state if was executed.

 How faster is that, like, few milliseconds? Are you seriously
 considering this the right optimization in a process that is
 magnitudes slower?
 Ok the speed is not so important, but I still do not like the idea of
 executing the code which is not needed to be executed, because I know
 the version is the same as was before last restart, so nothing changed.

 Weren't clever optimizations like this what got us into this whole
 refactoring bussiness in the first place?
 The clever optimizations worked in past, but IPA grown and now contains
 constraints/requirements which nobody expected.  What if we will need some
 update which needs to execute time-consuming system check during every upgrade
 in future?
 User can always run the upgrade manually, with --skip-version-check, and then
 configuration plugins will decide if the upgrade is needed.

I tend to agree with Martin, I would prefer to be on the safe side and not run
config upgrades every time, unless we are explicitly 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Jan Cholasta

Dne 3.3.2015 v 09:55 Martin Basti napsal(a):

On 03/03/15 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions
appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one
of:

  a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

  b) ipa-server-install [install], ipa-server-install
uninstall,
ipa-server-install upgrade

  c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will let
us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like data
update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it always,
even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade would
be re-running the installer and we should aim to do just that. We
kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% sure the
configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one
should
use --skip-version-check option, and the IPA upgrade will be executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.

However you must keep the version because of schema and data upgrade, so
why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what
features are actually available. A state file is tons better than a
single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?

at least ipa-dnskeysyncd service, requires updated schema for keys
metadata.
This service is mandratory for DNS, so it is newly configured during
upgrade.
Now it works because schema update is the first, so during configuration
upgrade we have actual schema.


Right, but what's your point? We are not discussing order of updates 
here, I'm perfectly fine with schema updates being done before 
configuration updates.





Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not
work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration
which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one
value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?

The clever optimizations worked in past, but IPA grown and now
contains constraints/requirements which nobody expected.


Yes, then why do we need another one, especially so when it does not 
provide any significant speed-up?



What if we
will need some update which needs to execute time-consuming system check
during every upgrade in future?


Then we deal with the optimization in the future, instead of 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Petr Spacek
On 2.3.2015 18:54, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
 On 03/02/2015 06:12 PM, Martin Basti wrote:
 On 02/03/15 15:43, Rob Crittenden wrote:
 Martin Basti wrote:
 ...
 But you haven't explained any case why LDAPI would fail. If LDAPI fails
 then you've got more serious problems that I'm not sure binding as DM is
 going to solve.

 The only case where DM would be handy IMHO is either some worst case
 scenario upgrade where 389-ds is up but not binding to LDAPI or if you
 want to allow running LDAP updates as non-root.
 I don't know cases when LDAPI would failed, except the case LDAPI is
 misconfigured by user, or disabled by user.
 Wasn't LDAPI needed for the DM password less upgrade so that upgrader could
 simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password, preferred is
 LDAPI.
 Question is, what is the use case for using DM password instead of LDAPI
 during upgrade.

 It is not big effort to keep both DM binding and LDAPI in code.  A user can
 always found som unexpected use case for LDAP update with DM password.

 On ipactl, would it be overkill if there is a tty to prompt the user to
 upgrade? In a non-container world it might be surprising to have an
 upgrade happen esp since upgrades take a while.
 In non-container enviroment, we can still use upgrade during RPM
 transaction.

 So you suggest  not to do update automaticaly, just write Error the IPA
 upgrade is required?
 People do all sorts of strange things. Installing the packages with
 --no-script isn't in the range of impossible. A prompt, and I'm not
 saying it's a great idea, is 2 lines of code.

 I guess it just makes me nervous.
 So lets summarize this:
 * DO upgrade if possible during RPM transaction
 Umm, I thought we want to get rid of running upgrade during RPM transaction. 
 It
 is extremely difficult to debug upgrade stuck during RPM transaction, it also
 makes RPM upgrade run longer than needed. It also makes admins nervous when
 their rpm upgrade is suddenly waiting right before the end. I even see the
 fingers slowly reaching to CTRL+C combo... (You can see the consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM
 transaction.
 
 I have no more objections.

IMHO the problem with long-running RPM upgrade should be approached from the
other way: Just print message 'IPA server upgrade is running, press CTRL+C if
you want to destroy your IPA server'.

bind-dyndb-ldap prints a message about SELinux in RPM scriptlets for couple
releases now and nobody complained (yet? :-).

Petr^2 Spacek

 

 * ipactl will NOT run upgrade, just print Error: 'please upgrade '
 * User has to run ipa-server-upgrade manually

 Does I understand it correctly?
 With --skip-version-check what sorts of problems can we foresee? I
 assume a big warning will be added to at least the man page, if not
 the cli?
 For this big warning everywhere.
 The main problem is try to run older IPA with newer data. In containers
 the problem is to run different platform specific versions which differ
 in functionality/bugfixes etc..
 Ok, pretty much the things I was thinking as well. A scary warning
 definitely seems warranted.

 Where does platform come from? I'm wondering how Debian will handle this.
 platform is derived from ipaplatform file which is used with the
 particular build. Debian should have own platform file.
 Ok, I'd add that detail to the design.

 rob

 
 


-- 
Petr^2 Spacek

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Basti

On 03/03/15 10:55, Jan Cholasta wrote:

Dne 3.3.2015 v 09:55 Martin Basti napsal(a):

On 03/03/15 09:33, Jan Cholasta wrote:

Dne 3.3.2015 v 09:06 Martin Basti napsal(a):

On 03/03/15 07:31, Jan Cholasta wrote:

Dne 2.3.2015 v 13:51 Martin Basti napsal(a):

On 02/03/15 13:12, Jan Cholasta wrote:

Dne 2.3.2015 v 12:23 Martin Kosek napsal(a):

On 03/02/2015 07:49 AM, Jan Cholasta wrote:

Hi,

Dne 24.2.2015 v 19:10 Martin Basti napsal(a):

Hello all,

please read the design page, any objections/suggestions
appreciated
http://www.freeipa.org/page/V4/Server_Upgrade_Refactoring



1)


* Merge server update commands into the one command
(ipa-server-upgrade)


So there is ipa-server-install to install the server,
ipa-server-install
--uninstall to uninstall the server and ipa-server-upgrade to
upgrade the
server. Maybe we should bring some consistency here and have one
of:

  a) ipa-server-install [--install], ipa-server-install
--uninstall,
ipa-server-install --upgrade

  b) ipa-server-install [install], ipa-server-install
uninstall,
ipa-server-install upgrade

  c) ipa-server-install, ipa-server-uninstall,
ipa-server-upgrade


Long term, I think we want C. Besides other advantages, it will 
let

us have
independent sets of options, based on what you want to do.


2)


* Prevent to run IPA service, if code version and configuration
version does
not match
   * ipactl should execute ipa-server-upgrade if needed


There should be no configuration version, configuration update
should be run
always. It's fast and hence does not need to be optimized like 
data

update by
using a monolithic version number, which brings more than a few
problems on its
own.


I do not agree in this section. Why would you like to run it 
always,

even if it
was fast? No run is still faster than fast run.


In the ideal case the installer would be idempotent and upgrade 
would

be re-running the installer and we should aim to do just that. We
kind
of do that already, but there is a lot of code duplication in
installers and ipa-upgradeconfig (I would like to fix that when
refactoring installers). IMO it's better to always make 100% 
sure the

configuration is correct rather than to save a second or two.

I doesn't like this idea, if user wants to fix something, the one
should
use --skip-version-check option, and the IPA upgrade will be 
executed.


Well, what I don't like is dealing with meaningless version numbers.
They are causing us grief in API versioning and I don't see why it
would be any different here.
However you must keep the version because of schema and data 
upgrade, so

why not to execute update as one batch instead of doing config upgrade
all the time, and then data upgrade only if required.


Because there is no exact mapping between version number and what
features are actually available. A state file is tons better than a
single version number.



Some configuration upgrades, like adding new DNS related services,
requires new schema, how we can handle this?


This does not sound right. Could you be more specific?

at least ipa-dnskeysyncd service, requires updated schema for keys
metadata.
This service is mandratory for DNS, so it is newly configured during
upgrade.
Now it works because schema update is the first, so during configuration
upgrade we have actual schema.


Right, but what's your point? We are not discussing order of updates 
here, I'm perfectly fine with schema updates being done before 
configuration updates.

So you want to run schema update before configuration upgrade every restart?
OR you want to run schema update if needed based on version, and then 
run configuration upgrade?







Running schema upgrade every time?



What if a service changes in a way, the IPA configuration will not
work?


Then it's a bug and needs to be fixed, like any other bug. IIRC there
was only one or two occurences of such bug in the past 3 years (I
remember sshd_config), so I don't think you have a strong case here.

Ok



The user will need to change it manually, but after each restart,
upgrade will change the value back into IPA required configuration
which
will not work.


Says who? It's our code, we can do whatever we want, it doesn't have
to be dumb like this.


Yes, we have upgrade state file, but then the comparing of one
value is
faster then checking each state if was executed.


How faster is that, like, few milliseconds? Are you seriously
considering this the right optimization in a process that is
magnitudes slower?

Ok the speed is not so important, but I still do not like the idea of
executing the code which is not needed to be executed, because I know
the version is the same as was before last restart, so nothing 
changed.


Weren't clever optimizations like this what got us into this whole
refactoring bussiness in the first place?

The clever optimizations worked in past, but IPA grown and now
contains constraints/requirements which nobody expected.


Yes, then why do we need another one, especially so when 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 03:16 PM, Simo Sorce wrote:
 On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
 On 03/02/2015 06:12 PM, Martin Basti wrote:
 On 02/03/15 15:43, Rob Crittenden wrote:
 Martin Basti wrote:
 ...
 But you haven't explained any case why LDAPI would fail. If LDAPI fails
 then you've got more serious problems that I'm not sure binding as DM is
 going to solve.

 The only case where DM would be handy IMHO is either some worst case
 scenario upgrade where 389-ds is up but not binding to LDAPI or if you
 want to allow running LDAP updates as non-root.
 I don't know cases when LDAPI would failed, except the case LDAPI is
 misconfigured by user, or disabled by user.
 Wasn't LDAPI needed for the DM password less upgrade so that upgrader could
 simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password, 
 preferred is LDAPI.
 Question is, what is the use case for using DM password instead of LDAPI 
 during upgrade.
 
 There is no use case for using the DM password.

+1, so we will only use LDAPI and ditch DM password options and querying that
we now have with ipa-ldap-updater?

 
 It is not big effort to keep both DM binding and LDAPI in code.  A user can
 always found som unexpected use case for LDAP update with DM password.

 On ipactl, would it be overkill if there is a tty to prompt the user to
 upgrade? In a non-container world it might be surprising to have an
 upgrade happen esp since upgrades take a while.
 In non-container enviroment, we can still use upgrade during RPM
 transaction.

 So you suggest  not to do update automaticaly, just write Error the IPA
 upgrade is required?
 People do all sorts of strange things. Installing the packages with
 --no-script isn't in the range of impossible. A prompt, and I'm not
 saying it's a great idea, is 2 lines of code.

 I guess it just makes me nervous.
 So lets summarize this:
 * DO upgrade if possible during RPM transaction
 Umm, I thought we want to get rid of running upgrade during RPM 
 transaction. It
 is extremely difficult to debug upgrade stuck during RPM transaction, it 
 also
 makes RPM upgrade run longer than needed. It also makes admins nervous when
 their rpm upgrade is suddenly waiting right before the end. I even see the
 fingers slowly reaching to CTRL+C combo... (You can see the consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM 
 transaction.
 
 This is true, stopping IPA and requiring manual intervention is not ok.

What is the plan then? Keep upgrades done during RPM transaction? Note that RPM
transaction was already stuck several times because IPA, or rather DS, 
deadlocked.

We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
original plan was to do the upgrade during ipactl start, this would fix this
ticket. Alternatively, should we remove the upgrade from RPM scriptlet and only
call asynchronous systemctl restart ipa.service that would trigger the
upgrade in separate process and log results in ipa.service?

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Simo Sorce
On Tue, 2015-03-03 at 15:33 +0100, Martin Kosek wrote:
 On 03/03/2015 03:16 PM, Simo Sorce wrote:
  On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
  On 02/03/15 18:28, Martin Kosek wrote:
  On 03/02/2015 06:12 PM, Martin Basti wrote:
  On 02/03/15 15:43, Rob Crittenden wrote:
  Martin Basti wrote:
  ...
  But you haven't explained any case why LDAPI would fail. If LDAPI fails
  then you've got more serious problems that I'm not sure binding as DM is
  going to solve.
 
  The only case where DM would be handy IMHO is either some worst case
  scenario upgrade where 389-ds is up but not binding to LDAPI or if you
  want to allow running LDAP updates as non-root.
  I don't know cases when LDAPI would failed, except the case LDAPI is
  misconfigured by user, or disabled by user.
  Wasn't LDAPI needed for the DM password less upgrade so that upgrader 
  could
  simply bind as root with EXTERNAL auth?
  We can do upgrade in both way, using LDAPI or using DM password, 
  preferred is LDAPI.
  Question is, what is the use case for using DM password instead of LDAPI 
  during upgrade.
  
  There is no use case for using the DM password.
 
 +1, so we will only use LDAPI and ditch DM password options and querying that
 we now have with ipa-ldap-updater?
 
  
  It is not big effort to keep both DM binding and LDAPI in code.  A user 
  can
  always found som unexpected use case for LDAP update with DM password.
 
  On ipactl, would it be overkill if there is a tty to prompt the user 
  to
  upgrade? In a non-container world it might be surprising to have an
  upgrade happen esp since upgrades take a while.
  In non-container enviroment, we can still use upgrade during RPM
  transaction.
 
  So you suggest  not to do update automaticaly, just write Error the IPA
  upgrade is required?
  People do all sorts of strange things. Installing the packages with
  --no-script isn't in the range of impossible. A prompt, and I'm not
  saying it's a great idea, is 2 lines of code.
 
  I guess it just makes me nervous.
  So lets summarize this:
  * DO upgrade if possible during RPM transaction
  Umm, I thought we want to get rid of running upgrade during RPM 
  transaction. It
  is extremely difficult to debug upgrade stuck during RPM transaction, it 
  also
  makes RPM upgrade run longer than needed. It also makes admins nervous 
  when
  their rpm upgrade is suddenly waiting right before the end. I even see the
  fingers slowly reaching to CTRL+C combo... (You can see the consequences)
  People are used to have IPA upgraded and ready after RPM upgrade.
  They may be shocked if IPA services will be in shutdown state after RPM 
  transaction.
  
  This is true, stopping IPA and requiring manual intervention is not ok.
 
 What is the plan then? Keep upgrades done during RPM transaction? Note that 
 RPM
 transaction was already stuck several times because IPA, or rather DS, 
 deadlocked.
 
 We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
 original plan was to do the upgrade during ipactl start, this would fix this
 ticket. Alternatively, should we remove the upgrade from RPM scriptlet and 
 only
 call asynchronous systemctl restart ipa.service that would trigger the
 upgrade in separate process and log results in ipa.service?

It's a hard problem.
First of all DS should never deadlock, we shouldn't change all our
process because one component has a bug
If DS deadlocks, it will do so whether it is run in RPM or not, we need
to add some way to terminate the upgrade ourselves if it lasts too long
I guess.

The problem with an asynchronous restart is ... when do you do that ?
What triggers it ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


[Freeipa-devel] [PATCHES 399-401] Allow multiple API instances

2015-03-03 Thread Jan Cholasta

Hi,

the attached patches provide an attempt to fix 
https://fedorahosted.org/freeipa/ticket/3090.


Patch 401 serves as an example and modifies ipa-advise to use its own 
API instance for Advice plugins.


Honza

--
Jan Cholasta
From 3715c9b4ca43eab6c5ad01b34cd1b14838241bde Mon Sep 17 00:00:00 2001
From: Jan Cholasta jchol...@redhat.com
Date: Mon, 16 Feb 2015 13:11:38 +
Subject: [PATCH 1/3] ipalib: Allow multiple API instances

Merged the Registrar class into the Registry class. Plugins are now
registered globally instead of in ipalib.api and are instantiated per-API
instance. Different set of plugin base classes can be used in each API
instance.

https://fedorahosted.org/freeipa/ticket/3090
---
 ipalib/backend.py |   3 +
 ipalib/frontend.py|  10 +-
 ipalib/plugable.py| 204 --
 ipatests/test_ipalib/test_plugable.py | 119 
 4 files changed, 185 insertions(+), 151 deletions(-)

diff --git a/ipalib/backend.py b/ipalib/backend.py
index 2100589..4c1001d 100644
--- a/ipalib/backend.py
+++ b/ipalib/backend.py
@@ -27,7 +27,10 @@ import os
 from errors import PublicError, InternalError, CommandError
 from request import context, Connection, destroy_context
 
+register = plugable.Registry()
 
+
+@register.base()
 class Backend(plugable.Plugin):
 
 Base class for all backend plugins.
diff --git a/ipalib/frontend.py b/ipalib/frontend.py
index 98070b8..e82a03a 100644
--- a/ipalib/frontend.py
+++ b/ipalib/frontend.py
@@ -27,7 +27,7 @@ from distutils import version
 from ipapython.version import API_VERSION
 from ipapython.ipa_log_manager import root_logger
 from base import NameSpace
-from plugable import Plugin, is_production_mode
+from plugable import Plugin, Registry, is_production_mode
 from parameters import create_param, Param, Str, Flag, Password
 from output import Output, Entry, ListOfEntries
 from text import _
@@ -40,6 +40,9 @@ from textwrap import wrap
 
 RULE_FLAG = 'validation_rule'
 
+register = Registry()
+
+
 def rule(obj):
 assert not hasattr(obj, RULE_FLAG)
 setattr(obj, RULE_FLAG, True)
@@ -366,6 +369,7 @@ class HasParam(Plugin):
 setattr(self, name, namespace)
 
 
+@register.base()
 class Command(HasParam):
 
 A public IPA atomic operation.
@@ -1120,6 +1124,7 @@ class Local(Command):
 return self.forward(*args, **options)
 
 
+@register.base()
 class Object(HasParam):
 finalize_early = False
 
@@ -1278,6 +1283,7 @@ class Attribute(Plugin):
 super(Attribute, self)._on_finalize()
 
 
+@register.base()
 class Method(Attribute, Command):
 
 A command with an associated object.
@@ -1364,6 +1370,7 @@ class Method(Attribute, Command):
 yield param
 
 
+@register.base()
 class Updater(Method):
 
 An LDAP update with an associated object (always update).
@@ -1423,6 +1430,7 @@ class _AdviceOutput(object):
 self.content.append(line)
 
 
+@register.base()
 class Advice(Plugin):
 
 Base class for advices, plugins for ipa-advise.
diff --git a/ipalib/plugable.py b/ipalib/plugable.py
index a6504d1..aae7626 100644
--- a/ipalib/plugable.py
+++ b/ipalib/plugable.py
@@ -74,18 +74,94 @@ class Registry(object):
 For forward compatibility, make sure that the module-level instance of
 this object is named register.
 
-# TODO: Instead of auto-loading when plugin modules are imported,
-# plugins should be stored in this object.
-# The API should examine it and load plugins explicitly.
-def __call__(self):
-from ipalib import api
 
-def decorator(cls):
-api.register(cls)
-return cls
+__allowed = {}
+__registered = set()
+
+def base(self):
+def decorator(base):
+if not inspect.isclass(base):
+raise TypeError('plugin base must be a class; got %r' % base)
+
+if base in self.__allowed:
+raise errors.PluginDuplicateError(plugin=base)
+
+self.__allowed[base] = {}
+
+return base
+
+return decorator
+
+def __findbases(self, klass):
+
+Iterates through allowed bases that ``klass`` is a subclass of.
+
+Raises `errors.PluginSubclassError` if ``klass`` is not a subclass of
+any allowed base.
+
+:param klass: The plugin class to find bases for.
+
+found = False
+for (base, sub_d) in self.__allowed.iteritems():
+if issubclass(klass, base):
+found = True
+yield (base, sub_d)
+if not found:
+raise errors.PluginSubclassError(
+plugin=klass, bases=self.__allowed.keys()
+)
+
+def __call__(self, override=False):
+def decorator(klass):
+if not inspect.isclass(klass):
+raise TypeError('plugin must be a class; got %r' % klass)
+
+# Raise DuplicateError if 

Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Simo Sorce
On Tue, 2015-03-03 at 15:40 +0100, Martin Basti wrote:
 On 03/03/15 15:33, Martin Kosek wrote:
  On 03/03/2015 03:16 PM, Simo Sorce wrote:
  On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
  On 02/03/15 18:28, Martin Kosek wrote:
  On 03/02/2015 06:12 PM, Martin Basti wrote:
  On 02/03/15 15:43, Rob Crittenden wrote:
  Martin Basti wrote:
  ...
  But you haven't explained any case why LDAPI would fail. If LDAPI fails
  then you've got more serious problems that I'm not sure binding as DM 
  is
  going to solve.
 
  The only case where DM would be handy IMHO is either some worst case
  scenario upgrade where 389-ds is up but not binding to LDAPI or if you
  want to allow running LDAP updates as non-root.
  I don't know cases when LDAPI would failed, except the case LDAPI is
  misconfigured by user, or disabled by user.
  Wasn't LDAPI needed for the DM password less upgrade so that upgrader 
  could
  simply bind as root with EXTERNAL auth?
  We can do upgrade in both way, using LDAPI or using DM password,
  preferred is LDAPI.
  Question is, what is the use case for using DM password instead of LDAPI
  during upgrade.
  There is no use case for using the DM password.
  +1, so we will only use LDAPI and ditch DM password options and querying 
  that
  we now have with ipa-ldap-updater?
 
  It is not big effort to keep both DM binding and LDAPI in code.  A user 
  can
  always found som unexpected use case for LDAP update with DM password.
 
  On ipactl, would it be overkill if there is a tty to prompt the user 
  to
  upgrade? In a non-container world it might be surprising to have an
  upgrade happen esp since upgrades take a while.
  In non-container enviroment, we can still use upgrade during RPM
  transaction.
 
  So you suggest  not to do update automaticaly, just write Error the 
  IPA
  upgrade is required?
  People do all sorts of strange things. Installing the packages with
  --no-script isn't in the range of impossible. A prompt, and I'm not
  saying it's a great idea, is 2 lines of code.
 
  I guess it just makes me nervous.
  So lets summarize this:
  * DO upgrade if possible during RPM transaction
  Umm, I thought we want to get rid of running upgrade during RPM 
  transaction. It
  is extremely difficult to debug upgrade stuck during RPM transaction, it 
  also
  makes RPM upgrade run longer than needed. It also makes admins nervous 
  when
  their rpm upgrade is suddenly waiting right before the end. I even see 
  the
  fingers slowly reaching to CTRL+C combo... (You can see the consequences)
  People are used to have IPA upgraded and ready after RPM upgrade.
  They may be shocked if IPA services will be in shutdown state after RPM
  transaction.
  This is true, stopping IPA and requiring manual intervention is not ok.
  What is the plan then? Keep upgrades done during RPM transaction? Note that 
  RPM
  transaction was already stuck several times because IPA, or rather DS, 
  deadlocked.
 
  We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
  original plan was to do the upgrade during ipactl start, this would fix this
  ticket. Alternatively, should we remove the upgrade from RPM scriptlet and 
  only
  call asynchronous systemctl restart ipa.service that would trigger the
  upgrade in separate process and log results in ipa.service?
 
  Martin
 The plan is do upgrade during RPM transaction if possible. If not then 
 ipactl start, will show warning for user to do manual upgrade (Rob 
 wanted it in this way, not doing auto upgrade by ipactl)
 
 So the fedup case is: RPM upgrade failed, ipactl start will detect 
 version mismatch, show error and prompt user to run ipa-server-upgrade

I think this is wrong, sorry.
It means unattended installs will just fail to restart and require
manual intervention.
At the very least this process needs to be conditional, and upgrade
needs to be done automatically.
If the admin insist he can set something in default.conf to block
autoupgrades or something.

Simo.


-- 
Simo Sorce * Red Hat, Inc * New York

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel


Re: [Freeipa-devel] IPA Server upgrade 4.2 design

2015-03-03 Thread Martin Kosek
On 03/03/2015 03:49 PM, Simo Sorce wrote:
 On Tue, 2015-03-03 at 15:33 +0100, Martin Kosek wrote:
 On 03/03/2015 03:16 PM, Simo Sorce wrote:
 On Mon, 2015-03-02 at 18:54 +0100, Martin Basti wrote:
 On 02/03/15 18:28, Martin Kosek wrote:
 On 03/02/2015 06:12 PM, Martin Basti wrote:
 On 02/03/15 15:43, Rob Crittenden wrote:
 Martin Basti wrote:
 ...
 But you haven't explained any case why LDAPI would fail. If LDAPI fails
 then you've got more serious problems that I'm not sure binding as DM is
 going to solve.

 The only case where DM would be handy IMHO is either some worst case
 scenario upgrade where 389-ds is up but not binding to LDAPI or if you
 want to allow running LDAP updates as non-root.
 I don't know cases when LDAPI would failed, except the case LDAPI is
 misconfigured by user, or disabled by user.
 Wasn't LDAPI needed for the DM password less upgrade so that upgrader 
 could
 simply bind as root with EXTERNAL auth?
 We can do upgrade in both way, using LDAPI or using DM password, 
 preferred is LDAPI.
 Question is, what is the use case for using DM password instead of LDAPI 
 during upgrade.

 There is no use case for using the DM password.

 +1, so we will only use LDAPI and ditch DM password options and querying that
 we now have with ipa-ldap-updater?


 It is not big effort to keep both DM binding and LDAPI in code.  A user 
 can
 always found som unexpected use case for LDAP update with DM password.

 On ipactl, would it be overkill if there is a tty to prompt the user 
 to
 upgrade? In a non-container world it might be surprising to have an
 upgrade happen esp since upgrades take a while.
 In non-container enviroment, we can still use upgrade during RPM
 transaction.

 So you suggest  not to do update automaticaly, just write Error the IPA
 upgrade is required?
 People do all sorts of strange things. Installing the packages with
 --no-script isn't in the range of impossible. A prompt, and I'm not
 saying it's a great idea, is 2 lines of code.

 I guess it just makes me nervous.
 So lets summarize this:
 * DO upgrade if possible during RPM transaction
 Umm, I thought we want to get rid of running upgrade during RPM 
 transaction. It
 is extremely difficult to debug upgrade stuck during RPM transaction, it 
 also
 makes RPM upgrade run longer than needed. It also makes admins nervous 
 when
 their rpm upgrade is suddenly waiting right before the end. I even see the
 fingers slowly reaching to CTRL+C combo... (You can see the consequences)
 People are used to have IPA upgraded and ready after RPM upgrade.
 They may be shocked if IPA services will be in shutdown state after RPM 
 transaction.

 This is true, stopping IPA and requiring manual intervention is not ok.

 What is the plan then? Keep upgrades done during RPM transaction? Note that 
 RPM
 transaction was already stuck several times because IPA, or rather DS, 
 deadlocked.

 We also need to address https://fedorahosted.org/freeipa/ticket/3849. The
 original plan was to do the upgrade during ipactl start, this would fix this
 ticket. Alternatively, should we remove the upgrade from RPM scriptlet and 
 only
 call asynchronous systemctl restart ipa.service that would trigger the
 upgrade in separate process and log results in ipa.service?
 
 It's a hard problem.
 First of all DS should never deadlock, we shouldn't change all our
 process because one component has a bug
 If DS deadlocks, it will do so whether it is run in RPM or not, we need
 to add some way to terminate the upgrade ourselves if it lasts too long
 I guess.
 
 The problem with an asynchronous restart is ... when do you do that ?
 What triggers it ?

systemctl restart ipa.service --no-block :-)

If there is no better place for the upgrade outside of RPM transaction, I am
fine, as long as ipactl writes the warning for FedUp users, as Martin
mentioned, to make sure they do not run un-upgraded IPA.

Martin

___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel