Re: [MERGE] stable-2.15 to stable-2.16

2016-12-16 Thread 'Federico Pareschi' via ganeti-devel 
LGTM

On Friday, December 16, 2016 at 4:11:30 PM UTC, Brian Foley wrote:
>
> commit 711fbc08fd895b826d63c1ffc7cb75f35dc4331e 
> Merge: 703e23e da3f300 
> Author: Brian Foley <bpfo...@google.com> 
> Date:   Fri Dec 16 16:01:48 2016 + 
>
>     Merge branch 'stable-2.15' into stable-2.16 
> 
> Merge forward patches from stable-2.15 
> 
> * stable-2.15 
>   Fix gnt-instance console instance unpausing for xl toolstack 
>   Disable pylint too-many-nested-blocks in _RunCmdPipe 
>   Reduce nesting in import-export ProcessChildIO 
>   Reduce nesting in LUOobCommand.Exec 
>   Reduce nesting in LUInstanceCreate.RunOsScripts 
>   Reduce nesting in RemoveNodeSshKeyBulk key calculation 
>   Reduce nesting in RemoveNodeSshKeyBulk ssh logic 
>   Reduce nesting in gnt-cluster VerifyDisks missing disk loop 
>   Reduce nesting in _CheckVLANArguments 
>   Reduce nesting in StartDaemon 
>   Disable pylint bad-continuation warning 
>   Disable pylint superfluous-parens warning 
>   Disable pylint redefined-variable-type warning 
>   Disable pylint too-many-branches warnings 
>   Disable pylint broad-except warnings 
>   Disable incorrect pylint assigning-non-slot warning 
>   Quell pylint unbalanced-tuple-unpacking warning 
>   Cleanup: Use new-style classes everywhere 
>   Quell pylint socket.timeout warning 
>   Quell the pylint wrong-import-order warnings 
>   Quell cell-var-from-loop warning 
>   Use default value lambda param to avoid cell-var-from-loop 
>   Quell too-many-boolean-expressions 
>   Remove pylint tests removed in pylint 2.0 
>   Quell trailing newline 
>   Quell bad-whitespace warning 
>   Quell consider-using-enumerate warning 
>   Disable pylint unsubscriptable-object warning 
>   Disable pylint bare-except warning 
>   Disable unwanted pylint wrong-import-position warnings 
>   Disable pylint unused-wildcard-import warning 
>   Disable incorrect pylint not-callable warning 
>   Disable pylint unpacking-non-sequence warning 
>   Disable pylint misplaced-comparison-constant warning 
>   Disable incorect pylint simplify-if-statement warning 
>   Disable pylint eval-used warning 
>   Disable pylint invalid-name warning 
>   Disable pylint import-self warning 
>   Disable some pylint unused-import warnings 
>   Replace deprecated pylint >=0.27 pragma with new form 
>   Delete old warning disables removed from pylint 1.6 
>   Fix pylint >1.4 pycurl no-member warnings 
>   Cleanup: Remove unused/duplicate module/fn import 
>   Cleanup: Fix unidiomatic-typecheck 
>   Cleanup: Remove some unneeded pylint disables 
>   Cleanup: Iterate dict rather than key list 
>   Cleanup: Remove unused format key 
>   Cleanup: StartInstance and RebootInstance return None 
>   Cleanup: Fix for/else with no break in AddAuthorizedKeys 
>   Cleanup: Replace map/filters with list comprehensions 
>   Cleanup: del is a statement not a function 
>   Cleanup: Use FOO not in BAR instead of not FOO in BAR 
>   Cleanup: Simplify boolean assignment 
>   Cleanup: Remove some unnecessary if (...) parens 
>   Fix invalid variable error for file-based disks 
>   FIX: Refactor DiagnoseOS to use a loop, not an inner fn 
>   FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined 
>   Fix for incorrect parsing of DRBD versions 
>   Change a few errors to report names, not UUIDs 
>   Give atomicWriteFile temp filenames a more distinct pattern 
>   LV check failure should print instance name 
>   Disable logging CallRPCMethod timings in non-debug configs 
>   568 Update hv_kvm to handle output from qemu >= 1.6.0 
> 
> Trivial merge conflicts: 
>   lib/cli.py   -- whitespace 
>   lib/cmdlib/cluster/verify.py -- code previously removed from 2.16 
>   lib/hypervisor/hv_xen.py -- tuple vs list 
>   lib/luxi.py  -- whitespace 
>   lib/server/masterd.py-- code cleaned up in 2.15 and 
>   previously removed in 2.16 
>   lib/storage/filestorage.py   -- whitespace & generator instead of 
> list 
>   lib/tools/node_cleanup.py-- whitespace 
> 
> Signed-off-by: Brian Foley <bpfo...@google.com> 
>
> diff --cc lib/client/gnt_cluster.py 
> index f834d2b,e23fb50..8b99c2c 
> --- a/lib/client/gnt_cluster.py 
> +++ b/lib/client/gnt_cluster.py 
> @@@ -299,19 -296,8 +301,16 @@@ def InitCluster(opts, args) 
>   
> default_ialloc_params = opts.default_iallocator_para

[MERGE] stable-2.15 to stable-2.16

2016-12-16 Thread 'Brian Foley' via ganeti-devel 
commit 711fbc08fd895b826d63c1ffc7cb75f35dc4331e
Merge: 703e23e da3f300
Author: Brian Foley <bpfo...@google.com>
Date:   Fri Dec 16 16:01:48 2016 +

    Merge branch 'stable-2.15' into stable-2.16
    
    Merge forward patches from stable-2.15

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  Fix for incorrect parsing of DRBD versions
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0

Trivial merge conflicts:
  lib/cli.py   -- whitespace
  lib/cmdlib/cluster/verify.py -- code previously removed from 2.16
  lib/hypervisor/hv_xen.py -- tuple vs list
  lib/luxi.py  -- whitespace
  lib/server/masterd.py-- code cleaned up in 2.15 and
  previously removed in 2.16
  lib/storage/filestorage.py   -- whitespace & generator instead of list
  lib/tools/node_cleanup.py-- whitespace

Signed-off-by: Brian Foley <bpfo...@google.com>

diff --cc lib/client/gnt_cluster.py
index f834d2b,e23fb50..8b99c2c
--- a/lib/client/gnt_cluster.py
+++ b/lib/client/gnt_cluster.py
@@@ -299,19 -296,8 +301,16 @@@ def InitCluster(opts, args)
  
default_ialloc_params = opts.default_iallocator_params
  
-   if opts.enabled_user_shutdown:
- enabled_user_shutdown = True
-   else:
- enabled_user_shutdown = False
+   enabled_user_shutdown = bool(opts.enabled_user_shutdown)
  
 +  if opts.ssh_key_type:
 +ssh_key_type = opts.ssh_key_type
 +  else:
 +ssh_key_type = constants.SSH_DEFAULT_KEY_TYPE
 +
 +  ssh_key_bits = ssh.DetermineKeyBits(ssh_key_type, opts.ssh_key_bits, None,
 +  None)
 +
bootstrap.InitCluster(cluster_name=args[0],
  secondary_ip=opts.secondary_ip,
  vg_name=vg_name,
diff --cc lib/jqueue/__init__.py
index

Re: [MERGE] stable-2.15 to stable 2.16

2016-08-22 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM, thanks!

On Mon, Aug 22, 2016 at 12:12 PM, 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit 3b16955afc001fd3bf06f06edd9e3a7ea013ab60
> Merge: a1ae7d3 236a237
> Author: Brian Foley <bpfo...@google.com>
> Date:   Mon Aug 22 12:08:27 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.13
>   Bugfix: migrate needs HypervisorClass, not an instance
>
> Signed-off-by: Brian Foley <bpfo...@google.com>
>
>

[MERGE] stable-2.15 to stable 2.16

2016-08-22 Thread 'Brian Foley' via ganeti-devel 
commit 3b16955afc001fd3bf06f06edd9e3a7ea013ab60
Merge: a1ae7d3 236a237
Author: Brian Foley <bpfo...@google.com>
Date:   Mon Aug 22 12:08:27 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfo...@google.com>

Re: [MERGE] stable-2.15 to stable 2.16

2016-07-07 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM, thanks!

On Thursday, July 7, 2016 at 11:39:07 AM UTC+1, Brian Foley wrote:
>
> commit ad83a3d144c18a7191b7852c01871b43e83b54bd 
> Merge: 24da2b1 55ad60e 
> Author: Brian Foley <bpfo...@google.com> 
> Date:   Thu Jul 7 11:31:19 2016 +0100 
>
>     Merge branch 'stable-2.15' into stable-2.16 
> 
> * stable-2.15 
>   Cleanup more pylint/pep8/apidoc errors 
>   KVM: handle gracefully too old/too new psutil versions 
> 
> Manually fix up merge conflicts. 
> 
> Signed-off-by: Brian Foley <bpfo...@google.com> 
>
>

[MERGE] stable-2.15 to stable 2.16

2016-07-07 Thread 'Brian Foley' via ganeti-devel 
commit ad83a3d144c18a7191b7852c01871b43e83b54bd
Merge: 24da2b1 55ad60e
Author: Brian Foley <bpfo...@google.com>
Date:   Thu Jul 7 11:31:19 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Cleanup more pylint/pep8/apidoc errors
  KVM: handle gracefully too old/too new psutil versions

Manually fix up merge conflicts.

Signed-off-by: Brian Foley <bpfo...@google.com>

Re: [MERGE] stable-2.15 to stable-2.16

2016-06-14 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM, although there is nothing to review :)

On Tue, Jun 14, 2016 at 12:48 PM, 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit 2dc71a7814258445b2a8de49747aebc282642343
> Merge: 90281b4 d27c6da
> Author: Brian Foley <bpfo...@google.com>
> Date:   Tue Jun 14 12:28:28 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> Manually resolve conflicts.
>
> Signed-off-by: Brian Foley <bpfo...@google.com>
>
>

[MERGE] stable-2.15 to stable-2.16

2016-06-14 Thread 'Brian Foley' via ganeti-devel 
commit 2dc71a7814258445b2a8de49747aebc282642343
Merge: 90281b4 d27c6da
Author: Brian Foley <bpfo...@google.com>
Date:   Tue Jun 14 12:28:28 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

Manually resolve conflicts.

Signed-off-by: Brian Foley <bpfo...@google.com>

Re: [MERGE] stable-2.15 to stable 2.16

2016-06-10 Thread 'Iustin Pop' via ganeti-devel 
2016-06-10 15:28 GMT+02:00 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com>:

> commit 5785f214a9e728465a4bfc1aef7ded306225cfa2
> Merge: 40cd52f 2429235
> Author: Brian Foley <bpfo...@google.com>
> Date:   Fri Jun 10 14:23:10 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.15
>   Fixup compatibility with GHC 7.4/base 4.5
>
> Signed-off-by: Brian Foley <bpfo...@google.com>
>

LGTM, thanks.

iustin

[MERGE] stable-2.15 to stable 2.16

2016-06-10 Thread 'Brian Foley' via ganeti-devel 
commit 5785f214a9e728465a4bfc1aef7ded306225cfa2
Merge: 40cd52f 2429235
Author: Brian Foley <bpfo...@google.com>
Date:   Fri Jun 10 14:23:10 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Fixup compatibility with GHC 7.4/base 4.5

Signed-off-by: Brian Foley <bpfo...@google.com>

Re: [MERGE] stable-2.15 to stable-2.16

2016-05-17 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM

On Tuesday, May 17, 2016 at 1:28:21 PM UTC+1, Brian Foley wrote:
>
> commit 6387dcf3b01ee5dce7a2c4abd438b877b34de462 
> Merge: 9b6d390 671d167 
> Author: Brian Foley <bpfo...@google.com> 
> Date:   Tue May 17 12:44:44 2016 +0100 
>
>     Merge branch 'stable-2.15' into stable-2.16 
> 
> * stable-2.15 
>   mcpu: Raise caught OpPrereqErrors with too few args 
>   Make EnsureDirs print chmod mode on errors 
>   Fix CLI option typo in NEWS file 
> 
> Signed-off-by: Brian Foley <bpfo...@google.com> 
>
> diff --cc test/py/ganeti.mcpu_unittest.py 
> index 54bd640,796c2ba..42446c7 
> --- a/test/py/ganeti.mcpu_unittest.py 
> +++ b/test/py/ganeti.mcpu_unittest.py 
> @@@ -33,14 -33,16 +33,17 @@@ 
>   
>   import unittest 
>   import itertools 
> + import mocks 
> + from cmdlib.testsupport.rpc_runner_mock import CreateRpcRunnerMock 
>   
>   from ganeti import compat 
>  +from ganeti import errors 
>   from ganeti import mcpu 
>   from ganeti import opcodes 
>   from ganeti import cmdlib 
>   from ganeti import locking 
>  +from ganeti import serializer 
> + from ganeti import ht 
>  -from ganeti import errors 
>   from ganeti import constants 
>   from ganeti.constants import \ 
>   LOCK_ATTEMPTS_TIMEOUT, \ 
> @@@ -177,42 -179,37 +180,73 @@@ class TestProcessResult(unittest.TestCa 
>   self.assertEqual(op2.comment, "foobar") 
>   self.assertEqual(op2.debug_level, 3) 
>   
> + class TestExecLU(unittest.TestCase): 
> +   class OpTest(opcodes.OpCode): 
> + OP_DSC_FIELD = "data" 
> + OP_PARAMS = [ 
> +   ("data", ht.NoDefault, ht.TString, None), 
> + ] 
> + 
> +   def setUp(self): 
> + self.ctx = mocks.FakeContext() 
> + self.cfg = self.ctx.GetConfig("ec_id") 
> + self.rpc = CreateRpcRunnerMock() 
> + self.proc = mcpu.Processor(self.ctx, "ec_id", enable_locks = False) 
> + self.op = self.OpTest() 
> + self.calc_timeout = lambda: 42 
> + 
> +   def testRunLU(self): 
> + lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, None) 
> + self.proc._ExecLU(lu) 
> + 
> +   def testRunLUWithPrereqError(self): 
> + prereq = errors.OpPrereqError(self.op, errors.ECODE_INVAL) 
> + lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, prereq) 
> + self.assertRaises(errors.OpPrereqError, self.proc._LockAndExecLU, 
> + lu, locking.LEVEL_CLUSTER, self.calc_timeout) 
> + 
> +   def testRunLUWithPrereqErrorMissingECode(self): 
> + prereq = errors.OpPrereqError(self.op) 
> + lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, prereq) 
> + self.assertRaises(errors.OpPrereqError, self.proc._LockAndExecLU, 
> + lu, locking.LEVEL_CLUSTER, self.calc_timeout) 
> + 
>   
>  +class TestSecretParams(unittest.TestCase): 
>  +  def testSecretParamsCheckNoError(self): 
>  +op = opcodes.OpInstanceCreate( 
>  +  instance_name="plain.example.com", 
>  +  pnode="master.example.com", 
>  +  disk_template=constants.DT_PLAIN, 
>  +  mode=constants.INSTANCE_CREATE, 
>  +  nics=[{}], 
>  +  disks=[{ 
>  +constants.IDISK_SIZE: 1024 
>  +  }], 
>  +  osparams_secret= serializer.PrivateDict({"foo":"bar", 
> "foo2":"bar2"}), 
>  +  os_type="debian-image") 
>  + 
>  +try: 
>  +  mcpu._CheckSecretParameters(op) 
>  +except errors.OpPrereqError: 
>  +  self.fail("OpPrereqError raised unexpectedly in 
> _CheckSecretParameters") 
>  + 
>  +  def testSecretParamsCheckWithError(self): 
>  +op = opcodes.OpInstanceCreate( 
>  +  instance_name="plain.example.com", 
>  +  pnode="master.example.com", 
>  +  disk_template=constants.DT_PLAIN, 
>  +  mode=constants.INSTANCE_CREATE, 
>  +  nics=[{}], 
>  +  disks=[{ 
>  +constants.IDISK_SIZE: 1024 
>  +  }], 
>  +  osparams_secret= serializer.PrivateDict({"foo":"bar", 
>  + 
>  "secret_param":""}), 
>  +  os_type="debian-image") 
>  + 
>  +self.assertRaises(errors.OpPrereqError, mcpu._CheckSecretParameters, 
> op) 
>  + 
>  + 
>   if __name__ == "__main__": 
> testutils.GanetiTestProgram() 
>

[MERGE] stable-2.15 to stable-2.16

2016-05-17 Thread 'Brian Foley' via ganeti-devel 
commit 6387dcf3b01ee5dce7a2c4abd438b877b34de462
Merge: 9b6d390 671d167
Author: Brian Foley <bpfo...@google.com>
Date:   Tue May 17 12:44:44 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  mcpu: Raise caught OpPrereqErrors with too few args
  Make EnsureDirs print chmod mode on errors
  Fix CLI option typo in NEWS file

Signed-off-by: Brian Foley <bpfo...@google.com>

diff --cc test/py/ganeti.mcpu_unittest.py
index 54bd640,796c2ba..42446c7
--- a/test/py/ganeti.mcpu_unittest.py
+++ b/test/py/ganeti.mcpu_unittest.py
@@@ -33,14 -33,16 +33,17 @@@
  
  import unittest
  import itertools
+ import mocks
+ from cmdlib.testsupport.rpc_runner_mock import CreateRpcRunnerMock
  
  from ganeti import compat
 +from ganeti import errors
  from ganeti import mcpu
  from ganeti import opcodes
  from ganeti import cmdlib
  from ganeti import locking
 +from ganeti import serializer
+ from ganeti import ht
 -from ganeti import errors
  from ganeti import constants
  from ganeti.constants import \
  LOCK_ATTEMPTS_TIMEOUT, \
@@@ -177,42 -179,37 +180,73 @@@ class TestProcessResult(unittest.TestCa
  self.assertEqual(op2.comment, "foobar")
  self.assertEqual(op2.debug_level, 3)
  
+ class TestExecLU(unittest.TestCase):
+   class OpTest(opcodes.OpCode):
+ OP_DSC_FIELD = "data"
+ OP_PARAMS = [
+   ("data", ht.NoDefault, ht.TString, None),
+ ]
+ 
+   def setUp(self):
+ self.ctx = mocks.FakeContext()
+ self.cfg = self.ctx.GetConfig("ec_id")
+ self.rpc = CreateRpcRunnerMock()
+ self.proc = mcpu.Processor(self.ctx, "ec_id", enable_locks = False)
+ self.op = self.OpTest()
+ self.calc_timeout = lambda: 42
+ 
+   def testRunLU(self):
+ lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, None)
+ self.proc._ExecLU(lu)
+ 
+   def testRunLUWithPrereqError(self):
+ prereq = errors.OpPrereqError(self.op, errors.ECODE_INVAL)
+ lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, prereq)
+ self.assertRaises(errors.OpPrereqError, self.proc._LockAndExecLU,
+ lu, locking.LEVEL_CLUSTER, self.calc_timeout)
+ 
+   def testRunLUWithPrereqErrorMissingECode(self):
+ prereq = errors.OpPrereqError(self.op)
+ lu = mocks.FakeLU(self.proc, self.op, self.cfg, self.rpc, prereq)
+ self.assertRaises(errors.OpPrereqError, self.proc._LockAndExecLU,
+ lu, locking.LEVEL_CLUSTER, self.calc_timeout)
+ 
  
 +class TestSecretParams(unittest.TestCase):
 +  def testSecretParamsCheckNoError(self):
 +op = opcodes.OpInstanceCreate(
 +  instance_name="plain.example.com",
 +  pnode="master.example.com",
 +  disk_template=constants.DT_PLAIN,
 +  mode=constants.INSTANCE_CREATE,
 +  nics=[{}],
 +  disks=[{
 +constants.IDISK_SIZE: 1024
 +  }],
 +  osparams_secret= serializer.PrivateDict({"foo":"bar", "foo2":"bar2"}),
 +  os_type="debian-image")
 +
 +try:
 +  mcpu._CheckSecretParameters(op)
 +except errors.OpPrereqError:
 +  self.fail("OpPrereqError raised unexpectedly in _CheckSecretParameters")
 +
 +  def testSecretParamsCheckWithError(self):
 +op = opcodes.OpInstanceCreate(
 +  instance_name="plain.example.com",
 +  pnode="master.example.com",
 +  disk_template=constants.DT_PLAIN,
 +  mode=constants.INSTANCE_CREATE,
 +  nics=[{}],
 +  disks=[{
 +constants.IDISK_SIZE: 1024
 +  }],
 +  osparams_secret= serializer.PrivateDict({"foo":"bar",
 +  "secret_param":""}),
 +  os_type="debian-image")
 +
 +self.assertRaises(errors.OpPrereqError, mcpu._CheckSecretParameters, op)
 +
 +
  if __name__ == "__main__":
testutils.GanetiTestProgram()

Re: [MERGE] stable-2.15 to stable-2.16

2016-05-03 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM

On Tue, May 3, 2016 at 11:18 AM, 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit 3afc662c93d2d5292874b76fcbb6aa7eb1b74f57
> Merge: eb3f956 d5d747d
> Author: Brian Foley <bpfo...@google.com>
> Date:   Tue May 3 10:59:05 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.15
>   Use socat method string compatible with <1.73 & >=1.73
>   Reduce heap when parsing & storing ConfigData 10%
>   Cancel RAPI job if the client drops the connection
>   Make JQScheduler queues more strict to avoid leaks
>
> Manually resolve merge conflicts to Constants.hs
>
> Signed-off-by: Brian Foley <bpfo...@google.com>
>
> diff --cc src/Ganeti/Constants.hs
> index 39a2f6d,09783d4..420ccb6
> --- a/src/Ganeti/Constants.hs
> +++ b/src/Ganeti/Constants.hs
> @@@ -5164,12 -5046,12 +5164,12 @@@ luxiReqAll
> ]
>
>   luxiDefCtmo :: Int
>  -luxiDefCtmo = 10
>  +luxiDefCtmo = 30
>
>   luxiDefRwto :: Int
>  -luxiDefRwto = 60
>  +luxiDefRwto = 180
>
> - -- | 'WaitForJobChange' timeout
> + -- | Luxi 'WaitForJobChange' timeout
>   luxiWfjcTimeout :: Int
>   luxiWfjcTimeout = (luxiDefRwto - 1) `div` 2
>
> @@@ -5491,7 -5369,10 +5491,15 @@@ dataCollectorsEnabledName = "enabled_da
>   dataCollectorsIntervalName :: String
>   dataCollectorsIntervalName = "data_collector_interval"
>
>  +-- * HTools tag prefixes
>  +
>  +exTagsPrefix :: String
>  +exTagsPrefix = Tags.exTagsPrefix
> ++
> + -- | The polling frequency to wait for a job status change
> + cliWfjcFrequency :: Int
> + cliWfjcFrequency = 20
> +
> + -- | Default 'WaitForJobChange' timeout in seconds
> + defaultWfjcTimeout :: Int
> + defaultWfjcTimeout = 60
>

[MERGE] stable-2.15 to stable-2.16

2016-05-03 Thread 'Brian Foley' via ganeti-devel 
commit 3afc662c93d2d5292874b76fcbb6aa7eb1b74f57
Merge: eb3f956 d5d747d
Author: Brian Foley <bpfo...@google.com>
Date:   Tue May 3 10:59:05 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Use socat method string compatible with <1.73 & >=1.73
  Reduce heap when parsing & storing ConfigData 10%
  Cancel RAPI job if the client drops the connection
  Make JQScheduler queues more strict to avoid leaks

Manually resolve merge conflicts to Constants.hs

Signed-off-by: Brian Foley <bpfo...@google.com>

diff --cc src/Ganeti/Constants.hs
index 39a2f6d,09783d4..420ccb6
--- a/src/Ganeti/Constants.hs
+++ b/src/Ganeti/Constants.hs
@@@ -5164,12 -5046,12 +5164,12 @@@ luxiReqAll 
]
  
  luxiDefCtmo :: Int
 -luxiDefCtmo = 10
 +luxiDefCtmo = 30
  
  luxiDefRwto :: Int
 -luxiDefRwto = 60
 +luxiDefRwto = 180
  
- -- | 'WaitForJobChange' timeout
+ -- | Luxi 'WaitForJobChange' timeout
  luxiWfjcTimeout :: Int
  luxiWfjcTimeout = (luxiDefRwto - 1) `div` 2
  
@@@ -5491,7 -5369,10 +5491,15 @@@ dataCollectorsEnabledName = "enabled_da
  dataCollectorsIntervalName :: String
  dataCollectorsIntervalName = "data_collector_interval"
  
 +-- * HTools tag prefixes
 +
 +exTagsPrefix :: String
 +exTagsPrefix = Tags.exTagsPrefix
++
+ -- | The polling frequency to wait for a job status change
+ cliWfjcFrequency :: Int
+ cliWfjcFrequency = 20
+ 
+ -- | Default 'WaitForJobChange' timeout in seconds
+ defaultWfjcTimeout :: Int
+ defaultWfjcTimeout = 60

Re: [MERGE] stable-2.15 to stable 2.16

2016-04-22 Thread 'Brian Foley' via ganeti-devel 
On Fri, Apr 22, 2016 at 02:42:51PM +0100, 'Viktor Bachraty' via ganeti-devel 
wrote:
> commit 008fd45761cde7979ba56d823fe79d94b032b91c
> Merge: ed5786c b3c0fb0
> Author: Viktor Bachraty <vbachr...@google.com>
> Date:   Fri Apr 22 14:38:20 2016 +0100
> 
>     Merge branch 'stable-2.15' into stable-2.16
> 
> * stable-2.15
>   Allow newer cabal deps for latest Debian/Ubuntu

LGTM.

[MERGE] stable-2.15 to stable 2.16

2016-04-22 Thread 'Viktor Bachraty' via ganeti-devel 
commit 008fd45761cde7979ba56d823fe79d94b032b91c
Merge: ed5786c b3c0fb0
Author: Viktor Bachraty <vbachr...@google.com>
Date:   Fri Apr 22 14:38:20 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Allow newer cabal deps for latest Debian/Ubuntu

Re: [MERGE] stable-2.15 to stable 2.16

2016-04-12 Thread 'Viktor Bachraty' via ganeti-devel 
LGTM

On Tuesday, April 12, 2016 at 4:20:28 PM UTC+1, Brian Foley wrote:
>
> commit e08fb1113fbf918096edab60b0fc6f666e3b35e6 
> Merge: 750467a 3b0794a 
> Author: Brian Foley <bpfo...@google.com> 
> Date:   Tue Apr 12 16:09:56 2016 +0100 
>
>     Merge branch 'stable-2.15' into stable-2.16 
> 
> * stable-2.15 
>   Update misleading comment in noded RPC code 
>   Fix ganeti-rapi/noded exit-under-load bug 
>   Remove extremely frequent debug logging message 
>   Fix ClusterVerifyConfig() causing high mem usage 
>   Use threaded runtime when linking Haskell unit tests 
>   Give JQueue test dirs unique prefixes so they can't conflict 
>   Fix hlint errors 
>   Fix pylint errors 
>   Update Hlint rules 
>   Hide TH "Loading package" messages when compiling GHC <7.10 
> 
> Manually fixup merge conflicts. 
> 
> Signed-off-by: Brian Foley <bpfo...@google.com> 
>
>

[MERGE] stable-2.15 to stable 2.16

2016-04-12 Thread 'Brian Foley' via ganeti-devel 
commit e08fb1113fbf918096edab60b0fc6f666e3b35e6
Merge: 750467a 3b0794a
Author: Brian Foley <bpfo...@google.com>
Date:   Tue Apr 12 16:09:56 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Update misleading comment in noded RPC code
  Fix ganeti-rapi/noded exit-under-load bug
  Remove extremely frequent debug logging message
  Fix ClusterVerifyConfig() causing high mem usage
  Use threaded runtime when linking Haskell unit tests
  Give JQueue test dirs unique prefixes so they can't conflict
  Fix hlint errors
  Fix pylint errors
  Update Hlint rules
  Hide TH "Loading package" messages when compiling GHC <7.10

Manually fixup merge conflicts.

Signed-off-by: Brian Foley <bpfo...@google.com>

Re: Merge 'stable-2.15' into 'stable-2.16'

2016-01-05 Thread 'Helga Velroyen' via ganeti-devel 
Thanks, will send a separate patch for the nit.

On Mon, 4 Jan 2016 at 17:34 Hrvoje Ribicic <r...@google.com> wrote:

> LGTM - the nit might be better removed in a separate patch though.
>
> On Mon, Jan 4, 2016 at 5:11 PM, 'Helga Velroyen' via ganeti-devel <
> ganeti-devel@googlegroups.com> wrote:
>
>> commit 1f87aa036cd887e15240415d73c5ea5fc5b2e18a
>> Merge: ceb09b5 625c8ea
>> Author: Helga Velroyen <hel...@google.com>
>> Date:   Mon Jan 4 17:07:50 2016 +0100
>>
>> Merge branch 'stable-2.15' into stable-2.16
>>
>> * stable-2.15
>>   Add more documentation to testutils_ssh.py
>>   renew-crypto: use bulk-removal of SSH keys
>>   Use bulk-removal of SSH keys for single keys
>>   Bulk-removing SSH keys of diverse set of nodes
>>   Bulk-removal of SSH keys of normal nodes
>>   Bulk-remove SSH keys of potential master candidates
>>   Bulk-removal of SSH keys
>>   testutils: add keys to own 'authorized_keys' file
>>   Make mock SSH file manager deal with lists
>>   Don't deepcopy the config if the old value is not needed
>>   Revision bump for 2.15.2
>>   Update NEWS file for 2.15.2
>>   Compute lock allocation strictly
>>
>> * stable-2.14
>>   Revision bump for 2.14.2
>>   Update NEWS file for 2.14.2
>>   Fix lines with more than 80 characters
>>   Add more detach/attach sequence tests
>>   Allow disk attachment to diskless instances
>>   Improve tests for attaching disks
>>
>> * stable-2.13
>>   Revision bump for 2.13.3
>>   Update NEWS file for 2.13.3
>>
>> * stable-2.12
>>   Bump revision number for 2.12.6
>>   Update NEWS file for 2.12.6
>>   Restrict showing of DRBD secret using types
>>   Calculate correct affected nodes set in InstanceChangeGroup
>>
>> * stable-2.11
>>   Revision bump for 2.11.8
>>   Update NEWS file for 2.11.8
>>
>> * stable-2.10
>>   Version bump for 2.10.8
>>   Update NEWS file for 2.10.8
>>
>> * stable-2.9
>>   Bump revision number
>>   Update NEWS file for 2.9.7 release
>>   Improve RAPI section on security
>>   QA: Ensure the DRBD secret is not retrievable via RAPI
>>   Redact the DRBD secret in instance queries
>>   Do not attempt to use the DRBD secret in gnt-instance info
>>
>> Conflicts:
>>   NEWS
>>   configure.ac
>>
>> Resolutions:
>>   NEWS: merge contents in right order
>>   configure.ac: keep version number of 2.16
>>
>> diff --cc NEWS
>> index 898a739,f212ca2..3e8e00a
>> --- a/NEWS
>> +++ b/NEWS
>> @@@ -2,55 -2,87 +2,137 @@@ New
>>   
>>
>>
>>  +Version 2.16.0 beta2
>>  +
>>  +
>>  +*(unreleased)*
>>  +
>>  +Incompatible/important changes
>>  +~~
>>  +
>>  +- The options ``--no-node-setup`` of ``gnt-node add`` is disabled.
>>
>
> Nit: s/options/option/
>
>
>>  +  Instead, the cluster configuration parameter ``modify_ssh_setup`` is
>>  +  used to determine whether or not to manipulate the SSH setup of a new
>>  +  node.
>>  +
>>  +
>>  +Version 2.16.0 beta1
>>  +
>>  +
>>  +*(Released Tue, 28 Jul 2015)*
>>  +
>>  +Incompatible/important changes
>>  +~~
>>  +
>>  +- The IAllocator protocol has been extended by a new
>> ``allocate-secondary``
>>  +  request type. Currently, this new request type is only used when in
>> disk
>>  +  conversion to DRBD no secondary node is specified. As long as this new
>>  +  feature is not used, a third-party IAllocator not aware of this
>> extension can
>>  +  be continued to be used.
>>  +- ``htools`` now also take into account N+1 redundancy for plain and
>> shared
>>  +  storage. To obtain the old behavior, add the ``--no-capacity-checks``
>> option.
>>  +- ``hail`` now tries to keep the overall cluster balanced; in
>> particular it
>>  +  now prefers more empty groups over groups that are internally more
>> balanced.
>>  +
>>  +New features
>>  +
>>  +
>>  +- ``hbal`` can now be made aware of common causes of failures (for
>>  +  nodes). Look at ``hbal`` man page's LOCATION TAGS section for more
>> details.
>>  +- ``hbal`` can now be made aware of desired location for instances. Look
>

Re: Merge 'stable-2.15' into 'stable-2.16'

2016-01-04 Thread 'Helga Velroyen' via ganeti-devel 
On Mon, 4 Jan 2016 at 17:11 Helga Velroyen <hel...@google.com> wrote:

> commit 1f87aa036cd887e15240415d73c5ea5fc5b2e18a
> Merge: ceb09b5 625c8ea
> Author: Helga Velroyen <hel...@google.com>
> Date:   Mon Jan 4 17:07:50 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.15
>   Add more documentation to testutils_ssh.py
>   renew-crypto: use bulk-removal of SSH keys
>   Use bulk-removal of SSH keys for single keys
>   Bulk-removing SSH keys of diverse set of nodes
>   Bulk-removal of SSH keys of normal nodes
>   Bulk-remove SSH keys of potential master candidates
>   Bulk-removal of SSH keys
>   testutils: add keys to own 'authorized_keys' file
>   Make mock SSH file manager deal with lists
>   Don't deepcopy the config if the old value is not needed
>   Revision bump for 2.15.2
>   Update NEWS file for 2.15.2
>   Compute lock allocation strictly
>
> * stable-2.14
>   Revision bump for 2.14.2
>   Update NEWS file for 2.14.2
>   Fix lines with more than 80 characters
>   Add more detach/attach sequence tests
>   Allow disk attachment to diskless instances
>   Improve tests for attaching disks
>
> * stable-2.13
>   Revision bump for 2.13.3
>   Update NEWS file for 2.13.3
>
> * stable-2.12
>   Bump revision number for 2.12.6
>   Update NEWS file for 2.12.6
>   Restrict showing of DRBD secret using types
>   Calculate correct affected nodes set in InstanceChangeGroup
>
> * stable-2.11
>   Revision bump for 2.11.8
>   Update NEWS file for 2.11.8
>
> * stable-2.10
>   Version bump for 2.10.8
>   Update NEWS file for 2.10.8
>
> * stable-2.9
>   Bump revision number
>   Update NEWS file for 2.9.7 release
>   Improve RAPI section on security
>   QA: Ensure the DRBD secret is not retrievable via RAPI
>   Redact the DRBD secret in instance queries
>   Do not attempt to use the DRBD secret in gnt-instance info
>
> Conflicts:
>   NEWS
>   configure.ac
>
> Resolutions:
>   NEWS: merge contents in right order
>   configure.ac: keep version number of 2.16
>
> diff --cc NEWS
> index 898a739,f212ca2..3e8e00a
> --- a/NEWS
> +++ b/NEWS
> @@@ -2,55 -2,87 +2,137 @@@ New
>   
>
>
>  +Version 2.16.0 beta2
>  +
>  +
>  +*(unreleased)*
>  +
>  +Incompatible/important changes
>  +~~
>  +
>  +- The options ``--no-node-setup`` of ``gnt-node add`` is disabled.
>  +  Instead, the cluster configuration parameter ``modify_ssh_setup`` is
>  +  used to determine whether or not to manipulate the SSH setup of a new
>  +  node.
>  +
>  +
>  +Version 2.16.0 beta1
>  +
>  +
>  +*(Released Tue, 28 Jul 2015)*
>  +
>  +Incompatible/important changes
>  +~~
>  +
>  +- The IAllocator protocol has been extended by a new
> ``allocate-secondary``
>  +  request type. Currently, this new request type is only used when in
> disk
>  +  conversion to DRBD no secondary node is specified. As long as this new
>  +  feature is not used, a third-party IAllocator not aware of this
> extension can
>  +  be continued to be used.
>  +- ``htools`` now also take into account N+1 redundancy for plain and
> shared
>  +  storage. To obtain the old behavior, add the ``--no-capacity-checks``
> option.
>  +- ``hail`` now tries to keep the overall cluster balanced; in particular
> it
>  +  now prefers more empty groups over groups that are internally more
> balanced.
>  +
>  +New features
>  +
>  +
>  +- ``hbal`` can now be made aware of common causes of failures (for
>  +  nodes). Look at ``hbal`` man page's LOCATION TAGS section for more
> details.
>  +- ``hbal`` can now be made aware of desired location for instances. Look
>  +  at ``hbal`` man page's DESIRED LOCATION TAGS section for more details.
>  +- Secret parameters are now readacted in job files
>  +
>  +New dependencies
>  +
>  +
>  +- Using the metadata daemon now requires the presence of the 'setcap'
> utility.
>  +  On Debian-based systems, it is available as a part of the 'libcap2-bin'
>  +  package.
>  +
>  +
> + Version 2.15.2
> + --
> +
> + *(Released Wed, 16 Dec 2015)*
> +
> + Important changes and security notes
> + 
> +
> + Security release.
> +
> + CVE-2015-7944
> +
> + Ganeti provides a RESTful control interface called the RAPI. Its HTTPS
> + implementation is vulnerable to D

Merge 'stable-2.15' into 'stable-2.16'

2016-01-04 Thread 'Helga Velroyen' via ganeti-devel 
commit 1f87aa036cd887e15240415d73c5ea5fc5b2e18a
Merge: ceb09b5 625c8ea
Author: Helga Velroyen <hel...@google.com>
Date:   Mon Jan 4 17:07:50 2016 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Add more documentation to testutils_ssh.py
  renew-crypto: use bulk-removal of SSH keys
  Use bulk-removal of SSH keys for single keys
  Bulk-removing SSH keys of diverse set of nodes
  Bulk-removal of SSH keys of normal nodes
  Bulk-remove SSH keys of potential master candidates
  Bulk-removal of SSH keys
  testutils: add keys to own 'authorized_keys' file
  Make mock SSH file manager deal with lists
  Don't deepcopy the config if the old value is not needed
  Revision bump for 2.15.2
  Update NEWS file for 2.15.2
  Compute lock allocation strictly

* stable-2.14
  Revision bump for 2.14.2
  Update NEWS file for 2.14.2
  Fix lines with more than 80 characters
  Add more detach/attach sequence tests
  Allow disk attachment to diskless instances
  Improve tests for attaching disks

* stable-2.13
  Revision bump for 2.13.3
  Update NEWS file for 2.13.3

* stable-2.12
  Bump revision number for 2.12.6
  Update NEWS file for 2.12.6
  Restrict showing of DRBD secret using types
  Calculate correct affected nodes set in InstanceChangeGroup

* stable-2.11
  Revision bump for 2.11.8
  Update NEWS file for 2.11.8

* stable-2.10
  Version bump for 2.10.8
  Update NEWS file for 2.10.8

* stable-2.9
  Bump revision number
  Update NEWS file for 2.9.7 release
  Improve RAPI section on security
  QA: Ensure the DRBD secret is not retrievable via RAPI
  Redact the DRBD secret in instance queries
  Do not attempt to use the DRBD secret in gnt-instance info

Conflicts:
  NEWS
  configure.ac

Resolutions:
  NEWS: merge contents in right order
  configure.ac: keep version number of 2.16

diff --cc NEWS
index 898a739,f212ca2..3e8e00a
--- a/NEWS
+++ b/NEWS
@@@ -2,55 -2,87 +2,137 @@@ New
  


 +Version 2.16.0 beta2
 +
 +
 +*(unreleased)*
 +
 +Incompatible/important changes
 +~~
 +
 +- The options ``--no-node-setup`` of ``gnt-node add`` is disabled.
 +  Instead, the cluster configuration parameter ``modify_ssh_setup`` is
 +  used to determine whether or not to manipulate the SSH setup of a new
 +  node.
 +
 +
 +Version 2.16.0 beta1
 +
 +
 +*(Released Tue, 28 Jul 2015)*
 +
 +Incompatible/important changes
 +~~
 +
 +- The IAllocator protocol has been extended by a new
``allocate-secondary``
 +  request type. Currently, this new request type is only used when in disk
 +  conversion to DRBD no secondary node is specified. As long as this new
 +  feature is not used, a third-party IAllocator not aware of this
extension can
 +  be continued to be used.
 +- ``htools`` now also take into account N+1 redundancy for plain and
shared
 +  storage. To obtain the old behavior, add the ``--no-capacity-checks``
option.
 +- ``hail`` now tries to keep the overall cluster balanced; in particular
it
 +  now prefers more empty groups over groups that are internally more
balanced.
 +
 +New features
 +
 +
 +- ``hbal`` can now be made aware of common causes of failures (for
 +  nodes). Look at ``hbal`` man page's LOCATION TAGS section for more
details.
 +- ``hbal`` can now be made aware of desired location for instances. Look
 +  at ``hbal`` man page's DESIRED LOCATION TAGS section for more details.
 +- Secret parameters are now readacted in job files
 +
 +New dependencies
 +
 +
 +- Using the metadata daemon now requires the presence of the 'setcap'
utility.
 +  On Debian-based systems, it is available as a part of the 'libcap2-bin'
 +  package.
 +
 +
+ Version 2.15.2
+ --
+
+ *(Released Wed, 16 Dec 2015)*
+
+ Important changes and security notes
+ 
+
+ Security release.
+
+ CVE-2015-7944
+
+ Ganeti provides a RESTful control interface called the RAPI. Its HTTPS
+ implementation is vulnerable to DoS attacks via client-initiated SSL
+ parameter renegotiation. While the interface is not meant to be exposed
+ publicly, due to the fact that it binds to all interfaces, we believe
+ some users might be exposing it unintentionally and are vulnerable. A
+ DoS attack can consume resources meant for Ganeti daemons and instances
+ running on the master node, making both perform badly.
+
+ Fixes are not feasible due to the OpenSSL Python library not exposing
+ functionality needed to disable client-side renegotiation. Instead, we
+ offer instructions on how to control RAPI's exposure, along with info
+ on how RAPI can be setup alongside an HTTPS proxy in case users still
+ want or need to expose the RAPI interface. The instructions are
+ outlined in Ganeti's security document: do

Re: Merge 'stable-2.15' into 'stable-2.16'

2016-01-04 Thread 'Hrvoje Ribicic' via ganeti-devel 
LGTM - the nit might be better removed in a separate patch though.

On Mon, Jan 4, 2016 at 5:11 PM, 'Helga Velroyen' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit 1f87aa036cd887e15240415d73c5ea5fc5b2e18a
> Merge: ceb09b5 625c8ea
> Author: Helga Velroyen <hel...@google.com>
> Date:   Mon Jan 4 17:07:50 2016 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.15
>   Add more documentation to testutils_ssh.py
>   renew-crypto: use bulk-removal of SSH keys
>   Use bulk-removal of SSH keys for single keys
>   Bulk-removing SSH keys of diverse set of nodes
>   Bulk-removal of SSH keys of normal nodes
>   Bulk-remove SSH keys of potential master candidates
>   Bulk-removal of SSH keys
>   testutils: add keys to own 'authorized_keys' file
>   Make mock SSH file manager deal with lists
>   Don't deepcopy the config if the old value is not needed
>   Revision bump for 2.15.2
>   Update NEWS file for 2.15.2
>   Compute lock allocation strictly
>
> * stable-2.14
>   Revision bump for 2.14.2
>   Update NEWS file for 2.14.2
>   Fix lines with more than 80 characters
>   Add more detach/attach sequence tests
>   Allow disk attachment to diskless instances
>   Improve tests for attaching disks
>
> * stable-2.13
>   Revision bump for 2.13.3
>   Update NEWS file for 2.13.3
>
> * stable-2.12
>   Bump revision number for 2.12.6
>   Update NEWS file for 2.12.6
>   Restrict showing of DRBD secret using types
>   Calculate correct affected nodes set in InstanceChangeGroup
>
> * stable-2.11
>   Revision bump for 2.11.8
>   Update NEWS file for 2.11.8
>
> * stable-2.10
>   Version bump for 2.10.8
>   Update NEWS file for 2.10.8
>
> * stable-2.9
>   Bump revision number
>   Update NEWS file for 2.9.7 release
>   Improve RAPI section on security
>   QA: Ensure the DRBD secret is not retrievable via RAPI
>   Redact the DRBD secret in instance queries
>   Do not attempt to use the DRBD secret in gnt-instance info
>
> Conflicts:
>   NEWS
>   configure.ac
>
> Resolutions:
>   NEWS: merge contents in right order
>   configure.ac: keep version number of 2.16
>
> diff --cc NEWS
> index 898a739,f212ca2..3e8e00a
> --- a/NEWS
> +++ b/NEWS
> @@@ -2,55 -2,87 +2,137 @@@ New
>   
>
>
>  +Version 2.16.0 beta2
>  +
>  +
>  +*(unreleased)*
>  +
>  +Incompatible/important changes
>  +~~
>  +
>  +- The options ``--no-node-setup`` of ``gnt-node add`` is disabled.
>

Nit: s/options/option/


>  +  Instead, the cluster configuration parameter ``modify_ssh_setup`` is
>  +  used to determine whether or not to manipulate the SSH setup of a new
>  +  node.
>  +
>  +
>  +Version 2.16.0 beta1
>  +
>  +
>  +*(Released Tue, 28 Jul 2015)*
>  +
>  +Incompatible/important changes
>  +~~
>  +
>  +- The IAllocator protocol has been extended by a new
> ``allocate-secondary``
>  +  request type. Currently, this new request type is only used when in
> disk
>  +  conversion to DRBD no secondary node is specified. As long as this new
>  +  feature is not used, a third-party IAllocator not aware of this
> extension can
>  +  be continued to be used.
>  +- ``htools`` now also take into account N+1 redundancy for plain and
> shared
>  +  storage. To obtain the old behavior, add the ``--no-capacity-checks``
> option.
>  +- ``hail`` now tries to keep the overall cluster balanced; in particular
> it
>  +  now prefers more empty groups over groups that are internally more
> balanced.
>  +
>  +New features
>  +
>  +
>  +- ``hbal`` can now be made aware of common causes of failures (for
>  +  nodes). Look at ``hbal`` man page's LOCATION TAGS section for more
> details.
>  +- ``hbal`` can now be made aware of desired location for instances. Look
>  +  at ``hbal`` man page's DESIRED LOCATION TAGS section for more details.
>  +- Secret parameters are now readacted in job files
>  +
>  +New dependencies
>  +
>  +
>  +- Using the metadata daemon now requires the presence of the 'setcap'
> utility.
>  +  On Debian-based systems, it is available as a part of the 'libcap2-bin'
>  +  package.
>  +
>  +
> + Version 2.15.2
> + --
> +
> + *(Released Wed, 16 Dec 2015)*
> +
> + Important changes and security notes
> + 
> +
> + Security release.
> +
> + CVE-2

Re: Merge 'stable-2.15' to 'stable-2.16' (2nd try)

2015-11-20 Thread 'Hrvoje Ribicic' via ganeti-devel 
LGTM, thanks

On Fri, Nov 20, 2015 at 2:08 PM, 'Helga Velroyen' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit 8337653769f617abfe39876d7cb794d68064ab13
> Merge: 2ebf4e8 598897c
> Author: Helga Velroyen <hel...@google.com>
> Date:   Fri Nov 20 11:34:44 2015 +0100
>
> Merge branch 'stable-2.15' into stable-2.16
>
> * stable-2.15
>   Document the decission why optimisation is turned off
>   Don't keep input for error messages
>   Use dict.copy instead of deepcopy
>   Use bulk-adding of keys in renew-crypto
>   Make NodeSshKeyAdd use its *Bulk companion
>   Unit test bulk-adding normal nodes
>   Unit test for bulk-adding pot. master candidates
>   Introduce bulk-adding of SSH keys
>   Pause watcher during performance QA
>   Send answers strictly
>   Store keys as ByteStrings
>   Encode UUIDs as ByteStrings
>   Prefer the UuidObject type class over specific functions
>   Assign the variables before use (bugfix for dee6adb9)
>   Extend QA to detect autopromotion errors
>   Handle SSH key distribution on auto promotion
>   Do not remove authorized key of node itself
>   Fix indentation
>   Support force option for deactivate disks on RAPI
>
> * stable-2.14
>   Fix faulty iallocator type check
>   Improve cfgupgrade output in case of errors
>
> * stable-2.13
>   Extend timeout for gnt-cluster renew-crypto
>   Reduce flakyness of GetCmdline test on slow machines
>   Remove duplicated words
>
> * stable-2.12
>   Revert "Also consider connection time out a network error"
>   Clone lists before modifying
>   Make lockConfig call retryable
>   Return the correct error code in the post-upgrade script
>   Make openssl refrain from DH altogether
>   Fix upgrades of instances with missing creation time
>
> * stable-2.11
>   (none)
>
> * stable-2.10
>   Remove -X from hspace man page
>   Make htools tolerate missing "dtotal" and "dfree" on luxi
>
> Conflicts:
>   lib/backend.py
>   lib/cmdlib/node.py
>   src/Ganeti/WConfd/ConfigModifications.hs
>
> Resolutions:
>   lib/backend.py
> use bulk-adding keys with renamed public key file variable
>   lib/cmdlib/node.py
> use self.cfg.RemoveNode rather than self.context.RemoveNode
>   src/Ganeti/WConfd/ConfigModifications.hs
> fix imports
> add UTF8.{to,from}String at appropriate places
>
> Signed-off-by: Helga Velroyen <hel...@google.com>
>
> diff --cc lib/backend.py
> index 7ebbdb9,d470060..6c51df8
> --- a/lib/backend.py
> +++ b/lib/backend.py
> @@@ -2002,21 -2073,24 +2080,23 @@@ def RenewSshKeys(node_uuids, node_names
> " (UUID %s)" % (node_name, node_uuid))
>
>   if potential_master_candidate:
>  -  ssh.RemovePublicKey(node_uuid, key_file=pub_key_file)
>  -  ssh.AddPublicKey(node_uuid, pub_key, key_file=pub_key_file)
>  +  ssh.RemovePublicKey(node_uuid, key_file=ganeti_pub_keys_file)
>  +  ssh.AddPublicKey(node_uuid, pub_key, key_file=ganeti_pub_keys_file)
>
> --logging.debug("Add ssh key of node '%s'.", node_name)
> - node_errors = AddNodeSshKey(
> - node_uuid, node_name, potential_master_candidates,
> - to_authorized_keys=master_candidate,
> - to_public_keys=potential_master_candidate,
> - get_public_keys=True,
> - pub_key_file=ganeti_pub_keys_file,
> - ssconf_store=ssconf_store,
> - noded_cert_file=noded_cert_file,
> - run_cmd_fn=run_cmd_fn)
> - if node_errors:
> -   all_node_errors = all_node_errors + node_errors
> + node_info = SshAddNodeInfo(name=node_name,
> +uuid=node_uuid,
> +to_authorized_keys=master_candidate,
> +to_public_keys=potential_master_candidate,
> +get_public_keys=True)
> + node_keys_to_add.append(node_info)
> +
> +   node_errors = AddNodeSshKeyBulk(
> +   node_keys_to_add, potential_master_candidates,
>  -  pub_key_file=pub_key_file, ssconf_store=ssconf_store,
> ++  pub_key_file=ganeti_pub_keys_file, ssconf_store=ssconf_store,
> +   noded_cert_file=noded_cert_file,
> +   run_cmd_fn=run_cmd_fn)
> +   if node_errors:
> + all_node_errors = all_node_errors + node_errors
>
> # Renewing the master node's key
>
> diff --cc lib/cmdlib/common.py
> index 696a331,1d79a3e..638abd7
> --- a/lib/cmdlib/common.py
>

Re: Merge 'stable-2.15' into 'stable-2.16'

2015-11-20 Thread 'Oleg Ponomarev' via ganeti-devel 

LGTM, but I still have two remarks:

On 11/20/2015 01:42 PM, 'Helga Velroyen' via ganeti-devel wrote:

backend.py


--logging.debug("Add ssh key of node '%s'.", node_name)


Why did you decide to remove the logging.debug call?

Disk.hs


++  addInst i = csConfigDataL . configInstancesL
++  . alterContainerL (UTF8.fromString $ uuidOf inst)
++.~ Just i


From my point of view, it's better to preserver (... $ uuidOf i) 
instead. It doesn't affect the current functionality but can prevent 
misunderstands in the future.

Merge 'stable-2.15' into 'stable-2.16'

2015-11-20 Thread 'Helga Velroyen' via ganeti-devel 
commit 2df3a2c59dc677a4a2dbe7a9fae2e6341d1d
Merge: 2ebf4e8 598897c
Author: Helga Velroyen <hel...@google.com>
Date:   Fri Nov 20 11:34:44 2015 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Document the decission why optimisation is turned off
  Don't keep input for error messages
  Use dict.copy instead of deepcopy
  Use bulk-adding of keys in renew-crypto
  Make NodeSshKeyAdd use its *Bulk companion
  Unit test bulk-adding normal nodes
  Unit test for bulk-adding pot. master candidates
  Introduce bulk-adding of SSH keys
  Pause watcher during performance QA
  Send answers strictly
  Store keys as ByteStrings
  Encode UUIDs as ByteStrings
  Prefer the UuidObject type class over specific functions
  Assign the variables before use (bugfix for dee6adb9)
  Extend QA to detect autopromotion errors
  Handle SSH key distribution on auto promotion
  Do not remove authorized key of node itself
  Fix indentation
  Support force option for deactivate disks on RAPI

* stable-2.14
  Fix faulty iallocator type check
  Improve cfgupgrade output in case of errors

* stable-2.13
  Extend timeout for gnt-cluster renew-crypto
  Reduce flakyness of GetCmdline test on slow machines
  Remove duplicated words

* stable-2.12
  Revert "Also consider connection time out a network error"
  Clone lists before modifying
  Make lockConfig call retryable
  Return the correct error code in the post-upgrade script
  Make openssl refrain from DH altogether
  Fix upgrades of instances with missing creation time

* stable-2.11
  (none)

* stable-2.10
  Remove -X from hspace man page
  Make htools tolerate missing "dtotal" and "dfree" on luxi

Conflicts:
  lib/backend.py
  lib/cmdlib/node.py
  src/Ganeti/WConfd/ConfigModifications.hs

Resolutions:
  lib/backend.py
use bulk-adding keys with renamed public key file variable
  lib/cmdlib/node.py
use self.cfg.RemoveNode rather than self.context.RemoveNode
  src/Ganeti/WConfd/ConfigModifications.hs
fix imports
add UTF8.{to,from}String at appropriate places

Signed-off-by: Helga Velroyen <hel...@google.com>

diff --cc lib/backend.py
index 7ebbdb9,d470060..6c51df8
--- a/lib/backend.py
+++ b/lib/backend.py
@@@ -2002,21 -2073,24 +2080,23 @@@ def RenewSshKeys(node_uuids, node_names
" (UUID %s)" % (node_name, node_uuid))

  if potential_master_candidate:
 -  ssh.RemovePublicKey(node_uuid, key_file=pub_key_file)
 -  ssh.AddPublicKey(node_uuid, pub_key, key_file=pub_key_file)
 +  ssh.RemovePublicKey(node_uuid, key_file=ganeti_pub_keys_file)
 +  ssh.AddPublicKey(node_uuid, pub_key, key_file=ganeti_pub_keys_file)

--logging.debug("Add ssh key of node '%s'.", node_name)
- node_errors = AddNodeSshKey(
- node_uuid, node_name, potential_master_candidates,
- to_authorized_keys=master_candidate,
- to_public_keys=potential_master_candidate,
- get_public_keys=True,
- pub_key_file=ganeti_pub_keys_file,
- ssconf_store=ssconf_store,
- noded_cert_file=noded_cert_file,
- run_cmd_fn=run_cmd_fn)
- if node_errors:
-   all_node_errors = all_node_errors + node_errors
+ node_info = SshAddNodeInfo(name=node_name,
+uuid=node_uuid,
+to_authorized_keys=master_candidate,
+to_public_keys=potential_master_candidate,
+get_public_keys=True)
+ node_keys_to_add.append(node_info)
+
+   node_errors = AddNodeSshKeyBulk(
+   node_keys_to_add, potential_master_candidates,
 -  pub_key_file=pub_key_file, ssconf_store=ssconf_store,
++  pub_key_file=ganeti_pub_keys_file, ssconf_store=ssconf_store,
+   noded_cert_file=noded_cert_file,
+   run_cmd_fn=run_cmd_fn)
+   if node_errors:
+ all_node_errors = all_node_errors + node_errors

# Renewing the master node's key

diff --cc lib/cmdlib/common.py
index 696a331,1d79a3e..638abd7
--- a/lib/cmdlib/common.py
+++ b/lib/cmdlib/common.py
@@@ -485,7 -511,12 +511,11 @@@ def AdjustCandidatePool
  lu.LogInfo("Promoted nodes to master candidate role: %s",
 utils.CommaJoin(node.name for node in mod_list))
  for node in mod_list:
 -  lu.context.ReaddNode(node)
AddNodeCertToCandidateCerts(lu, lu.cfg, node.uuid)
+   if modify_ssh_setup:
+ AddMasterCandidateSshKey(
+ lu, master_node, node, potential_master_candidates,
feedback_fn)
+
mc_now, mc_max, _ = lu.cfg.GetMasterCandidateStats(exceptions)
if mc_now > mc_max:
  lu.LogInfo("Note: more nodes are candidates (%d) than desired (%d)" %
diff --cc lib/cmdlib/nod

Merge 'stable-2.15' to 'stable-2.16' (2nd try)

2015-11-20 Thread 'Helga Velroyen' via ganeti-devel 
commit 8337653769f617abfe39876d7cb794d68064ab13
Merge: 2ebf4e8 598897c
Author: Helga Velroyen <hel...@google.com>
Date:   Fri Nov 20 11:34:44 2015 +0100

    Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Document the decission why optimisation is turned off
  Don't keep input for error messages
  Use dict.copy instead of deepcopy
  Use bulk-adding of keys in renew-crypto
  Make NodeSshKeyAdd use its *Bulk companion
  Unit test bulk-adding normal nodes
  Unit test for bulk-adding pot. master candidates
  Introduce bulk-adding of SSH keys
  Pause watcher during performance QA
  Send answers strictly
  Store keys as ByteStrings
  Encode UUIDs as ByteStrings
  Prefer the UuidObject type class over specific functions
  Assign the variables before use (bugfix for dee6adb9)
  Extend QA to detect autopromotion errors
  Handle SSH key distribution on auto promotion
  Do not remove authorized key of node itself
  Fix indentation
  Support force option for deactivate disks on RAPI

* stable-2.14
  Fix faulty iallocator type check
  Improve cfgupgrade output in case of errors

* stable-2.13
  Extend timeout for gnt-cluster renew-crypto
  Reduce flakyness of GetCmdline test on slow machines
  Remove duplicated words

* stable-2.12
  Revert "Also consider connection time out a network error"
  Clone lists before modifying
  Make lockConfig call retryable
  Return the correct error code in the post-upgrade script
  Make openssl refrain from DH altogether
  Fix upgrades of instances with missing creation time

* stable-2.11
  (none)

* stable-2.10
  Remove -X from hspace man page
  Make htools tolerate missing "dtotal" and "dfree" on luxi

Conflicts:
  lib/backend.py
  lib/cmdlib/node.py
  src/Ganeti/WConfd/ConfigModifications.hs

Resolutions:
  lib/backend.py
use bulk-adding keys with renamed public key file variable
  lib/cmdlib/node.py
use self.cfg.RemoveNode rather than self.context.RemoveNode
  src/Ganeti/WConfd/ConfigModifications.hs
fix imports
add UTF8.{to,from}String at appropriate places

Signed-off-by: Helga Velroyen <hel...@google.com>

diff --cc lib/backend.py
index 7ebbdb9,d470060..6c51df8
--- a/lib/backend.py
+++ b/lib/backend.py
@@@ -2002,21 -2073,24 +2080,23 @@@ def RenewSshKeys(node_uuids, node_names
" (UUID %s)" % (node_name, node_uuid))

  if potential_master_candidate:
 -  ssh.RemovePublicKey(node_uuid, key_file=pub_key_file)
 -  ssh.AddPublicKey(node_uuid, pub_key, key_file=pub_key_file)
 +  ssh.RemovePublicKey(node_uuid, key_file=ganeti_pub_keys_file)
 +  ssh.AddPublicKey(node_uuid, pub_key, key_file=ganeti_pub_keys_file)

--logging.debug("Add ssh key of node '%s'.", node_name)
- node_errors = AddNodeSshKey(
- node_uuid, node_name, potential_master_candidates,
- to_authorized_keys=master_candidate,
- to_public_keys=potential_master_candidate,
- get_public_keys=True,
- pub_key_file=ganeti_pub_keys_file,
- ssconf_store=ssconf_store,
- noded_cert_file=noded_cert_file,
- run_cmd_fn=run_cmd_fn)
- if node_errors:
-   all_node_errors = all_node_errors + node_errors
+ node_info = SshAddNodeInfo(name=node_name,
+uuid=node_uuid,
+to_authorized_keys=master_candidate,
+to_public_keys=potential_master_candidate,
+get_public_keys=True)
+ node_keys_to_add.append(node_info)
+
+   node_errors = AddNodeSshKeyBulk(
+   node_keys_to_add, potential_master_candidates,
 -  pub_key_file=pub_key_file, ssconf_store=ssconf_store,
++  pub_key_file=ganeti_pub_keys_file, ssconf_store=ssconf_store,
+   noded_cert_file=noded_cert_file,
+   run_cmd_fn=run_cmd_fn)
+   if node_errors:
+ all_node_errors = all_node_errors + node_errors

# Renewing the master node's key

diff --cc lib/cmdlib/common.py
index 696a331,1d79a3e..638abd7
--- a/lib/cmdlib/common.py
+++ b/lib/cmdlib/common.py
@@@ -485,7 -511,12 +511,11 @@@ def AdjustCandidatePool
  lu.LogInfo("Promoted nodes to master candidate role: %s",
 utils.CommaJoin(node.name for node in mod_list))
  for node in mod_list:
 -  lu.context.ReaddNode(node)
AddNodeCertToCandidateCerts(lu, lu.cfg, node.uuid)
+   if modify_ssh_setup:
+ AddMasterCandidateSshKey(
+ lu, master_node, node, potential_master_candidates,
feedback_fn)
+
mc_now, mc_max, _ = lu.cfg.GetMasterCandidateStats(exceptions)
if mc_now > mc_max:
  lu.LogInfo("Note: more nodes are candidates (%d) than desired (%d)" %
diff --cc lib/cmdlib/nod

Re: Merge 'stable-2.15' into 'stable-2.16'

2015-11-20 Thread 'Helga Velroyen' via ganeti-devel 
On Fri, 20 Nov 2015 at 14:00 Oleg Ponomarev  wrote:

> LGTM, but I still have two remarks:
>
> On 11/20/2015 01:42 PM, 'Helga Velroyen' via ganeti-devel wrote:
>
> backend.py
>
> > --logging.debug("Add ssh key of node '%s'.", node_name)
>
> Why did you decide to remove the logging.debug call?
>

Because with the refactoring it is not actually added there anymore (but as
a bulk at the end). I'd rather send another patch later to add a proper
message there than keeping the obsolete one here.


>
> Disk.hs
>
> > ++  addInst i = csConfigDataL . configInstancesL
> > ++  . alterContainerL (UTF8.fromString $ uuidOf inst)
> > ++.~ Just i
>
>  From my point of view, it's better to preserver (... $ uuidOf i)
> instead. It doesn't affect the current functionality but can prevent
> misunderstands in the future.
>

Actually, I seem to have sent the wrong version of this. I'll resend it
soon, but no need for you to review it. Riba was doing that already.

Thanks,
Helga


-- 

Helga Velroyen
Software Engineer
hel...@google.com

Google Germany GmbH
Dienerstraße 12
80331 München

Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind,
leiten Sie diese bitte nicht weiter, informieren Sie den Absender und
löschen Sie die E-Mail und alle Anhänge. Vielen Dank.

This e-mail is confidential. If you are not the right addressee please do
not forward it, please inform the sender, and please erase this e-mail
including any attachments. Thanks.