Re: [MERGE] stable-2.17 to master

['Federico Pareschi' via ganeti-devel]

LGTM, thanks for doing this, Brian

On Monday, December 19, 2016 at 11:33:46 AM UTC, Brian Foley wrote:
>
> commit e18785cdd56cf614fe43b603db3a2a51c8c9c2aa 
> Merge: e763a23 7f2183e 
> Author: Brian Foley  
> Date:   Mon Dec 19 11:31:06 2016 + 
>
> Merge branch 'stable-2.17' 
> 
> * stable-2.17 
>   1143: Add gnt-cluster modify --modify-ssh-setup option 
> 
> * stable-2.15 
>   Fix gnt-instance console instance unpausing for xl toolstack 
>   Disable pylint too-many-nested-blocks in _RunCmdPipe 
>   Reduce nesting in import-export ProcessChildIO 
>   Reduce nesting in LUOobCommand.Exec 
>   Reduce nesting in LUInstanceCreate.RunOsScripts 
>   Reduce nesting in RemoveNodeSshKeyBulk key calculation 
>   Reduce nesting in RemoveNodeSshKeyBulk ssh logic 
>   Reduce nesting in gnt-cluster VerifyDisks missing disk loop 
>   Reduce nesting in _CheckVLANArguments 
>   Reduce nesting in StartDaemon 
>   Disable pylint bad-continuation warning 
>   Disable pylint superfluous-parens warning 
>   Disable pylint redefined-variable-type warning 
>   Disable pylint too-many-branches warnings 
>   Disable pylint broad-except warnings 
>   Disable incorrect pylint assigning-non-slot warning 
>   Quell pylint unbalanced-tuple-unpacking warning 
>   Cleanup: Use new-style classes everywhere 
>   Quell pylint socket.timeout warning 
>   Quell the pylint wrong-import-order warnings 
>   Quell cell-var-from-loop warning 
>   Use default value lambda param to avoid cell-var-from-loop 
>   Quell too-many-boolean-expressions 
>   Remove pylint tests removed in pylint 2.0 
>   Quell trailing newline 
>   Quell bad-whitespace warning 
>   Quell consider-using-enumerate warning 
>   Disable pylint unsubscriptable-object warning 
>   Disable pylint bare-except warning 
>   Disable unwanted pylint wrong-import-position warnings 
>   Disable pylint unused-wildcard-import warning 
>   Disable incorrect pylint not-callable warning 
>   Disable pylint unpacking-non-sequence warning 
>   Disable pylint misplaced-comparison-constant warning 
>   Disable incorect pylint simplify-if-statement warning 
>   Disable pylint eval-used warning 
>   Disable pylint invalid-name warning 
>   Disable pylint import-self warning 
>   Disable some pylint unused-import warnings 
>   Replace deprecated pylint >=0.27 pragma with new form 
>   Delete old warning disables removed from pylint 1.6 
>   Fix pylint >1.4 pycurl no-member warnings 
>   Cleanup: Remove unused/duplicate module/fn import 
>   Cleanup: Fix unidiomatic-typecheck 
>   Cleanup: Remove some unneeded pylint disables 
>   Cleanup: Iterate dict rather than key list 
>   Cleanup: Remove unused format key 
>   Cleanup: StartInstance and RebootInstance return None 
>   Cleanup: Fix for/else with no break in AddAuthorizedKeys 
>   Cleanup: Replace map/filters with list comprehensions 
>   Cleanup: del is a statement not a function 
>   Cleanup: Use FOO not in BAR instead of not FOO in BAR 
>   Cleanup: Simplify boolean assignment 
>   Cleanup: Remove some unnecessary if (...) parens 
>   Fix invalid variable error for file-based disks 
>   FIX: Refactor DiagnoseOS to use a loop, not an inner fn 
>   FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined 
>   StartInstance restores instance state if running 
>   Allow migrate --cleanup to adopt an instance 
>   Make migrate --cleanup more robust 
>   Make finalize_migration_{src,dst} a single op 
>   Make FinalizeMigration{Src,Dst} more robust 
>   Fix instance state detection in _Shutdowninstance 
>   Code cleanup in hypervisor backend 
>   Fix for incorrect parsing of DRBD versions 
>   Fix for instance reinstall not updating config 
>   Change a few errors to report names, not UUIDs 
>   Give atomicWriteFile temp filenames a more distinct pattern 
>   LV check failure should print instance name 
>   Add ganeti-noded and ganeti-rapi --max-clients options 
>   Disable logging CallRPCMethod timings in non-debug configs 
>   568 Update hv_kvm to handle output from qemu >= 1.6.0 
>   Improve cluster verify ssh key errors 
> 
> Resolve merge conflicts: 
>   lib/cmdlib/common.py 
> caused by 81a68a3 use names not uuids in warnings 
> 
>   lib/rapi/testutils.py 
> trivial module import tweaks 
> 
>   lib/server/rapi.py 
> caused by c1e18c4 --max-clients and pam options 
> 
>   lib/storage/drbd_info.py 
>   test/py/ganeti.storage.drbd_unittest.py 
> caused by 49980c6 DRBD version parsing -- fixed differently (and 
> less flexibly) on master. Use 2.17's version 
> 
>   test/hs/Test/Ganeti/OpCodes.hs 
> caused by e763a23 

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit e18785cdd56cf614fe43b603db3a2a51c8c9c2aa
Merge: e763a23 7f2183e
Author: Brian Foley 
Date:   Mon Dec 19 11:31:06 2016 +

Merge branch 'stable-2.17'

* stable-2.17
  1143: Add gnt-cluster modify --modify-ssh-setup option

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors

Resolve merge conflicts:
  lib/cmdlib/common.py
caused by 81a68a3 use names not uuids in warnings

  lib/rapi/testutils.py
trivial module import tweaks

  lib/server/rapi.py
caused by c1e18c4 --max-clients and pam options

  lib/storage/drbd_info.py
  test/py/ganeti.storage.drbd_unittest.py
caused by 49980c6 DRBD version parsing -- fixed differently (and
less flexibly) on master. Use 2.17's version

  test/hs/Test/Ganeti/OpCodes.hs
caused by e763a23 improve test data generation touching code
code in 2.16, and modify_ssh_setup cluster param

Signed-off-by: Brian Foley 

diff --cc lib/cmdlib/common.py
index 95a1186,b6e673e..08c6e4a
--- a/lib/cmdlib/common.py
+++ b/lib/cmdlib/common.py
@@@ -1600,10 -1595,9 +1601,10 @@@ def EnsureKvmdOnNodes(lu, feedback_fn, 
# Stop KVM where necessary
if stop_nodes:
  

Re: [MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

LGTM, thanks!

On Monday, August 22, 2016 at 12:31:00 PM UTC+1, Brian Foley wrote:
>
> commit 310c43061f6fdac2e5c10fea1466066fd4e0655e 
> Merge: aa50df9 3be37ca 
> Author: Brian Foley  
> Date:   Mon Aug 22 12:23:36 2016 +0100 
>
> Merge branch 'stable-2.17' 
> 
> * stable-2.17 
>   Fix hlint warnings 
> 
> * stable-2.16 
>   Fix inconsistent spaces vs tabs indent in makefile 
>   Fix some typos/poor phrasing in gnt-node man page 
>   Add missing cluster modify --modify-etc-hosts to man 
>   Fix/quell hlint warnings 
> 
> * stable-2.13 
>   Bugfix: migrate needs HypervisorClass, not an instance 
> 
> Signed-off-by: Brian Foley  
>
>

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit 310c43061f6fdac2e5c10fea1466066fd4e0655e
Merge: aa50df9 3be37ca
Author: Brian Foley 
Date:   Mon Aug 22 12:23:36 2016 +0100

Merge branch 'stable-2.17'

* stable-2.17
  Fix hlint warnings

* stable-2.16
  Fix inconsistent spaces vs tabs indent in makefile
  Fix some typos/poor phrasing in gnt-node man page
  Add missing cluster modify --modify-etc-hosts to man
  Fix/quell hlint warnings

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley 

Re: [MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

LGTM, thanks!

On Tuesday, August 16, 2016 at 2:06:44 PM UTC+1, Brian Foley wrote:
>
> commit a98bd0ea60e7586b2dc351d859869025f1815398 
> Merge: dadafed 3b5c416 
> Author: Brian Foley  
> Date:   Tue Aug 16 14:04:32 2016 +0100 
>
> Merge branch 'stable-2.17' 
> 
> * stable-2.17 
>   Introduce a non-strict flag for group verify disk 
> 
> * stable-2.16 
>   Improve error reporting in _VerifyClientCertificates 
>   Simplify some inscrutable map/map/ifilter/zip code 
>   Avoid overuse of operator in watcher *.py 
>   Sprinkle some more list comprehensions 
>   Replace map/partial with list comprehension 
>   Replace uses of map/lambda with more Pythonic code 
>   Replace map(operator.attrgetter, ...) uses 
>   Fix typos in gnt-cluster man page 
>   Hide errors for expected inotify failures in unittest 
>   Add gnt-instance rename --force option 
>   Improve documentation for gnt-instance failover 
>   Allow master failover to ignore offline nodes 
>   Fix LogicalVolume code to work with older /sbin/lvs 
>   Shorten verifyMasterVote failure message 
>   Adding a confirmation before gnt-node --offline no 
>   Removed unnecessary dependency from rpc in cli 
>   Refactor cli exception to its appropriate module 
>   Clean-up of code and fix of pylint warnings 
>   Use fork instead of spawnv in the watcher 
>   Make 'make pep8' happy 
> 
> Manually fix conflicts in lib/errors.py 
> 
> Signed-off-by: Brian Foley  
>
> diff --cc lib/client/gnt_cluster.py 
> index 00d5a46,2cc8328..7c575c0 
> --- a/lib/client/gnt_cluster.py 
> +++ b/lib/client/gnt_cluster.py 
> @@@ -2515,10 -2523,11 +2523,10 @@@ commands = 
> "verify": ( 
>   VerifyCluster, ARGS_NONE, 
>   [VERBOSE_OPT, DEBUG_SIMERR_OPT, ERROR_CODES_OPT, NONPLUS1_OPT, 
>  - DRY_RUN_OPT, PRIORITY_OPT, NODEGROUP_OPT, IGNORE_ERRORS_OPT, 
>  - VERIFY_CLUTTER_OPT], 
>  + PRIORITY_OPT, NODEGROUP_OPT, IGNORE_ERRORS_OPT, 
> VERIFY_CLUTTER_OPT], 
>   "", "Does a check on the cluster configuration"), 
> "verify-disks": ( 
> - VerifyDisks, ARGS_NONE, [PRIORITY_OPT, NODEGROUP_OPT], 
> + VerifyDisks, ARGS_NONE, [PRIORITY_OPT, NODEGROUP_OPT, STRICT_OPT], 
>   "", "Does a check on the cluster disk status"), 
> "repair-disk-sizes": ( 
>   RepairDiskSizes, ARGS_MANY_INSTANCES, [DRY_RUN_OPT, PRIORITY_OPT], 
> diff --cc lib/errors.py 
> index 048b377,826e761..5fb885b 
> --- a/lib/errors.py 
> +++ b/lib/errors.py 
> @@@ -469,12 -469,17 +469,22 @@@ class SshUpdateError(GenericError) 
> """ 
>   
>   
>  +class PamRapiAuthError(GenericError): 
>  +  """Error in PAM remote API authenticator initialization. 
>  + 
>  +  """ 
>  + 
>  + 
> + class JobSubmittedException(Exception): 
> +   """Job was submitted, client should exit. 
> + 
> +   This exception has one argument, the ID of the job that was 
> +   submitted. The handler should print this ID. 
> + 
> +   This is not an error, just a structured way to exit from clients. 
>  - 
> +   """ 
> + 
> + 
>   # errors should be added above 
>   
>   
>

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit a98bd0ea60e7586b2dc351d859869025f1815398
Merge: dadafed 3b5c416
Author: Brian Foley 
Date:   Tue Aug 16 14:04:32 2016 +0100

Merge branch 'stable-2.17'

* stable-2.17
  Introduce a non-strict flag for group verify disk

* stable-2.16
  Improve error reporting in _VerifyClientCertificates
  Simplify some inscrutable map/map/ifilter/zip code
  Avoid overuse of operator in watcher *.py
  Sprinkle some more list comprehensions
  Replace map/partial with list comprehension
  Replace uses of map/lambda with more Pythonic code
  Replace map(operator.attrgetter, ...) uses
  Fix typos in gnt-cluster man page
  Hide errors for expected inotify failures in unittest
  Add gnt-instance rename --force option
  Improve documentation for gnt-instance failover
  Allow master failover to ignore offline nodes
  Fix LogicalVolume code to work with older /sbin/lvs
  Shorten verifyMasterVote failure message
  Adding a confirmation before gnt-node --offline no
  Removed unnecessary dependency from rpc in cli
  Refactor cli exception to its appropriate module
  Clean-up of code and fix of pylint warnings
  Use fork instead of spawnv in the watcher
  Make 'make pep8' happy

Manually fix conflicts in lib/errors.py

Signed-off-by: Brian Foley 

diff --cc lib/client/gnt_cluster.py
index 00d5a46,2cc8328..7c575c0
--- a/lib/client/gnt_cluster.py
+++ b/lib/client/gnt_cluster.py
@@@ -2515,10 -2523,11 +2523,10 @@@ commands = 
"verify": (
  VerifyCluster, ARGS_NONE,
  [VERBOSE_OPT, DEBUG_SIMERR_OPT, ERROR_CODES_OPT, NONPLUS1_OPT,
 - DRY_RUN_OPT, PRIORITY_OPT, NODEGROUP_OPT, IGNORE_ERRORS_OPT,
 - VERIFY_CLUTTER_OPT],
 + PRIORITY_OPT, NODEGROUP_OPT, IGNORE_ERRORS_OPT, VERIFY_CLUTTER_OPT],
  "", "Does a check on the cluster configuration"),
"verify-disks": (
- VerifyDisks, ARGS_NONE, [PRIORITY_OPT, NODEGROUP_OPT],
+ VerifyDisks, ARGS_NONE, [PRIORITY_OPT, NODEGROUP_OPT, STRICT_OPT],
  "", "Does a check on the cluster disk status"),
"repair-disk-sizes": (
  RepairDiskSizes, ARGS_MANY_INSTANCES, [DRY_RUN_OPT, PRIORITY_OPT],
diff --cc lib/errors.py
index 048b377,826e761..5fb885b
--- a/lib/errors.py
+++ b/lib/errors.py
@@@ -469,12 -469,17 +469,22 @@@ class SshUpdateError(GenericError)
"""
  
  
 +class PamRapiAuthError(GenericError):
 +  """Error in PAM remote API authenticator initialization.
 +
 +  """
 +
 +
+ class JobSubmittedException(Exception):
+   """Job was submitted, client should exit.
+ 
+   This exception has one argument, the ID of the job that was
+   submitted. The handler should print this ID.
+ 
+   This is not an error, just a structured way to exit from clients.
 -
+   """
+ 
+ 
  # errors should be added above
  
  

Re: [MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

LGTM, thanks.

On Thursday, July 7, 2016 at 1:43:04 PM UTC+1, Brian Foley wrote:
>
> commit 3d17266c14d6e021253844b199b9518068875e4f 
> Merge: b7a0dde a824a2e 
> Author: Brian Foley  
> Date:   Thu Jul 7 13:34:40 2016 +0100 
>
> Merge branch 'stable-2.17' 
> 
> * stable-2.17 
>   Improve missing job error handling in maintd 
>   Prevent the watcher from submitting too many jobs 
>   Fix hscan failing loading network field 
>   Prevent watcher from holding lock on verify group 
> 
> * stable-2.16 
>   Fix some trivial pep8/pylint errors 
>   Make executeRpcCall only compute rpcCallData once 
>   Remove storage unit selection from rpcCallData 
>   Only import needed functions from Ganeti.JSON 
>   Special case WaitForJobChange to reduce heap use 
> 
> * stable-2.15 
>   Cleanup more pylint/pep8/apidoc errors 
>   KVM: handle gracefully too old/too new psutil versions 
>   Fix small typo in opcode tests 
>   Fix line-too long errors 
> 
> Signed-off-by: Brian Foley  
>
> diff --cc src/Ganeti/Types.hs 
> index c332ba0,8da06d4..66507cc 
> --- a/src/Ganeti/Types.hs 
> +++ b/src/Ganeti/Types.hs 
> @@@ -200,9 -200,9 +200,9 @@@ import Data.Ratio (numerator, denominat 
>   import System.Time (ClockTime) 
>   
>   import qualified Ganeti.ConstantUtils as ConstantUtils 
> - import Ganeti.JSON 
> + import Ganeti.JSON (Container, HasStringRepr(..)) 
>   import qualified Ganeti.THH as THH 
>  +import qualified Ganeti.THH.Field as THH (TagSet) 
>   import Ganeti.Utils 
>   
>   -- * Generic types 
> diff --cc test/py/mocks.py 
> index 66b9cf1,b48125b..5c78918 
> --- a/test/py/mocks.py 
> +++ b/test/py/mocks.py 
> @@@ -203,7 -202,5 +206,8 @@@ class FakeLU(object) 
>   return [] 
>   
> def HooksCallBack(self, phase, hook_results, feedback_fn, lu_result): 
> + # pylint: disable=W0613 
>   return lu_result 
>  + 
>  +  def HooksAbortCallBack(self, phase, feedback_fn, exception): 
>  +pass 
>

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit 3d17266c14d6e021253844b199b9518068875e4f
Merge: b7a0dde a824a2e
Author: Brian Foley 
Date:   Thu Jul 7 13:34:40 2016 +0100

Merge branch 'stable-2.17'

* stable-2.17
  Improve missing job error handling in maintd
  Prevent the watcher from submitting too many jobs
  Fix hscan failing loading network field
  Prevent watcher from holding lock on verify group

* stable-2.16
  Fix some trivial pep8/pylint errors
  Make executeRpcCall only compute rpcCallData once
  Remove storage unit selection from rpcCallData
  Only import needed functions from Ganeti.JSON
  Special case WaitForJobChange to reduce heap use

* stable-2.15
  Cleanup more pylint/pep8/apidoc errors
  KVM: handle gracefully too old/too new psutil versions
  Fix small typo in opcode tests
  Fix line-too long errors

Signed-off-by: Brian Foley 

diff --cc src/Ganeti/Types.hs
index c332ba0,8da06d4..66507cc
--- a/src/Ganeti/Types.hs
+++ b/src/Ganeti/Types.hs
@@@ -200,9 -200,9 +200,9 @@@ import Data.Ratio (numerator, denominat
  import System.Time (ClockTime)
  
  import qualified Ganeti.ConstantUtils as ConstantUtils
- import Ganeti.JSON
+ import Ganeti.JSON (Container, HasStringRepr(..))
  import qualified Ganeti.THH as THH
 +import qualified Ganeti.THH.Field as THH (TagSet)
  import Ganeti.Utils
  
  -- * Generic types
diff --cc test/py/mocks.py
index 66b9cf1,b48125b..5c78918
--- a/test/py/mocks.py
+++ b/test/py/mocks.py
@@@ -203,7 -202,5 +206,8 @@@ class FakeLU(object)
  return []
  
def HooksCallBack(self, phase, hook_results, feedback_fn, lu_result):
+ # pylint: disable=W0613
  return lu_result
 +
 +  def HooksAbortCallBack(self, phase, feedback_fn, exception):
 +pass

Re: [MERGE] stable-2.17 to master

['Iustin Pop' via ganeti-devel]

2016-06-10 16:25 GMT+02:00 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com>:

> commit 7eb49311e18865db76c4e8da5eb4b2e166db2d55
> Merge: a32eb3c 17a1c27
> Author: Brian Foley 
> Date:   Fri Jun 10 15:20:33 2016 +0100
>
> Merge branch 'stable-2.17'
>
> * stable-2.17
>   
>
> * stable-2.16
>   Fix optimisation: Correctly extract secondary node
>   Tune getNodeInstances DRBD secondary computation
>   Get haskell daemons to only compress files > 4kB
>   Use zlib compression level 3 in Haskell RPC code
>
> * stable-2.15
>   Fixup compatibility with GHC 7.4/base 4.5
>
> Signed-off-by: Brian Foley 
>

LGTM, thanks.

iustin

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit 7eb49311e18865db76c4e8da5eb4b2e166db2d55
Merge: a32eb3c 17a1c27
Author: Brian Foley 
Date:   Fri Jun 10 15:20:33 2016 +0100

Merge branch 'stable-2.17'

* stable-2.17
  

* stable-2.16
  Fix optimisation: Correctly extract secondary node
  Tune getNodeInstances DRBD secondary computation
  Get haskell daemons to only compress files > 4kB
  Use zlib compression level 3 in Haskell RPC code

* stable-2.15
  Fixup compatibility with GHC 7.4/base 4.5

Signed-off-by: Brian Foley 

diff --cc src/Ganeti/Codec.hs
index a703ba8,6f54c0d..b3b4c87
--- a/src/Ganeti/Codec.hs
+++ b/src/Ganeti/Codec.hs
@@@ -42,12 -40,12 +42,12 @@@ module Ganeti.Code
  import Prelude ()
  import Ganeti.Prelude
  
- import Codec.Compression.Zlib (compress)
+ import Codec.Compression.Zlib
  import qualified Codec.Compression.Zlib.Internal as I
  import Control.Monad (liftM)
 -import Control.Monad.Error.Class (MonadError(..))
  import qualified Data.ByteString.Lazy as BL
  import qualified Data.ByteString.Lazy.Internal as BL
 +import Control.Monad.Error.Class (MonadError(..))
  
  import Ganeti.BasicTypes
  

Re: [MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

LGTM

On Tuesday, May 17, 2016 at 2:11:21 PM UTC+1, Brian Foley wrote:
>
> commit 314d7a116027a1aaa925abe0290174a368ac2179 
> Merge: 99c2b44 20c24a8 
> Author: Brian Foley  
> Date:   Tue May 17 13:45:23 2016 +0100 
>
> Merge branch 'stable-2.17' 
> 
> * stable-2.17 
>
> 
> * stable-2.16 
>   Fix LogicalVolume Attach failure on missing path 
>   Set USE_VERSION_FULL=no if --enable-versionfull=no 
>   Generate concatenated error strings in parser 
>   Make the TH fieldsDictsKeys more efficient 
>   Don't verify disks when all disk templates are ext 
>   Improve test coverage for block device creation 
>   Docstring for LUClusterVerifyGroup 
>   Fix broken parameters in storage create methods 
>   Implement localized cache for lvs commands 
> 
> * stable-2.15 
>   mcpu: Raise caught OpPrereqErrors with too few args 
>   Make EnsureDirs print chmod mode on errors 
>   Fix CLI option typo in NEWS file 
> 
> Manually fix up python tests by adding the new HooksAbortCallBack to 
> FakeLU, and update FakeConfig to return valid node UUIDs as tested by 
> _ExecLU instead of dummy node names. 
> 
> Signed-off-by: Brian Foley  
>
> diff --cc test/py/mocks.py 
> index 649b032,406caca..66b9cf1 
> --- a/test/py/mocks.py 
> +++ b/test/py/mocks.py 
> @@@ -52,7 -58,7 +58,11 @@@ class FakeConfig(object) 
>   return True 
>   
> def GetNodeList(self): 
> --return ["a", "b", "c"] 
> ++return [ 
> ++  "01234567-89ab-cdef-fedc-", 
> ++  "01234567-89ab-cdef-fedc-", 
> ++  "01234567-89ab-cdef-fedc-" 
> ++] 
>   
> def GetRsaHostKey(self): 
>   return FAKE_CLUSTER_KEY 
> @@@ -64,7 -70,7 +74,7 @@@ 
>   return "test.cluster" 
>   
> def GetMasterNode(self): 
> --return "a" 
> ++return "01234567-89ab-cdef-fedc-" 
>   
> def GetMasterNodeName(self): 
>   return netutils.Hostname.GetSysName() 
> @@@ -139,3 -153,50 +157,53 @@@ class FakeGetentResolver(object) 
>   
> def LookupGid(self, gid): 
>   return "group%s" % gid 
> + 
> + class FakeLU(object): 
> +   HPATH = "fake-lu" 
> +   HTYPE = None 
> + 
> +   def __init__(self, processor, op, cfg, rpc_runner, prereq_err): 
> + self.proc = processor 
> + self.cfg = cfg 
> + self.op  = op 
> + self.rpc = rpc_runner 
> + self.prereq_err = prereq_err 
> + 
> + self.needed_locks = {} 
> + self.opportunistic_locks = dict.fromkeys(locking.LEVELS, False) 
> + self.dont_collate_locks = dict.fromkeys(locking.LEVELS, False) 
> + self.add_locks = {} 
> + 
> + self.LogWarning = processor.LogWarning 
> + 
> +   def CheckArguments(self): 
> + pass 
> + 
> +   def ExpandNames(self): 
> + pass 
> + 
> +   def DeclareLocks(self, level): 
> + pass 
> + 
> +   def CheckPrereq(self): 
> + if self.prereq_err: 
> +   raise self.prereq_err 
> + pass 
> + 
> +   def Exec(self, feedback_fn): 
> + pass 
> + 
> +   def BuildHooksNodes(self): 
> + return ([], []) 
> + 
> +   def BuildHooksEnv(self): 
> + return {} 
> + 
> +   def PreparePostHookNodes(self, post_hook_node_uuids): 
> + return [] 
> + 
> +   def HooksCallBack(self, phase, hook_results, feedback_fn, lu_result): 
> + return lu_result 
> ++ 
> ++  def HooksAbortCallBack(self, phase, feedback_fn, exception): 
> ++pass 
>

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit 314d7a116027a1aaa925abe0290174a368ac2179
Merge: 99c2b44 20c24a8
Author: Brian Foley 
Date:   Tue May 17 13:45:23 2016 +0100

Merge branch 'stable-2.17'

* stable-2.17
  

* stable-2.16
  Fix LogicalVolume Attach failure on missing path
  Set USE_VERSION_FULL=no if --enable-versionfull=no
  Generate concatenated error strings in parser
  Make the TH fieldsDictsKeys more efficient
  Don't verify disks when all disk templates are ext
  Improve test coverage for block device creation
  Docstring for LUClusterVerifyGroup
  Fix broken parameters in storage create methods
  Implement localized cache for lvs commands

* stable-2.15
  mcpu: Raise caught OpPrereqErrors with too few args
  Make EnsureDirs print chmod mode on errors
  Fix CLI option typo in NEWS file

Manually fix up python tests by adding the new HooksAbortCallBack to
FakeLU, and update FakeConfig to return valid node UUIDs as tested by
_ExecLU instead of dummy node names.

Signed-off-by: Brian Foley 

diff --cc test/py/mocks.py
index 649b032,406caca..66b9cf1
--- a/test/py/mocks.py
+++ b/test/py/mocks.py
@@@ -52,7 -58,7 +58,11 @@@ class FakeConfig(object)
  return True
  
def GetNodeList(self):
--return ["a", "b", "c"]
++return [
++  "01234567-89ab-cdef-fedc-",
++  "01234567-89ab-cdef-fedc-",
++  "01234567-89ab-cdef-fedc-"
++]
  
def GetRsaHostKey(self):
  return FAKE_CLUSTER_KEY
@@@ -64,7 -70,7 +74,7 @@@
  return "test.cluster"
  
def GetMasterNode(self):
--return "a"
++return "01234567-89ab-cdef-fedc-"
  
def GetMasterNodeName(self):
  return netutils.Hostname.GetSysName()
@@@ -139,3 -153,50 +157,53 @@@ class FakeGetentResolver(object)
  
def LookupGid(self, gid):
  return "group%s" % gid
+ 
+ class FakeLU(object):
+   HPATH = "fake-lu"
+   HTYPE = None
+ 
+   def __init__(self, processor, op, cfg, rpc_runner, prereq_err):
+ self.proc = processor
+ self.cfg = cfg
+ self.op  = op
+ self.rpc = rpc_runner
+ self.prereq_err = prereq_err
+ 
+ self.needed_locks = {}
+ self.opportunistic_locks = dict.fromkeys(locking.LEVELS, False)
+ self.dont_collate_locks = dict.fromkeys(locking.LEVELS, False)
+ self.add_locks = {}
+ 
+ self.LogWarning = processor.LogWarning
+ 
+   def CheckArguments(self):
+ pass
+ 
+   def ExpandNames(self):
+ pass
+ 
+   def DeclareLocks(self, level):
+ pass
+ 
+   def CheckPrereq(self):
+ if self.prereq_err:
+   raise self.prereq_err
+ pass
+ 
+   def Exec(self, feedback_fn):
+ pass
+ 
+   def BuildHooksNodes(self):
+ return ([], [])
+ 
+   def BuildHooksEnv(self):
+ return {}
+ 
+   def PreparePostHookNodes(self, post_hook_node_uuids):
+ return []
+ 
+   def HooksCallBack(self, phase, hook_results, feedback_fn, lu_result):
+ return lu_result
++
++  def HooksAbortCallBack(self, phase, feedback_fn, exception):
++pass

Re: [MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

LGTM

On Tue, May 3, 2016 at 12:12 PM, 'Brian Foley' via ganeti-devel <
ganeti-devel@googlegroups.com> wrote:

> commit e986d58e14865c4082c634ce1c925562617611be
> Merge: 8b2ec2f 0b57a03
> Author: Brian Foley 
> Date:   Tue May 3 11:56:08 2016 +0100
>
> Merge branch 'stable-2.17'
>
> * stable-2.15
>   Use socat method string compatible with <1.73 & >=1.73
>   Reduce heap when parsing & storing ConfigData 10%
>   Cancel RAPI job if the client drops the connection
>   Make JQScheduler queues more strict to avoid leaks
>
> * stable-2.16
>   
>
> * stable-2.17
>   
>
> Manually resolve merge conflicts in http_unittest.py (HTTPServerReqeust
> constructor arglist changed), and Constants.hs.
>
> Signed-off-by: Brian Foley 
>
> diff --cc src/Ganeti/Constants.hs
> index 09ca78d,13bff2e..1259978
> --- a/src/Ganeti/Constants.hs
> +++ b/src/Ganeti/Constants.hs
> @@@ -5577,9 -5545,11 +5577,17 @@@ maintdSuccessTagPrefix = maintdPrefix +
>   maintdFailureTagPrefix :: String
>   maintdFailureTagPrefix = maintdPrefix ++ "repairfailed:"
>
>  +-- * RAPI PAM auth related constants
>  +
>  +-- | The name of ganeti rapi specific http header containing additional
> user
>  +-- credentials
>  +httpRapiPamCredential :: String
>  +httpRapiPamCredential = "Ganeti-RAPI-Credential"
> ++
> + -- | The polling frequency to wait for a job status change
> + cliWfjcFrequency :: Int
> + cliWfjcFrequency = 20
> +
> + -- | Default 'WaitForJobChange' timeout in seconds
> + defaultWfjcTimeout :: Int
> + defaultWfjcTimeout = 60
>  -
> diff --cc test/py/ganeti.http_unittest.py
> index 0f17e11,753ddf3..7e6ba38
> --- a/test/py/ganeti.http_unittest.py
> +++ b/test/py/ganeti.http_unittest.py
> @@@ -226,31 -216,27 +227,31 @@@ class TestHttpServerRequestAuthenticati
>
> def testNoRealm(self):
>   headers = { http.HTTP_AUTHORIZATION: "", }
> - req = http.server._HttpServerRequest("GET", "/", headers, None)
> + req = http.server._HttpServerRequest("GET", "/", headers, None, None)
>  -ra = _FakeRequestAuth(None, False, None)
>  +ac = _SimpleAuthenticator("foo", "bar")
>  +ra = _FakeRequestAuth(None, False, ac)
>   self.assertRaises(AssertionError, ra.PreHandleRequest, req)
>
> def testNoScheme(self):
>   headers = { http.HTTP_AUTHORIZATION: "", }
> - req = http.server._HttpServerRequest("GET", "/", headers, None)
> + req = http.server._HttpServerRequest("GET", "/", headers, None, None)
>  -ra = _FakeRequestAuth("area1", False, None)
>  +ac = _SimpleAuthenticator("foo", "bar")
>  +ra = _FakeRequestAuth("area1", False, ac)
>   self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
>
> def testUnknownScheme(self):
>   headers = { http.HTTP_AUTHORIZATION: "NewStyleAuth abc", }
> - req = http.server._HttpServerRequest("GET", "/", headers, None)
> + req = http.server._HttpServerRequest("GET", "/", headers, None, None)
>  -ra = _FakeRequestAuth("area1", False, None)
>  +ac = _SimpleAuthenticator("foo", "bar")
>  +ra = _FakeRequestAuth("area1", False, ac)
>   self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
>
> def testInvalidBase64(self):
>   headers = { http.HTTP_AUTHORIZATION: "Basic x_=_", }
> - req = http.server._HttpServerRequest("GET", "/", headers, None)
> + req = http.server._HttpServerRequest("GET", "/", headers, None, None)
>  -ra = _FakeRequestAuth("area1", False, None)
>  -self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
>  +ac = _SimpleAuthenticator("foo", "bar")
>  +ra = _FakeRequestAuth("area1", False, ac)
>  +self.assertRaises(http.HttpBadRequest, ra.PreHandleRequest, req)
>
> def testAuthForPublicResource(self):
>   headers = {
> @@@ -286,8 -271,8 +287,8 @@@
>   for exc, headers in checks.items():
> for i in headers:
>   headers = { http.HTTP_AUTHORIZATION: i, }
> - req = http.server._HttpServerRequest("GET", "/", headers, None)
> + req = http.server._HttpServerRequest("GET", "/", headers, None,
> None)
>  -ra = _FakeRequestAuth("area1", False, None)
>  +ra = _FakeRequestAuth("area1", False, ac)
>   self.assertRaises(exc, ra.PreHandleRequest, req)
>
> def testBasicAuth(self):
>

[MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

commit e986d58e14865c4082c634ce1c925562617611be
Merge: 8b2ec2f 0b57a03
Author: Brian Foley 
Date:   Tue May 3 11:56:08 2016 +0100

Merge branch 'stable-2.17'

* stable-2.15
  Use socat method string compatible with <1.73 & >=1.73
  Reduce heap when parsing & storing ConfigData 10%
  Cancel RAPI job if the client drops the connection
  Make JQScheduler queues more strict to avoid leaks

* stable-2.16
  

* stable-2.17
  

Manually resolve merge conflicts in http_unittest.py (HTTPServerReqeust
constructor arglist changed), and Constants.hs.

Signed-off-by: Brian Foley 

diff --cc src/Ganeti/Constants.hs
index 09ca78d,13bff2e..1259978
--- a/src/Ganeti/Constants.hs
+++ b/src/Ganeti/Constants.hs
@@@ -5577,9 -5545,11 +5577,17 @@@ maintdSuccessTagPrefix = maintdPrefix +
  maintdFailureTagPrefix :: String
  maintdFailureTagPrefix = maintdPrefix ++ "repairfailed:"
  
 +-- * RAPI PAM auth related constants
 +
 +-- | The name of ganeti rapi specific http header containing additional user
 +-- credentials
 +httpRapiPamCredential :: String
 +httpRapiPamCredential = "Ganeti-RAPI-Credential"
++
+ -- | The polling frequency to wait for a job status change
+ cliWfjcFrequency :: Int
+ cliWfjcFrequency = 20
+ 
+ -- | Default 'WaitForJobChange' timeout in seconds
+ defaultWfjcTimeout :: Int
+ defaultWfjcTimeout = 60
 -
diff --cc test/py/ganeti.http_unittest.py
index 0f17e11,753ddf3..7e6ba38
--- a/test/py/ganeti.http_unittest.py
+++ b/test/py/ganeti.http_unittest.py
@@@ -226,31 -216,27 +227,31 @@@ class TestHttpServerRequestAuthenticati
  
def testNoRealm(self):
  headers = { http.HTTP_AUTHORIZATION: "", }
- req = http.server._HttpServerRequest("GET", "/", headers, None)
+ req = http.server._HttpServerRequest("GET", "/", headers, None, None)
 -ra = _FakeRequestAuth(None, False, None)
 +ac = _SimpleAuthenticator("foo", "bar")
 +ra = _FakeRequestAuth(None, False, ac)
  self.assertRaises(AssertionError, ra.PreHandleRequest, req)
  
def testNoScheme(self):
  headers = { http.HTTP_AUTHORIZATION: "", }
- req = http.server._HttpServerRequest("GET", "/", headers, None)
+ req = http.server._HttpServerRequest("GET", "/", headers, None, None)
 -ra = _FakeRequestAuth("area1", False, None)
 +ac = _SimpleAuthenticator("foo", "bar")
 +ra = _FakeRequestAuth("area1", False, ac)
  self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
  
def testUnknownScheme(self):
  headers = { http.HTTP_AUTHORIZATION: "NewStyleAuth abc", }
- req = http.server._HttpServerRequest("GET", "/", headers, None)
+ req = http.server._HttpServerRequest("GET", "/", headers, None, None)
 -ra = _FakeRequestAuth("area1", False, None)
 +ac = _SimpleAuthenticator("foo", "bar")
 +ra = _FakeRequestAuth("area1", False, ac)
  self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
  
def testInvalidBase64(self):
  headers = { http.HTTP_AUTHORIZATION: "Basic x_=_", }
- req = http.server._HttpServerRequest("GET", "/", headers, None)
+ req = http.server._HttpServerRequest("GET", "/", headers, None, None)
 -ra = _FakeRequestAuth("area1", False, None)
 -self.assertRaises(http.HttpUnauthorized, ra.PreHandleRequest, req)
 +ac = _SimpleAuthenticator("foo", "bar")
 +ra = _FakeRequestAuth("area1", False, ac)
 +self.assertRaises(http.HttpBadRequest, ra.PreHandleRequest, req)
  
def testAuthForPublicResource(self):
  headers = {
@@@ -286,8 -271,8 +287,8 @@@
  for exc, headers in checks.items():
for i in headers:
  headers = { http.HTTP_AUTHORIZATION: i, }
- req = http.server._HttpServerRequest("GET", "/", headers, None)
+ req = http.server._HttpServerRequest("GET", "/", headers, None, None)
 -ra = _FakeRequestAuth("area1", False, None)
 +ra = _FakeRequestAuth("area1", False, ac)
  self.assertRaises(exc, ra.PreHandleRequest, req)
  
def testBasicAuth(self):

Re: [MERGE] stable-2.17 to master

['Brian Foley' via ganeti-devel]

On Fri, Apr 22, 2016 at 03:21:07PM +0100, 'Viktor Bachraty' via ganeti-devel 
wrote:
> commit 2cfaa3420a6bd4e51d0adaf117bc497fab0e9add
> Merge: 3b5ac4e b9de363
> Author: Viktor Bachraty 
> Date:   Fri Apr 22 15:20:33 2016 +0100
> 
> Merge branch 'stable-2.17'
> 
> * stable-2.16
>   Make debug http server more verbose
>   Reduce load in NV_NODENETTEST and NV_MASTERIP
>   Re-fix local port/ip logging in HttpClientRequest
> 
> * stable-2.15
>   Allow newer cabal deps for latest Debian/Ubuntu
> 

LGTM.

[MERGE] stable-2.17 to master

['Viktor Bachraty' via ganeti-devel]

commit 2cfaa3420a6bd4e51d0adaf117bc497fab0e9add
Merge: 3b5ac4e b9de363
Author: Viktor Bachraty 
Date:   Fri Apr 22 15:20:33 2016 +0100

Merge branch 'stable-2.17'

* stable-2.16
  Make debug http server more verbose
  Reduce load in NV_NODENETTEST and NV_MASTERIP
  Re-fix local port/ip logging in HttpClientRequest

* stable-2.15
  Allow newer cabal deps for latest Debian/Ubuntu