On Tue, Feb 04, 2014 at 05:00:19PM +0100, Santi Raffa wrote:
> This patch introduces private and secret parameters to Ganeti. The
> design principle here is that:
>
> * Existing code that should not get access to the parameters should
> remain untouched, with the value not accessible to them.
> * Existing code that should get access to the parameters should be
> modified to make access explicit, for example by explictly asking
> ToDict and DumpJSON to include those fields in their output.
> * Querying systems have not been modified and, as a result, should not
> report anything about private parameters. `gnt-job shiw`, for
> example, reports that the relevant opcode parameters are private.
> * It should not be possible for new code to accidentally leak these
> values after they have been encoded without writing code that
> explictly asks for them.
> * Before the values have been encoded - for example in the logging
> of command lines, when we don't yet know what should and should not
> be logged, all logging that could leak these parameters should be
> done at the debug level. The logging is kept for diagnosis purposes.
> We should advise the users not to use Ganeti in debug mode on a
> production server.
>
> The implementation of this design is done through a Private container
> object (in Python)/monad (in Haskell) that only give up the values of
> their contents if you ask nicely.
>
> The more complete implementation is provided for Python:
>
> * A Private object can have its attributes accessed and called
> transparently. Equality and non-zeroness checks are also supported:
>
> >>> serializer.Private("foo")
> Private(?, descr='redacted')
> >>> serializer.Private("foo").upper()
> Private(?, descr='redacted.upper()')
> >>> serializer.Private("foo").upper().Get()
> 'FOO'
> >>> serializer.Private("foo").upper().Get() == "FOO"
> True
> >>> bool(serializer.Private("foo"))
> True
>
> This should make it easy for developers to use Private objects
> normally and transparently. I expect that implementing Private as a
> monad in Haskell will allow at least some of these properties to
> also be available there.
> * A Private object resists pickling by having slots but no setstate.
> * PrivateDict's are added (dicts where all values are automatically
> wrapped in Private's).
> * At JSON deserialization time, for lack of a type system that embeds
> what is the expected type of a JSON request, I PrivateDict all JSON
> objects the keys of which are either "osparams_private" or
> "osparams_secret", plus all the values of a "osparams_private_cluster"
> object. (Having to treat cluster OS parameters and instance OS
> parameters differently explains the difference in names.)
> The measured overhead of this process is about 5%.
> * Only the code paths required for normal operations are whitelisted
> for Private access, including RPC sending and configuration saving.
>
> Current known shortcomings of this implementation:
>
> * Secret parameters ARE saved to disk as part of the job queue system.
> * The JSON serialization functions in Haskell always return the value
> of Private parameters. This has not been found to be a problem, yet.
> * Per-hypervisor OS parameter combinations remain public only. I
> would be hard pressed to come with a use case for private or secret
> per-hypervisor OS parameter.
>
> I trust that these issues, if necessary, can be handled between now and
> the release of 2.12.
>
> Santi Raffa (15):
> rpcs: fix docstrings for instance_os_add
> OpCodes test: fix argument order (expected/but got)
> algo: add GetRepeatedKeys
> constants: add visibility levels
> Add Private types to Python, Haskell
> serializer.py: emit and encode Private values
> Config: add private OS parameters to the configuration
> cli: add parameters for private and secret OS parameters
> opcodes: Modify InstanceSetParams
> opcodes: modify ClusterSetParams
> opcodes: modify InstanceCreate
> opcodes: modify InstanceReinstall
> luxid: give stern warnings about debug mode
> First version of unit test
> NEWS: update with public and private paramters
I will reply to all the titles here because it is easier.
The titles should be consistent in capitalization and acronyms.
For example
Algo: add GetRepeatedKeys
instead of
algo: ...
Also,
OpCodes: modify ...
instead of
opcodes: Modify ...
CLI instead of cli, and so on.
Thanks,
Jose
>
> NEWS | 23 ++++
> lib/backend.py | 7 +-
> lib/bootstrap.py | 2 +
> lib/cli.py | 33 ++++-
> lib/client/gnt_instance.py | 18 ++-
> lib/client/gnt_os.py | 12 +-
> lib/cmdlib/cluster.py | 32 ++++-
> lib/cmdlib/common.py | 7 +
> lib/cmdlib/instance.py | 58 +++++++-
> lib/cmdlib/instance_operation.py | 7 +-
> lib/config.py | 7 +-
> lib/daemon.py | 11 +-
> lib/ht.py | 19 ++-
> lib/luxi.py | 5 +-
> lib/objects.py | 138 +++++++++++++++----
> lib/rpc/client.py | 3 +-
> lib/rpc/node.py | 8 +-
> lib/rpc_defs.py | 10 +-
> lib/serializer.py | 211
> ++++++++++++++++++++++++++++--
> lib/server/noded.py | 3 +-
> lib/utils/algo.py | 28 ++++
> man/gnt-instance.rst | 20 ++-
> man/gnt-os.rst | 10 +-
> src/Ganeti/Config.hs | 2 +-
> src/Ganeti/Constants.hs | 34 +++++
> src/Ganeti/Daemon.hs | 6 +
> src/Ganeti/HTools/Program/Harep.hs | 4 +
> src/Ganeti/JSON.hs | 1 +
> src/Ganeti/Logging.hs | 8 ++
> src/Ganeti/Objects.hs | 110 ++++++++--------
> src/Ganeti/OpCodes.hs | 6 +
> src/Ganeti/OpParams.hs | 58 ++++++++
> src/Ganeti/Types.hs | 61 +++++++++
> src/Ganeti/UDSServer.hs | 15 ++-
> test/data/instance-prim-sec.txt | 1 +
> test/hs/Test/Ganeti/Objects.hs | 10 +-
> test/hs/Test/Ganeti/OpCodes.hs | 145 +++++++++++++++-----
> test/hs/Test/Ganeti/Query/Instance.hs | 2 +-
> test/py/cmdlib/cluster_unittest.py | 1 +
> test/py/cmdlib/instance_unittest.py | 1 +
> test/py/cmdlib/node_unittest.py | 1 -
> test/py/cmdlib/testsupport/config_mock.py | 5 +
> test/py/ganeti.config_unittest.py | 4 +-
> test/py/ganeti.objects_unittest.py | 101 ++++++++++++++
> test/py/ganeti.ovf_unittest.py | 2 +
> test/py/ganeti.serializer_unittest.py | 118 ++++++++++++++---
> 46 files changed, 1188 insertions(+), 180 deletions(-)
>
> --
> 1.9.0.rc1.175.g0b1dcb5
>
--
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370
