[pve-devel] [PATCH-SERIES 0/21] remote migration

2022-02-09 Thread Fabian Grünbichler
this series adds remote migration for VMs (and if the last pve-container patch is applied, CTs ;)). both live and offline migration of VMs including NBD and storage-migrated disks should work, containers don't have any live migration so both offline and restart mode work identical except for the r

[pve-devel] [PATCH v5 container 2/3] config: add strict parser

2022-02-09 Thread Fabian Grünbichler
as safeguard when migrating across clusters, which might have different versions installed. Signed-off-by: Fabian Grünbichler --- Notes: new in v5 src/PVE/LXC/Config.pm | 22 -- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/src/PVE/LXC/Config.pm b/src/P

[pve-devel] [PATCH v5 container 1/3] fix #1532: add target-storage support to migration

2022-02-09 Thread Fabian Grünbichler
re-using helpers that already exist for qemu-server. this is a pre-requisite for extending remote migration support to containers. Signed-off-by: Fabian Grünbichler --- Notes: new in v5, no GUI yet until possible wrinkles are ironed out requires pve-common/pve-guest-common with chan

[pve-devel] [PATCH v5 qemu-server 02/11] schema: use pve-bridge-id

2022-02-09 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- Notes: requires pve-common with pve-bridge-id PVE/QemuServer.pm | 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index 8c1aa30a..8893f86e 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer

[pve-devel] [PATCH v5 qemu-server 01/11] move map_storage to PVE::JSONSchema::map_id

2022-02-09 Thread Fabian Grünbichler
since we are going to reuse the same mechanism/code for network bridge mapping and pve-container. Signed-off-by: Fabian Grünbichler --- Notes: v5: move to pve-common/PVE::JSONSchema requires bumped pve-common with map_id (build and regular depends) PVE/QemuMigrate.pm | 6 +++---

[pve-devel] [PATCH v5 qemu-server 03/11] parse_config: optional strict mode

2022-02-09 Thread Fabian Grünbichler
when passing a config from one cluster to another, we want to be strict when parsing - it's better to fail the migration early and upgrade the target node instead of failing the migration later (when significant work for transferring disks and/or state has already been done) or not at all, but sile

[pve-devel] [PATCH v5 storage 1/3] storage_migrate_snapshot: skip for btrfs without snapshots

2022-02-09 Thread Fabian Grünbichler
this allows migrating from btrfs to other raw+size accepting storages, provided no snapshots exist. Signed-off-by: Fabian Grünbichler --- Notes: new in v5 PVE/Storage.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/PVE/Storage.pm b/PVE/Storage.pm index 93ae3ac.

[pve-devel] [PATCH v5 guest-common 3/3] add storage tunnel module

2022-02-09 Thread Fabian Grünbichler
encapsulating storage-related tunnel methods, currently - source-side storage-migrate helper - target-side disk-import handler - target-side query-disk-import handler - target-side bwlimit handler to be extended further with replication-related handlers and helpers. Signed-off-by: Fabian Grünbich

[pve-devel] [PATCH v5 qemu-server 06/11] migrate: move tunnel-helpers to pve-guest-common

2022-02-09 Thread Fabian Grünbichler
besides the log calls these don't need any parts of the migration state, so let's make them generic and re-use them for container migration and replication in the future. Signed-off-by: Fabian Grünbichler --- Notes: v4: switch log to use two-parameter logging sub like migration new in v3

[pve-devel] [PATCH v5 qemu-server 10/11] api: add remote migrate endpoint

2022-02-09 Thread Fabian Grünbichler
entry point for the remote migration on the source side, mainly preparing the API client that gets passed to the actual migration code and doing some parameter parsing. querying of the remote sides resources (like available storages, free VMIDs, lookup of endpoint details for specific nodes, ...)

[pve-devel] [PATCH PoC v5 container 3/3] migration: add remote migration

2022-02-09 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- Notes: new in v5 - PoC to ensure helpers and abstractions are re-usable src/PVE/API2/LXC.pm| 630 + src/PVE/LXC/Migrate.pm | 237 +--- 2 files changed, 828 insertions(+), 39 deletions(-) diff --gi

[pve-devel] [PATCH v5 qemu-server 09/11] migrate: add remote migration handling

2022-02-09 Thread Fabian Grünbichler
remote migration uses a websocket connection to a task worker running on the target node instead of commands via SSH to control the migration. this websocket tunnel is started earlier than the SSH tunnel, and allows adding UNIX-socket forwarding over additional websocket connections on-demand. the

[pve-devel] [PATCH v5 common 1/1] add 'map_id' helper for ID maps

2022-02-09 Thread Fabian Grünbichler
currently these are used by qemu-server for mapping source and target storages, but this mechanism will be extended to network bridge maps and re-used in pve-container as well, so let's put it next to the schema definitions/helpers. Signed-off-by: Fabian Grünbichler --- Notes: new in v5, req

[pve-devel] [PATCH v5 guest-common 1/3] migrate: add get_bwlimit helper

2022-02-09 Thread Fabian Grünbichler
given a source and target storage query either locally or both locally and remotely and merge the result. Signed-off-by: Fabian Grünbichler --- Notes: new in v5, replaces merge_bwlimits previously in PVE::QemuMigrate src/PVE/AbstractMigrate.pm | 26 ++ 1 file change

[pve-devel] [PATCH v5 qemu-server 04/11] update_vm: allow simultaneous setting of boot-order and dev

2022-02-09 Thread Fabian Grünbichler
else this fails if we check 'boot' before the device was put into the config or pending section. Signed-off-by: Fabian Grünbichler --- Notes: as happens when doing a remote migration and the full config is passed through update_vm_api PVE/API2/Qemu.pm | 2 +- 1 file changed, 1 inserti

[pve-devel] [PATCH v5 qemu-server 05/11] nbd alloc helper: allow passing in explicit format

2022-02-09 Thread Fabian Grünbichler
and make $volname optional, to support remote migration usage without code duplication. Signed-off-by: Fabian Grünbichler --- Notes: v2: new PVE/QemuServer.pm | 26 +- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.

[pve-devel] [PATCH v5 guest-common 2/3] add tunnel helper module

2022-02-09 Thread Fabian Grünbichler
lifted from PVE::QemuMigrate, abstracting away use-case specific data. Signed-off-by: Fabian Grünbichler --- Notes: v5: nits v4: - remove remaining 'migration' mentions - fix wrong 'log' call - set 'log' during WS tunnel setup already so it's available for error handling

[pve-devel] [PATCH v5 storage 2/3] storage_migrate: pull out import/export_prepare

2022-02-09 Thread Fabian Grünbichler
for re-use with remote migration, where import and export happen on different clusters connected via a websocket instead of SSH tunnel. Signed-off-by: Fabian Grünbichler --- Notes: v4: - unify array refs - small fixups new in v3 PVE/Storage.pm | 113

[pve-devel] [PATCH v5 storage 3/3] add volume_import/export_start helpers

2022-02-09 Thread Fabian Grünbichler
exposing the two halves of a storage migration for usage across cluster boundaries. Signed-off-by: Fabian Grünbichler --- Notes: v5: - re-use migration snapshot info from source in volume_import_start (fixes migration from non-btrfs to btrfs) - correctly pass snapshot (name), n

[pve-devel] [PATCH v5 qemu-server 11/11] qm: add remote-migrate command

2022-02-09 Thread Fabian Grünbichler
which wraps the remote_migrate_vm API endpoint, but does the precondition checks that can be done up front itself. this now just leaves the FP retrieval and target node name lookup to the sync part of the API endpoint, which should be do-able in <30s .. Signed-off-by: Fabian Grünbichler --- Not

[pve-devel] [PATCH v5 qemu-server 08/11] migrate: refactor remote VM/tunnel start

2022-02-09 Thread Fabian Grünbichler
no semantic changes intended, except for: - no longer passing the main migration UNIX socket to SSH twice for forwarding - dropping the 'unix:' prefix in start_remote_tunnel's timeout error message Signed-off-by: Fabian Grünbichler --- PVE/QemuMigrate.pm | 158 ---

[pve-devel] [PATCH v5 qemu-server 07/11] mtunnel: add API endpoints

2022-02-09 Thread Fabian Grünbichler
the following two endpoints are used for migration on the remote side POST /nodes/NODE/qemu/VMID/mtunnel which creates and locks an empty VM config, and spawns the main qmtunnel worker which binds to a VM-specific UNIX socket. this worker handles JSON-encoded migration commands coming in via thi

Re: [pve-devel] [PATCH proxmox-ve v2 2/2] apt-hook: add check preventing the removal of pinned kernels

2022-02-09 Thread Stoiko Ivanov
while talking off-list about this I realized that I forgot to mention that file_read_firstline is copied from PVE::Tools. The rationale was that we might end up in a situation where pve-common might not be available and the hook might still be called. Also we might eventually have this hook in som

Re: [pve-devel] [PATCH common 1/1] PBSClient: add option for extra parameter to file_restore_list

2022-02-09 Thread Thomas Lamprecht
On 27.01.22 11:55, Dominik Csapak wrote: > we will need some extra parameters here, and instead of hardcoding them, > have the option to set a list of arbitrary parameters > > Signed-off-by: Dominik Csapak > --- > src/PVE/PBSClient.pm | 8 ++-- > 1 file changed, 6 insertions(+), 2 deletions(

Re: [pve-devel] [PATCH-SERIES 0/21] remote migration

2022-02-09 Thread Thomas Lamprecht
On 09.02.22 14:07, Fabian Grünbichler wrote: > this series adds remote migration for VMs (and if the last pve-container > patch is applied, CTs ;)). > > both live and offline migration of VMs including NBD and storage-migrated > disks > should work, containers don't have any live migration so bot

Re: [pve-devel] [PATCH pve-manager] NetworkEdit: allow setting 'mtu' option for guest network devices

2022-02-09 Thread Thomas Lamprecht
commit subject should rather be "ui: vm network: allow to override mtu for virtio devices" On 07.02.22 13:18, Oguz Bektas wrote: > we already have the 'mtu' option inside the API, so we can just expose > that option inside the 'Advanced' menu for guest network interfaces. We have that property s