Hi,
Are you using some kind of network policy within your cluster ?
Best,
Davide
On Wed, 11 Jan 2023 at 10:53 Zsolt Kozak <koza...@gmail.com> wrote:
> Hello Davide!
>
> I am running the File Daemon on the master node, on the control plane.
> It's Kubernetes vanilla, version 1.25.4.
> No, the master node is running on the same subnet as the workers.
>
> It should be some network issue, I think.
>
> Best regards,
> Zsolt
>
> On Wed, Jan 11, 2023 at 8:45 AM Davide F. <bacula-...@dflc.ch> wrote:
>
>> Hello Kozak,
>>
>> I haven’t tried k8s plugin but let me try to understand what could be the
>> root cause of your problem.
>>
>> Could you explain further point 1 please ?
>> On which node are you running the file daemon ?
>>
>> Which version / flavor of Kubernetes are you using ?
>>
>> Is it Kubernetes vanilla ? OpenShift ? Tansu ?
>>
>> Depending on your feedback from the first question, does master nodes
>> runs in a different subnet than worker’s ?
>>
>> Thanks for your feedback
>>
>> Best,
>>
>> Davide
>>
>> On Tue, 10 Jan 2023 at 21:12 Zsolt Kozak <koza...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have some problems with backuping Kubernetes PVCs with Bacula
>>> Kubernetes Plugin. (I have asked it on bacula-users mailing list but got no
>>> answer.)
>>>
>>> I am using the latest 13.0.1 Bacula from the community builds on Debian
>>> Bullseye hosts.
>>>
>>> Backuping only the Kubernetes objects except Persistent Volume Claims
>>> (PVC) works like a charm. I've installed the Kubernetes plugin and the
>>> latest Bacula File Daemon on the master node (control plane) of our
>>> Kubernetes cluster. Bacula can access the Kubernetes cluster and backup
>>> every single object as YAML files.
>>>
>>> The interesting part comes with trying to backup a PVC...
>>>
>>> First of all I could build my own Bacula Backup Proxy Pod Image from the
>>> source and it's deployed into our local Docker image repository (repo). The
>>> Bacula File Daemon is configured properly I guess. Backup process started
>>> and the following things happened.
>>>
>>> 1. Bacula File Daemon deployed Bacula Backup Proxy Pod Image into the
>>> Kubernetes cluster, so Bacula-backup container pod started.
>>> 2. I got into the pod and I could see the Baculatar application started
>>> and running.
>>> 3. The k8s_backend application started on the Bacula File Daemon host
>>> (kubernetes.server) in 2 instances.
>>> 4. From the Bacula-backup pod I could check that Baculatar could connect
>>> to the k8s_backend at the default 9104 port (kubernetes.server:9104).
>>> 5. I checked the console messages of the job with Bat that Bacula File
>>> Daemon started to process the configured PVC, started to write a pvc.tar
>>> but nothing happened.
>>> 6. After default 600 sec, after timeout the job was cancelled.
>>> 7. It may be important that Bacula File Daemon could not delete the
>>> Bacula-backup pod. (It could create it but could not delete it.)
>>>
>>>
>>> Could you please tell me what's wrong?
>>>
>>>
>>> Here are some log parts. (I've changed some sensitive data.)
>>>
>>>
>>> Bacula File Daemon configuration:
>>>
>>> FileSet {
>>> Name = "Kubernetes Set"
>>> Include {
>>> Options {
>>> signature = SHA512
>>> compression = GZIP
>>> Verify = pins3
>>> }
>>> Plugin = "kubernetes: \
>>> debug=1 \
>>> baculaimage=repo/bacula-backup:04jan23 \
>>> namespace=namespace \
>>> pvcdata \
>>> pluginhost=kubernetes.server \
>>> timeout=120 \
>>> verify_ssl=0 \
>>> fdcertfile=/etc/bacula/certs/bacula-backup.cert \
>>> fdkeyfile=/etc/bacula/certs/bacula-backup.key"
>>> }
>>> }
>>>
>>>
>>>
>>> Bacula File Daemon debug log (parts):
>>>
>>>
>>> DEBUG:[baculak8s/jobs/estimation_job.py:134 in processing_loop]
>>> processing get_annotated_namespaced_pods_data:namespace:nrfound:0
>>> DEBUG:[baculak8s/plugins/kubernetes_plugin.py:319 in
>>> list_pvcdata_for_namespace] list pvcdata for namespace:namespace
>>> pvcfilter=True estimate=False
>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:108 in
>>> pvcdata_list_namespaced] pvcfilter: True
>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:112 in
>>> pvcdata_list_namespaced] found:some-claim
>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:127 in
>>> pvcdata_list_namespaced] add pvc: {'name': 'some-claim', 'node_name': None,
>>> 'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>> DEBUG:[baculak8s/jobs/estimation_job.py:165 in processing_loop]
>>> processing list_pvcdata_for_namespace:namespace:nrfound:1
>>> DEBUG:[baculak8s/jobs/estimation_job.py:172 in processing_loop]
>>> PVCDATA:some-claim:{'name': 'some-claim', 'node_name': 'node1',
>>> 'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> I000041
>>> Start backup volume claim: some-claim
>>>
>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:298 in prepare_bacula_pod]
>>> prepare_bacula_pod:token=xx88M5oggQJ....4YDbSwBRxTOhT namespace=namespace
>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:136 in prepare_pod_yaml]
>>> pvcdata: {'name': 'some-claim', 'node_name': 'node1', 'storage_class_name':
>>> 'nfs-client', 'capacity': '2Gi', 'fi':
>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>> DEBUG:[baculak8s/plugins/k8sbackend/baculabackup.py:102 in
>>> prepare_backup_pod_yaml] host:kubernetes.server port:9104
>>> namespace:namespace image:repo/bacula-backup:04jan23
>>> job:KubernetesBackup.2023-01-04_21.05.03_10:410706
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> I000149
>>> Prepare Bacula Pod on: node1 with: repo/bacula-backup:04jan23
>>> <IfNotPresent> kubernetes.server:9104
>>>
>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:198 in
>>> prepare_connection_server] prepare_connection_server:New ConnectionServer:
>>> 0.0.0.0:9104
>>> DEBUG:[baculak8s/util/sslserver.py:180 in listen]
>>> ConnectionServer:Listening...
>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:307 in prepare_bacula_pod]
>>> prepare_bacula_pod:start pod
>>> INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in backup_pod_isready]
>>> backup_pod_status:isReady: False / 0
>>> INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in backup_pod_isready]
>>> backup_pod_status:isReady: True / 1
>>> DEBUG:[baculak8s/jobs/estimation_job.py:183 in _estimate_file] {'name':
>>> 'some-claim', 'node_name': 'node1', 'storage_class_name': 'nfs-client',
>>> 'capacity': '2Gi', 'fi': <baculak8s.entities.file_info.FileInfo object at
>>> 0x7ffaa55bfcc0>}
>>> DEBUG:[baculak8s/jobs/estimation_job.py:190 in _estimate_file]
>>> file_info: {FileInfo
>>> name:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
>>> namespace:None type:F objtype:pvcdata cached:False}
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> C000079
>>>
>>> FNAME:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
>>>
>>>
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> C000040
>>> TSTAMP:1672861077 1672861077 1672861077
>>>
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> C000031
>>> STAT:F 2147483648 0 0 100640 1
>>>
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> F000000
>>> (EOD PACKET)
>>>
>>> DEBUG:[baculak8s/jobs/backup_job.py:77 in __backup_pvcdata]
>>> backup_pvcdata:data recv
>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>> C000005
>>> DATA
>>>
>>> DEBUG:[baculak8s/util/sslserver.py:193 in handle_connection]
>>> ConnectionServer:Connection from: ('192.168.XX.YY', 10541)
>>> DEBUG:[baculak8s/util/sslserver.py:145 in gethello] ['Hello',
>>> 'KubernetesBackup.2023-01-04_21.05.03_10', '410706']
>>> DEBUG:[baculak8s/util/token.py:57 in check_auth_data] AUTH_DATA:Token:
>>> xx88M5oggQJuGsPbtD........ohQjeU7PkA4YDbSwBRxTOhT
>>> DEBUG:[baculak8s/util/token.py:59 in check_auth_data]
>>> RECV_TOKEN_DATA:Token: xx88M5oggQJuGsPbtD....ohQjeU7PkA4YDbSwBRxTOhT
>>> DEBUG:[baculak8s/util/sslserver.py:105 in authenticate]
>>> ConnectionServer:Authenticated
>>>
>>> .... after timeout
>>>
>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:121 in handle_pod_data_recv]
>>> handle_pod_data_recv:EOT
>>> DEBUG:[baculak8s/util/sslserver.py:201 in handle_connection]
>>> ConnectionServer:Finish - disconnect.
>>> DEBUG:[baculak8s/jobs/backup_job.py:85 in __backup_pvcdata]
>>> backup_pvcdata:logs recv
>>>
>>>
>>>
>>>
>>> Job messages:
>>>
>>> bacula-dir No prior or suitable Full backup found in catalog for the
>>> current FileSet. Doing FULL backup.
>>> The FileSet "Kubernetes Set" was modified on 2023-01-04 20:20:41, this
>>> is after the last successful backup on 2023-01-04 19:19:49.
>>> bacula-sd Ready to append to end of Volume "Full-XXX" size=3,838,161,002
>>> bacula-fd Connected to Storage at bacula.server:9103 with TLS
>>> bacula-sd Volume "Full-XXXX" previously written, moving to end of data.
>>> bacula-dir Connected to Client "bacula-fd" at kubernetes.server:9102
>>> with TLS
>>> Using Device "FileStorageEeyoreFull" to write.
>>> Connected to Storage "InternalStorageFull" at bacula.server:9103 with
>>> TLS
>>> Start Backup JobId 410830, Job=KubernetesBackup.2023-01-04_21.05.03_10
>>> bacula-fd kubernetes: Prepare Bacula Pod on: node with:
>>> repo/bacula-backup:04jan23 kubernetes.server:9104
>>> kubernetes: Processing namespace: namespace
>>> kubernetes: Start backup volume claim: some-claim
>>> kubernetes: Connected to Kubernetes 1.25 - v1.25.4.
>>> bacula-dir
>>> Error: Bacula Enterprise bacula-dir 13.0.1 (05Aug22):
>>> Build OS: x86_64-pc-linux-gnu-bacula-enterprise debian
>>> 11.2
>>> JobId: 410830
>>> Job: KubernetesBackup.2023-01-04_21.05.03_10
>>> Backup Level: Full (upgraded from Differential)
>>> Client: "bacula-fd" 13.0.1 (05Aug22)
>>> x86_64-pc-linux-gnu-bacula-enterprise,debian,10.11
>>> FileSet: "Kubernetes Set" 2023-01-04 20:20:41
>>> Pool: "Full-Pool" (From Job FullPool override)
>>> Catalog: "MyCatalog" (From Client resource)
>>> Storage: "InternalStorageFull" (From Pool resource)
>>> Scheduled time: 04-Jan-2023 21:05:03
>>> Start time: 04-Jan-2023 21:27:04
>>> End time: 04-Jan-2023 21:29:06
>>> Elapsed time: 2 mins 2 secs
>>> Priority: 10
>>> FD Files Written: 23
>>> SD Files Written: 0
>>> FD Bytes Written: 52,784 (52.78 KB)
>>> SD Bytes Written: 0 (0 B)
>>> Rate: 0.4 KB/s
>>> Software Compression: 100.0% 1.0:1
>>> Comm Line Compression: 5.6% 1.1:1
>>> Snapshot/VSS: no
>>> Encryption: yes
>>> Accurate: yes
>>> Volume name(s): Full-XXXX
>>> Volume Session Id: 43
>>> Volume Session Time: 1672853724
>>> Last Volume Bytes: 3,838,244,105 (3.838 GB)
>>> Non-fatal FD errors: 3
>>> SD Errors: 0
>>> FD termination status: OK
>>> SD termination status: SD despooling Attributes
>>> Termination: *** Backup Error ***
>>> Fatal error: catreq.c:680 Restore object create error.
>>> bacula-fd
>>> Error: kubernetes: PTCOMM cannot get packet header from backend.
>>> bacula-dir Fatal error: sql_create.c:1273 Create db Object record INSERT
>>> INTO RestoreObject
>>> (ObjectName,PluginName,RestoreObject,ObjectLength,ObjectFullLength,ObjectIndex,ObjectType,ObjectCompression,FileIndex,JobId)
>>> VALUES ('RestoreOptions','kubernetes: \n debug=1 \n
>>> baculaimage=repo/bacula-backup:04jan23 \n namespace=namespace
>>> \n pvcdata \n
>>> pluginhost=kubernetes.server \n timeout=120 \n
>>> verify_ssl=0 \n fdcertfile=/etc/bacula/certs/bacula-backup.cert
>>> \n
>>> fdkeyfile=/etc/bacula/certs/bacula-backup.key','# Plugin configuration
>>> file\n# Version 1\nOptPrompt=\"K8S config
>>> file\"\nOptDefault=\"*None*\"\nconfig=@STR@\n\n
>>> OptPrompt=\"K8S API server
>>> URL/Host\"\nOptDefault=\"*None*\"\nhost=@STR@\n\nOptPrompt=\"K8S
>>> Bearertoken\"\nOptDefault=\"*None*\"\ntoken=@STR@\n\nOptPrompt=\"K8S
>>> API server cert verification\"\n
>>> OptDefault=\"True\"\nverify_ssl=@BOOL@\n\nOptPrompt=\"Custom CA Certs
>>> file to
>>> use\"\nOptDefault=\"*None*\"\nssl_ca_cert=@STR@\n\nOptPrompt=\"Output
>>> format when saving to file (JSON, YAML)\"\n
>>> OptDefault=\"RAW\"\noutputformat=@STR@\n\nOptPrompt=\"The address for
>>> listen to incoming backup pod
>>> data\"\nOptDefault=\"*FDAddress*\"\nfdaddress=@STR@\n\n
>>> OptPrompt=\"The port for opening socket for
>>> listen\"\nOptDefault=\"9104\"\nfdport=@INT32@\n\nOptPrompt=\"The
>>> endpoint address for backup pod to connect\"\n
>>> OptDefault=\"*FDAddress*\"\npluginhost=@STR@\n\nOptPrompt=\"The
>>> endpoint port to connect\"\nOptDefault=\"9104\"\n
>>> pluginport=@INT32@\n\n',859,859,0,27,0,1,410830) failed. ERR=Data too
>>> long for column 'PluginName' at row 1
>>>
>>> bacula-sd Sending spooled attrs to the Director. Despooling 8,214 bytes
>>> ...
>>> bacula-fd
>>> Error: kubernetes: Error closing backend. Err=Child exited with code 1
>>> Fatal error: kubernetes: Wrong backend response to JobEnd command.
>>> bacula-sd Elapsed time=00:02:02, Transfer rate=659 Bytes/second
>>> bacula-fd
>>> Error: kubernetes: PTCOMM cannot get packet header from backend.
>>>
>>> Error: kubernetes: Cannot successfully start bacula-backup pod in
>>> expected time!
>>>
>>> Error: kubernetes: Job already running in 'namespace' namespace. Check
>>> logs or delete bacula-backup Pod manually.
>>>
>>>
>>> Best regards,
>>> Zsolt
>>>
>>> _______________________________________________
>>> Bacula-devel mailing list
>>> Bacula-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bacula-devel
>>>
>>
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
