Hi Radosław,
Re timeout:
It was 120 sec because I used the following plugin configuration (yes, it
was 600 sec by default, I've just decreased it to 120):
Plugin = "kubernetes: debug=1 baculaimage=repo/bacula-backup:04jan23
namespace=some pvcdata pluginhost=kubernetes.server *timeout=120*
verify_ssl=0 fdcertfile=/etc/bacula/certs/bacula-backup.cert
fdkeyfile=/etc/bacula/certs/bacula-backup.key"
I'm in the POD:
root@bacula-backup:/# df -h
Filesystem
Size Used Avail Use% Mounted on
overlay
99G 69G 26G 73% /
tmpfs
64M 0 64M 0% /dev
server:/kubernetes/some-claim-pvc-9166-b3f881c404cf
99G 77G 23G 78% /backup
/dev/mapper/vg--ssd-var
99G 69G 26G 73% /etc/hosts
shm
64M 0 64M 0% /dev/shm
tmpfs
64M 12K 64M 1% /run/secrets/
kubernetes.io/serviceaccount
tmpfs
63G 0 63G 0% /proc/acpi
tmpfs
63G 0 63G 0% /sys/firmware
root@bacula-backup:/# ls -al /tmp/
total 292
drwxrwxrwt 1 root root 4096 Jan 13 07:45 .
drwxr-xr-x 1 root root 4096 Jan 13 07:45 ..
drwx------ 3 root root 4096 Jan 13 07:45 _MEIN9fjTi
prw-r--r-- 1 root root 0 Jan 13 07:45 baculatar.fifo
-rw-r--r-- 1 root root 0 Jan 13 07:45 baculatar.stderr
-rw-r--r-- 1 root root 0 Jan 13 07:45 baculatar.stdout
-rw------- 1 root root 286370 Dec 21 09:27 tmp1ueprfd1
So /tmp/baculatar.stdout and /tmp/baculatar.stderr are empty. :(
Re: token
:))))
Best regards,
Zsolt
On Fri, Jan 13, 2023 at 12:05 PM Radosław Korzeniewski <
rados...@korzeniewski.net> wrote:
> Hello,
>
> pt., 13 sty 2023 o 11:34 Zsolt Kozak <koza...@gmail.com> napisał(a):
>
>> Hello Radosław,
>>
>> The Bacula Filedaemon was logging these messages. Then after logging
>>
>> DEBUG:[baculak8s/util/sslserver.py:193 in handle_connection]
>> ConnectionServer:Connection from: ('192.168.XX.YY', 10541)
>> DEBUG:[baculak8s/util/sslserver.py:145 in gethello] ['Hello',
>> 'KubernetesBackup.2023-01-04_21.05.03_10', '410706']
>> DEBUG:[baculak8s/util/token.py:57 in check_auth_data] AUTH_DATA:Token:
>> xx88M5oggQJuGsPbtD........ohQjeU7PkA4YDbSwBRxTOhT
>> DEBUG:[baculak8s/util/token.py:59 in check_auth_data]
>> RECV_TOKEN_DATA:Token: xx88M5oggQJuGsPbtD....ohQjeU7PkA4YDbSwBRxTOhT
>> DEBUG:[baculak8s/util/sslserver.py:105 in authenticate]
>> ConnectionServer:Authenticated
>>
>> there was some waiting. I guess it was 60 or 120 sec or something. Then
>> the daemon was logging the following:
>>
>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:121 in handle_pod_data_recv]
>>> handle_pod_data_recv:EOT
>>>
>>
> This log message means - connection from Bacula Backup POD was closed,
> which is unusual when no data was transferred. It could be a timeout which
> by default is set to 600 secs (I could extend the `streamrecv` method to
> distinguish between closed connection and timeout). You can check if it is
> a timeout when it is 600 secs or a time set by a `timeout=xxx` plugin
> command parameter.
>
>
>> DEBUG:[baculak8s/util/sslserver.py:201 in handle_connection]
>>> ConnectionServer:Finish - disconnect.
>>> DEBUG:[baculak8s/jobs/backup_job.py:85 in __backup_pvcdata]
>>> backup_pvcdata:logs recv
>>>
>>
>> I guess there was some "waiting" between the 2 parts and after the
>> timeout the logging went on.
>>
>>
> No, in the debug log you should get the following entry showing a proper
> transfer:
> ...
> logging.debug('handle_pod_data_recv:D' + str(len(data)))
> ...
>
> so, you should get at least one entry for successful communication.
>
> Additionally you can check the Backup POD logs to see if there are any
>> issues for PVC backup.
>>
>> The POD logged the following (it was a brand new test today):
>>
>> 2023-01-13 07:45:48,817:INFO:Bacula Kubernetes backup helper version
>> 1.0-rpk. Copyright Copyright (C) 2000-2022 Kern Sibbald
>> 2023-01-13 07:45:48,819:INFO:BaculaConnection: mode=backup
>> host=kubernetes.server port=9104 token=aR8xp8......KPROZN
>> 2023-01-13 07:45:48,819:INFO:BaculaJob:
>> KubernetesBackup.2023-01-13_08.45.44_07:411957
>> 2023-01-13 07:45:48,820:INFO:Try to connect to: kubernetes.server:9104
>> 0/600
>> 2023-01-13 07:45:50,041:INFO:Connected.
>> 2023-01-13 07:45:50,041:INFO:Authenticated.
>>
>> And that's all. No more log messages but the back job went failure. :(
>>
>
> Could you check the contents of the files: /tmp/baculatar.stdout and
> /tmp/baculatar.stderr on Bacula Backup POD and what processes are running
> in this Pod (a `df -h` on Pod will be helpful too)?
> It is plausible that baculatar cannot read data from PVC and simply hangs
> on read call, so a timeout breaks your job.
>
> btw. a token is a random sequence of chars which is valid for a single job
> execution only. No need to hide it from the public. :)
>
> best regards
> --
> Radosław Korzeniewski
> rados...@korzeniewski.net
>
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
