Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
Hello Martin, I didn't know it exists, so I will take a look at it. I imagine it will probably be a rather large rewrite of the existing Bacula code for AWS S3, but if it can fix all the problems that we have with libs3, it would probably be worth the effort. Best regards, Kern On 7/1/19 4:29 PM, Martin Simmons wrote: On Mon, 1 Jul 2019 13:11:07 +0200, Kern Sibbald said: Do you see any other alternative for a C/C++ than to use libs3? Have you already ruled out using the official SDK (https://aws.amazon.com/sdk-for-cpp/)? __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
> On Mon, 1 Jul 2019 13:11:07 +0200, Kern Sibbald said: > > Do you see any other alternative for a C/C++ than to use libs3? Have you already ruled out using the official SDK (https://aws.amazon.com/sdk-for-cpp/)? __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
On 01.07.19 13:11, Kern Sibbald wrote: > Yes, this is a problem. I remember we discussed libs3 some time ago, > but I did not realize that it has a number of security issues. I am no C/C++ wizard but just the fact that I need to disable about all checks and warnings a modern compiler has to even get the code to compile tells me as much as I need to know. Also looking at the commits after December 2016 tells the same, as there are several commits fixing memory leaks and buffer overflows. > Do you see any other alternative for a C/C++ than to use libs3? I'm sorry, unfortunately everybody seems to have forked and embedded libs3 at some point in the past and just uses that version for their projects, with all the problems it contains. Everyone else seems to use Rust or Go for their S3-based projects. :) > For other S3 vendors, Bacula Systems has resorted in using the vendor > supplied command line tools to access the their clouds. This gets > around the incompatibility of S3 implementations problems, but it opens > new problems in that the vendors typically supply binaries, and if they > have problems or bugs there is no way to fix them. So far, I have no > good solution to the problem. I would be interested in any of your > suggestions. I am afraid I really have none at the moment. > In fact, if I thought there were enough Bacula users using AWS S3, I > would even consider fixing and maintaining the libs3 package myself. Of > course that would be workable only if Debian and perhaps other vendors > would adopt such a project. The Debian policy allows for embedding libraries in special circumstances, for example if the forked code diverges enough from the library already in the Debian repository that using the existing one is difficult or impossible. (Besides, the current libs3 has no reverse dependencies and is essentially useless and should be removed.) There should be no problems if you fork libs3, fix it to meet modern security and coding standards and just embed it into Bacula to be statically linked to the SD or just the plugin. But since I am not DD/DM, take my words with caution. Grüße, Sven. signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
Hello Sven, Yes, this is a problem. I remember we discussed libs3 some time ago, but I did not realize that it has a number of security issues. I do know that it is no longer maintained by the original maintainer. Bacula Systems has been updating the source code as changes are needed to make our cloud driver work. Do you see any other alternative for a C/C++ than to use libs3? For other S3 vendors, Bacula Systems has resorted in using the vendor supplied command line tools to access the their clouds. This gets around the incompatibility of S3 implementations problems, but it opens new problems in that the vendors typically supply binaries, and if they have problems or bugs there is no way to fix them. So far, I have no good solution to the problem. I would be interested in any of your suggestions. In fact, if I thought there were enough Bacula users using AWS S3, I would even consider fixing and maintaining the libs3 package myself. Of course that would be workable only if Debian and perhaps other vendors would adopt such a project. Best regards, Kern On 6/30/19 5:17 PM, Sven Hartge wrote: On 30.06.19 16:57, Sven Hartge wrote: We would need to ship the special libs3 either as embedded code inside the Bacula source package or package it as a separate libs3-bacula package. Both will get an instant veto from the Security Team and the Release Managers. (Debian Policy §4.13) The main problem here is that libs3 is already in Debian, but in the ancient version 2.0 which doesn't even work with today's S3 anymore. (It really should be removed from Debian, to be honest.) After version 2.0, there have no new official releases been made by the upstream developer and looking at the commit history on Github it seems largely abandoned, aside from some merges PRs here and there. When Bacula Systems implemented the Cloud driver in 2016.12.09 it used the then git-HEAD of the library but it was still in flux back then, so any version a bit older or a bit newer will not work with Bacula, so even if Debian shipped a newer version, it would be incompatible with Bacula. Now, that special version from 2016.12.09 still contains some very jarring problems including buffer overflows or memory leaks. No distribution would want to support that code. But we already went over this back when 9.4.0 was released and Carsten and I tried to package the Cloud driver for Debian but gave up when it dawned on us what an ordeal it would be to achieve this in the constraints of Debian. Grüße, Sve. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
On 30.06.19 16:57, Sven Hartge wrote: > We would need to ship the special libs3 either as embedded code inside > the Bacula source package or package it as a separate libs3-bacula > package. Both will get an instant veto from the Security Team and the > Release Managers. (Debian Policy §4.13) The main problem here is that libs3 is already in Debian, but in the ancient version 2.0 which doesn't even work with today's S3 anymore. (It really should be removed from Debian, to be honest.) After version 2.0, there have no new official releases been made by the upstream developer and looking at the commit history on Github it seems largely abandoned, aside from some merges PRs here and there. When Bacula Systems implemented the Cloud driver in 2016.12.09 it used the then git-HEAD of the library but it was still in flux back then, so any version a bit older or a bit newer will not work with Bacula, so even if Debian shipped a newer version, it would be incompatible with Bacula. Now, that special version from 2016.12.09 still contains some very jarring problems including buffer overflows or memory leaks. No distribution would want to support that code. But we already went over this back when 9.4.0 was released and Carsten and I tried to package the Cloud driver for Debian but gave up when it dawned on us what an ordeal it would be to achieve this in the constraints of Debian. Grüße, Sve. signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
On 30.06.19 14:21, Kern Sibbald wrote: > The Bacula project provides the correct libs3 package at > > https://www.bacula.org/downloads/libs3-20181010.tar.gz > > Normally when building, you download the .tar.gz file, detar it, build > the package then make sure your packaging uses the special version > rather than the one supplied on the system. This is all fine and good, Kern, but forbidden by the Debian policy. During build, not external downloads or even network connections are allowed. Doing so would be an instant FTBFS and thus release-critical bug. Debian Policy §4.9 cleary states: "For packages in the main archive, no required targets may attempt network access, except, via the loopback interface, to services on the build host that have been started by the build." We would need to ship the special libs3 either as embedded code inside the Bacula source package or package it as a separate libs3-bacula package. Both will get an instant veto from the Security Team and the Release Managers. (Debian Policy §4.13) So, unfortunately, it is impossible for Debian (and Ubuntu) to ship the Cloud Driver in an official package at the current time. Grüße, Sven. signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
Hello Sven, The Bacula project provides the correct libs3 package at https://www.bacula.org/downloads/libs3-20181010.tar.gz Normally when building, you download the .tar.gz file, detar it, build the package then make sure your packaging uses the special version rather than the one supplied on the system. Best regards, Kern On 6/28/19 6:24 PM, Sven Hartge wrote: On 28.06.19 17:32, Antonino Balsamo wrote: I'm running bacula 9.4.2 on stretch with debian packages from stretch-backports. I've added the cloud plugin from the Bacula package and I got the following error: JobId 2: Fatal error: cloud_dev.c:690 Could not open Cloud driver type=1 for Device=CloudStorage Then I removed all packages and installed 9.4.4 from Bacula archive including cloud plugin, the error remains the same. Going to the source code, it seems that HAVE_LIBS3 always evaluate to false. I do not think I have the talent to recompile the packages, but maybe I'm doing a mistake somewhere else? The Cloud driver is not compiled and supported in the Debian packages because it needs a very special, very old version of libs3, not present in Debian and unfortunately it is also not possible to get that version into Debian at the current moment. So I am afraid Debian will not be able to provide Cloud-support in Bacula for the foreseeable future. Grüße, Sven (Debian Bacula Co-Maintainer) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
On 28.06.19 17:32, Antonino Balsamo wrote: > I'm running bacula 9.4.2 on stretch with debian packages from > stretch-backports. I've added the cloud plugin from the Bacula package > and I got the following error: > > JobId 2: Fatal error: cloud_dev.c:690 Could not open Cloud driver type=1 > for Device=CloudStorage > > Then I removed all packages and installed 9.4.4 from Bacula archive > including cloud plugin, the error remains the same. > > Going to the source code, it seems that HAVE_LIBS3 always evaluate to > false. > > I do not think I have the talent to recompile the packages, but maybe > I'm doing a mistake somewhere else? The Cloud driver is not compiled and supported in the Debian packages because it needs a very special, very old version of libs3, not present in Debian and unfortunately it is also not possible to get that version into Debian at the current moment. So I am afraid Debian will not be able to provide Cloud-support in Bacula for the foreseeable future. Grüße, Sven (Debian Bacula Co-Maintainer) signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] cloud_dev could not open Cloud driver type=1 for Device=CloudStorage
HI. I'm running bacula 9.4.2 on stretch with debian packages from stretch-backports. I've added the cloud plugin from the Bacula package and I got the following error: JobId 2: Fatal error: cloud_dev.c:690 Could not open Cloud driver type=1 for Device=CloudStorage Then I removed all packages and installed 9.4.4 from Bacula archive including cloud plugin, the error remains the same. Going to the source code, it seems that HAVE_LIBS3 always evaluate to false. I do not think I have the talent to recompile the packages, but maybe I'm doing a mistake somewhere else? kind regards Ant ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
