Re: [PATCH v2 2/2] crypto: scomp - add support for deflate rfc1950 (zlib)
Hi,
I have a few queries regarding zlib and in general on whether changes
to acomp interface is planned or would be acceptable. Some of these
are from point of view of exporting compression to user space and
utilizing HW implementations.
- A setup api is required to support different configurations; this
was also proposed as part of initial acomp implementations.
- Zlib also supports notion of preset dictionary and for
decompression, history bytes upto window size might be required to be
provided as part of the request. For HW algorithms supporting these
features, the current acomp api's would need to be extended.
- Currently acomp supports operations only in one go, are there plans
to extend acomp beyond this? For hardware algos supporting partial
ops, we will need to add fields like bytes consumed, flush params,ctx
parameter etc.and export to user.
Thanks,
Abed (Cavium)
On Sat, Apr 22, 2017 at 2:24 AM, Giovanni Cabiddu
wrote:
> Add scomp backend for zlib-deflate compression algorithm.
> This backend outputs data using the format defined in rfc1950
> (raw deflate surrounded by zlib header and footer).
>
> Signed-off-by: Giovanni Cabiddu
> ---
> crypto/deflate.c | 61 -
> crypto/testmgr.c | 10
> crypto/testmgr.h | 75
>
> 3 files changed, 129 insertions(+), 17 deletions(-)
>
> diff --git a/crypto/deflate.c b/crypto/deflate.c
> index f942cb3..94ec3b3 100644
> --- a/crypto/deflate.c
> +++ b/crypto/deflate.c
> @@ -43,20 +43,24 @@ struct deflate_ctx {
> struct z_stream_s decomp_stream;
> };
>
> -static int deflate_comp_init(struct deflate_ctx *ctx)
> +static int deflate_comp_init(struct deflate_ctx *ctx, int format)
> {
> int ret = 0;
> struct z_stream_s *stream = &ctx->comp_stream;
>
> stream->workspace = vzalloc(zlib_deflate_workspacesize(
> - -DEFLATE_DEF_WINBITS, DEFLATE_DEF_MEMLEVEL));
> + MAX_WBITS, MAX_MEM_LEVEL));
> if (!stream->workspace) {
> ret = -ENOMEM;
> goto out;
> }
> - ret = zlib_deflateInit2(stream, DEFLATE_DEF_LEVEL, Z_DEFLATED,
> - -DEFLATE_DEF_WINBITS, DEFLATE_DEF_MEMLEVEL,
> - Z_DEFAULT_STRATEGY);
> + if (format)
> + ret = zlib_deflateInit(stream, 3);
> + else
> + ret = zlib_deflateInit2(stream, DEFLATE_DEF_LEVEL, Z_DEFLATED,
> + -DEFLATE_DEF_WINBITS,
> + DEFLATE_DEF_MEMLEVEL,
> + Z_DEFAULT_STRATEGY);
> if (ret != Z_OK) {
> ret = -EINVAL;
> goto out_free;
> @@ -68,7 +72,7 @@ static int deflate_comp_init(struct deflate_ctx *ctx)
> goto out;
> }
>
> -static int deflate_decomp_init(struct deflate_ctx *ctx)
> +static int deflate_decomp_init(struct deflate_ctx *ctx, int format)
> {
> int ret = 0;
> struct z_stream_s *stream = &ctx->decomp_stream;
> @@ -78,7 +82,10 @@ static int deflate_decomp_init(struct deflate_ctx *ctx)
> ret = -ENOMEM;
> goto out;
> }
> - ret = zlib_inflateInit2(stream, -DEFLATE_DEF_WINBITS);
> + if (format)
> + ret = zlib_inflateInit(stream);
> + else
> + ret = zlib_inflateInit2(stream, -DEFLATE_DEF_WINBITS);
> if (ret != Z_OK) {
> ret = -EINVAL;
> goto out_free;
> @@ -102,21 +109,21 @@ static void deflate_decomp_exit(struct deflate_ctx *ctx)
> vfree(ctx->decomp_stream.workspace);
> }
>
> -static int __deflate_init(void *ctx)
> +static int __deflate_init(void *ctx, int format)
> {
> int ret;
>
> - ret = deflate_comp_init(ctx);
> + ret = deflate_comp_init(ctx, format);
> if (ret)
> goto out;
> - ret = deflate_decomp_init(ctx);
> + ret = deflate_decomp_init(ctx, format);
> if (ret)
> deflate_comp_exit(ctx);
> out:
> return ret;
> }
>
> -static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
> +static void *gen_deflate_alloc_ctx(struct crypto_scomp *tfm, int format)
> {
> struct deflate_ctx *ctx;
> int ret;
> @@ -125,7 +132,7 @@ static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
> if (!ctx)
> return ERR_PTR(-ENOMEM);
>
> - ret = __deflate_init(ctx);
> + ret = __deflate_init(ctx, format);
> if (ret) {
> kfree(ctx);
> return ERR_PTR(ret);
> @@ -134,11 +141,21 @@ static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
> return ctx;
> }
>
> +static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
> +{
> + return gen_deflate_alloc_ctx(tfm, 0);
> +}
> +
> +static void *zlib_deflate_alloc_ctx(struct cry
Re: [RFC PATCH v1 1/1] crypto: algif_compression - User-space interface for compression
Hi Stephan, On Tue, Apr 18, 2017 at 3:42 PM, Stephan Müller wrote: > Am Donnerstag, 13. April 2017, 20:34:54 CEST schrieb Abed Kamaluddin: > > May I propose that you look into the patches for skcipher and aead regarding > memory handling updates that are currently discussed. There you will find the > sendmsg code with two parts: > > 1. checking of the input data of cmsg and setting the ctx respectively. > > 2. the big while loop for getting all user space data > > I guess you have seen that I would like to consolidate the algif > implementations once the memory handling patch is in and agreed on. My plan > is: Bullet 1 will be private to the algif implementation, so leave your code. > Bullet 2 will be moved into a common service function. Thus, may I propose > that you simply copy the entire while loop with the same TX data structures > into your sendmsg code. This way you do not have the mentioned limit. > > Also, simply copy the sendpage code from the patch set to support splice/ > vmsplice. > Thanks for the suggestions and helpful pointers, I will rework the patch and incorporate these, hoping that the compression interface will be exported. These changes would also remain unaffected by any changes to the acomp interface. Thanks Abed (Cavium) Regards,
Re: [RFC PATCH v1 1/1] crypto: algif_compression - User-space interface for compression
On Fri, Apr 21, 2017 at 9:20 PM, Stephan Müller wrote: > Am Freitag, 21. April 2017, 17:42:10 CEST schrieb abed mohammad kamaluddin: > > Just diff the just RFCed algif_kpp with the proposed patch set for > algif_skcipher and algif_aead. There you will see that 80% of all code is > identical (if you disregard the different namespace). And that is the code I > am referring to. > > Ciao > Stephan Thanks, I will use the pointer. Considering all the identical code, the proposal to consolidate would definitely help! Thanks, Abed
Re: [RFC PATCH v1 0/1] *** crypto: AF_ALG: add compression support ***
Hi Herbert, >> > I think we should convert ipcomp over to the new interface first >> > in order to make sure that we don't need to change the interface >> > which would be hard to do once we export it to user-space. >> >> Would it be acceptable if we export an algif interface using >> alg_type_compress, > > No we're not going to export an obsolete interface. > Are there any deficiencies expected in the acomp interface for use with ipcomp? One issue mentioned earlier on the lists was that the interface might require tweaks for piecemeal decompression. Has this been taken up? We identified few issues when using acomp with user-space zlib for file compression. If the the interface is tweaked keeping those use-cases in mind apart from the existing kernel apps (zswap/ipcomp/zram), the hardware algorithms can be utilized using the algif framework. Thanks Abed On Thu, Apr 20, 2017 at 1:30 PM, Herbert Xu wrote: > On Thu, Apr 20, 2017 at 12:39:58PM +0530, abed mohammad kamaluddin wrote: >> Hi Herbert, >> >> > I think we should convert ipcomp over to the new interface first >> > in order to make sure that we don't need to change the interface >> > which would be hard to do once we export it to user-space. >> >> Would it be acceptable if we export an algif interface using >> alg_type_compress, >> which is being used by ipcomp currently? We would like to use our zip >> device from >> userspace through this interface. > > No we're not going to export an obsolete interface. > > Cheers, > -- > Email: Herbert Xu > Home Page: http://gondor.apana.org.au/~herbert/ > PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [RFC PATCH v1 0/1] *** crypto: AF_ALG: add compression support ***
Hi Herbert, > I think we should convert ipcomp over to the new interface first > in order to make sure that we don't need to change the interface > which would be hard to do once we export it to user-space. Would it be acceptable if we export an algif interface using alg_type_compress, which is being used by ipcomp currently? We would like to use our zip device from userspace through this interface. Thanks Abed (Cavium) On Fri, Apr 14, 2017 at 2:02 PM, Herbert Xu wrote: > On Fri, Apr 14, 2017 at 12:04:53AM +0530, Abed Kamaluddin wrote: >> Hi Herbert, >> >> This patch adds compression support to the AF_ALG interface exported by the >> kernel crypto API. By extending AF_ALG, all compression algorithms of types >> scomp and acomp, which the kernel crypto API allows access to, are now also >> accessible from userspace. >> >> The new compression interface has been tested with both kernel software >> deflate(scomp) and HW accelerated ThunderX deflate(scomp) using the zpipe >> example application provided by zlib. > > I think we should convert ipcomp over to the new interface first > in order to make sure that we don't need to change the interface > which would be hard to do once we export it to user-space. > > Thanks, > -- > Email: Herbert Xu > Home Page: http://gondor.apana.org.au/~herbert/ > PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: algif for compression?
Hi Stephen,
> If you follow the AF_ALG implementations that are currently in the kernel, a
> calling application receives two file descriptors. The first file descriptor
> is the reference to a tfm, the second is the reference to a particular
> request.
>
> Wouldn't those file descriptors be the reference you are looking for?
Those descriptors are sufficient to pass data from userspace
applications to the algif interface.
However the issue is passing those from the interface to the driver
using the current acomp API's.
This is the currently exposed interface to the comp framework. Am I
missing something here?
int (*compress)(struct acomp_req *req);
int (*decompress)(struct acomp_req *req);
struct acomp_req {
struct crypto_async_request base;
struct scatterlist *src;
struct scatterlist *dst;
unsigned int slen;
unsigned int dlen;
u32 flags;
void *__ctx[] CRYPTO_MINALIGN_ATTR;
};
Thanks
Abed
On Mon, Apr 3, 2017 at 12:56 PM, Stephan Müller wrote:
> Am Montag, 3. April 2017, 08:10:19 CEST schrieb abed mohammad kamaluddin:
>
> Hi abed,
>
>> Hi Herbert,
>>
>> We have implemented algif acomp interface for user space applications
>> to utilize the kernel compression framework and the hardware drivers
>> using it and have been testing it using userspace zlib library.
>>
>> However, what we find lacking in the existing acomp implementation is
>> the ability to pass context data between the applications and the
>> drivers using the acomp interface. Currently the interface only allows
>> src/dest data and a flag argument with each request. There are two
>> context pointers, one in acomp_req and another in crypto_tfm but they
>> are for internal use and not available to applications through the
>> api's. Would it be acceptable to add fields that need to be
>> communicated b/w the driver and applications like history,
>> csum/adler32, EOF, stream ctx to acomp_req.
>>
>> Or is there any other way, which I may have missed, through which we
>> can pass ctx data between applications and drivers while using the
>> kernel compression framework?
>
> If you follow the AF_ALG implementations that are currently in the kernel, a
> calling application receives two file descriptors. The first file descriptor
> is the reference to a tfm, the second is the reference to a particular
> request.
>
> Wouldn't those file descriptors be the reference you are looking for?
>
> Ciao
> Stephan
Re: algif for compression?
Hi Herbert, We have implemented algif acomp interface for user space applications to utilize the kernel compression framework and the hardware drivers using it and have been testing it using userspace zlib library. However, what we find lacking in the existing acomp implementation is the ability to pass context data between the applications and the drivers using the acomp interface. Currently the interface only allows src/dest data and a flag argument with each request. There are two context pointers, one in acomp_req and another in crypto_tfm but they are for internal use and not available to applications through the api's. Would it be acceptable to add fields that need to be communicated b/w the driver and applications like history, csum/adler32, EOF, stream ctx to acomp_req. Or is there any other way, which I may have missed, through which we can pass ctx data between applications and drivers while using the kernel compression framework? Thanks, Abed (Cavium Inc) On Sat, Dec 10, 2016 at 1:40 PM, Herbert Xu wrote: > abed mohammad kamaluddin wrote: >> >> We are also looking for user-space access to the kernel crypto layer >> compression algorithms. I have gone through AF_ALG but couldn’t find >> support for compression ops through it. This is required for our >> hardware zip engines whose driver hooks up to the crypto layer. >> >> I was going through the lists and stumbled across this thread. Has >> there been any updates or efforts put up in this direction since this >> thread is a few years old. If not, are there any alternate >> implementations that allow user-space access to compression? I have >> gone through cryptodev and see the same limitation there. >> >> Would appreciate any pointers in this regard. > > The compression interface is currently in a state of flux. We > should make it settle down first before exporting it to user-space. > > For a start it would be good to actually switch IPsec/IPcomp over > to the new compression interface. > > Cheers, > -- > Email: Herbert Xu > Home Page: http://gondor.apana.org.au/~herbert/ > PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: algif for compression?
> > The compression interface is currently in a state of flux. We > should make it settle down first before exporting it to user-space. > > For a start it would be good to actually switch IPsec/IPcomp over > to the new compression interface. Thanks Herbert. Are there timelines or ongoing efforts for moving IPcomp/Ipsec to use acomp? Or any proposals that have been or need to be taken up in this regard. Thanks, Abed -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: algif for compression?
Hi, >> I see the algif_hash and algif_blkcipher implementations to allow >> userspace AF_ALG socket access to kernel blkcipher and hash >> algorithms, but has anyone done a algif_compression to allow userspace >> access to compression algs? I'm asking specifically wrt the 842 >> crypto module, which uses the hardware compression coprocessors on >> newer powerpc systems. >> >> If not, is there any reason against adding a algif_compression to >> provide the access? > > For a start nx-842 isn't even hooked into the crypto layer so > there is no chance of exporting it through algif as is. > > There is no reason why you couldn't add it of course but that'd > be a first step. We are also looking for user-space access to the kernel crypto layer compression algorithms. I have gone through AF_ALG but couldn’t find support for compression ops through it. This is required for our hardware zip engines whose driver hooks up to the crypto layer. I was going through the lists and stumbled across this thread. Has there been any updates or efforts put up in this direction since this thread is a few years old. If not, are there any alternate implementations that allow user-space access to compression? I have gone through cryptodev and see the same limitation there. Would appreciate any pointers in this regard. Thanks, Abed (Cavium Inc) -- To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
