Re: [openssl-users] Find size of available data prior to ssl_read
Although maybe the simple answer is to read into a temporary 32 KB buffer and then malloc and copy. -- View this message in context: http://openssl.6102.n7.nabble.com/Find-size-of-available-data-prior-to-ssl-read-tp61722p61734.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] Changing malloc/debug stuff
I want to change the memory alloc/debug things. Right now there are several undocumented functions to allow you to swap-out the malloc/realloc/free routines, wrappers that call those routines, debug versions of those wrappers, and functions to set the set-options versions of those functions. Yes, really :) Is anyone using that stuff? I want to change the model so that there are three wrappers around malloc/realloc/free, and that the only thing you can do is change that wrapper. This is vastly simpler and easier to understand. I also documented it. A version can be found at https://github.com/openssl/openssl/pull/450 I've posted about this before. But I'm asking again if this kind of change will cause problems for anyone. Thanks. -- Senior Architect, Akamai Technologies IM: richs...@jabber.at Twitter: RichSalz ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
Thanks to Michael and Kurt for explanatory comments. Is there an available setting that gives the upper limit on the amount of data that will be obtained by a single ssl_read()? The data stream is SQL requests, and often these are quite small, but they can run to megabytes. I need to malloc a buffer for the data. If it is too small, that will impose extra processing overheads in the rest of the system. If it is too large, it will impose memory wastage on the rest of the system. The system has an upper limit of 32 KB on the initial size of a buffer for reading, but that is way above the typical SQL request. So, accepting that I can't set the size precisely, if there is a limit for SSL data reads that is significantly lower than 32 KB then that might be a feasible fixed buffer size. If that isn't possible, maybe it will have to be a tunable configuration value. Any comments? -- View this message in context: http://openssl.6102.n7.nabble.com/Find-size-of-available-data-prior-to-ssl-read-tp61722p61733.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf > Of counterpoint > Sent: Thursday, December 17, 2015 04:51 > > Although maybe the simple answer is to read into a temporary 32 KB buffer and > then malloc and copy. That, more or less, was my recommendation in my previous post. The optimal size of the temporary buffer depends on factors we don't know. If most of your messages fit in 32KB, then that may save you extra calls to SSL_read. On the other hand, it could mean excessive copying - it might be better to use a smaller buffer to reduce the size of the additional copy operation, even at the cost of an extra call to SSL_read. (Obviously some copying is happening in the SSL/TLS processing anyway, and the cost of such copying is small relative to the cost of decryption and other compute-intensive operations. But if your application deals with a high transaction rate then cutting down that extra copy may be worthwhile anyway.) If your application is single-threaded, you can make that a static buffer; if not, it needs to go on the stack, which could be a problem if your threads are stack-constrained. That's another argument (if it applies to your case) for using a smaller initial buffer. If the first chunk of your message tells you how large the entire message will be, then this approach means only one call to the allocator per message received, which is good. And it means the same code path for every message regardless of size, which is good for program correctness and maintainability. Based on what you've told us, this is the approach I'd recommend. The only question is the size of that initial buffer, and you're in a better position to determine that. -- Michael Wojcik Technology Specialist, Micro Focus ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
On 17/12/2015 10:36, counterpoint wrote: Thanks to Michael and Kurt for explanatory comments. Is there an available setting that gives the upper limit on the amount of data that will be obtained by a single ssl_read()? The data stream is SQL requests, and often these are quite small, but they can run to megabytes. I need to malloc a buffer for the data. If it is too small, that will impose extra processing overheads in the rest of the system. If it is too large, it will impose memory wastage on the rest of the system. The system has an upper limit of 32 KB on the initial size of a buffer for reading, but that is way above the typical SQL request. So, accepting that I can't set the size precisely, if there is a limit for SSL data reads that is significantly lower than 32 KB then that might be a feasible fixed buffer size. If that isn't possible, maybe it will have to be a tunable configuration value. Any comments? The current SSL/TLS standards limits the per record data size to 16K exactly, see for example RFC5246 section 6.2.1. However the data you want in your (higher level) code probably has a completely different natural size limit/unit which may be larger and smaller. For SQL there is no natural limit however, unless your SQL parser happens to fail on statements above some arbitrary size. Enjoy and Merry Christmas Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Changing malloc/debug stuff
On 17/12/2015 10:28, Salz, Rich wrote: I want to change the memory alloc/debug things. Right now there are several undocumented functions to allow you to swap-out the malloc/realloc/free routines, wrappers that call those routines, debug versions of those wrappers, and functions to set the set-options versions of those functions. Yes, really J Is anyone using that stuff? I want to change the model so that there are three wrappers around malloc/realloc/free, and that the only thing you can do is change that wrapper. This is vastly simpler and easier to understand. I also documented it. A version can be found at https://github.com/openssl/openssl/pull/450 I’ve posted about this before. But I’m asking again if this kind of change will cause problems for anyone. I don't need it so I don't object. But if anyone objects, you could write a compatibility module that can be called to use the new interface to change in a compatible variant of the old wrappers, which would then provide the old hooks/facilities when needed while staying out of the way (not even being linked into static programs) for the rest of us. I guess this is because that interface is not a part of a commercial grade full featured SSL/TLS and general purpose crypto library, it is just a means to do quality assurance on said library. Enjoy and Merry Christmas Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
Thanks, that makes sense. My ability to optimise is constrained - the system is a product so I do not know what the actual pattern of usage will be. But there is a limit on buffer size within the system. It's a defined symbol, so can be altered from the default of 32 KB, but only by recompiling the system. I rely on a working assumption that people who change definitions and recompile know what they're doing. The system is threaded, but it is designed to operate with a relatively small number of highly active threads, so grabbing 32 KB on the stack for a short period shouldn't be too much of an issue. It would be much harder to figure out the actual message size because the calls to SSL are taking place in a generic core, whereas the protocol is in a different layer of code. There are ways it could be done, but I'm inclined to leave that for a future optimisation. That leaves me feeling that the fixed buffer on the stack is the cleanest solution, involving simple code. The copying overhead is there, but looks hard to eliminate, and as you say there is plenty of other overhead. I'm not sure that the small initial buffer offers me much gain, although it might help in some situations. (Personally I'm inclined to use SSH tunnels rather than SSL for SQL traffic, but that's another story!). One remaining point leaves me uncertain. Supposing an SSL write gets the response SSL_ERROR_WANT_READ. Then there is a POLLIN event. I take it the first thing that must happen is a retry of the write. Assuming that works, do I need to assume that there could be data to be read? Or will a further event occur, so that I should return to looking out for events? I guess the answer to the last question is probably no, but am unsure. -- View this message in context: http://openssl.6102.n7.nabble.com/Find-size-of-available-data-prior-to-ssl-read-tp61722p61741.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf > Of counterpoint > Sent: Thursday, December 17, 2015 11:35 > > Thanks, that makes sense. My ability to optimise is constrained - the system > is a product so I do not know what the actual pattern of usage will be. But > there is a limit on buffer size within the system. It's a defined symbol, so > can be altered from the default of 32 KB, but only by recompiling the > system. I rely on a working assumption that people who change definitions > and recompile know what they're doing. Fair enough. > The system is threaded, but it is designed to operate with a relatively > small number of highly active threads, so grabbing 32 KB on the stack for a > short period shouldn't be too much of an issue. It's not really a matter of how many threads there are (except indirectly), or of how long the item is on the stack. It's a question of how much space is available on the thread's stack when you try to allocate the buffer (which, assuming we're talking C or C++, is when you enter the function / method). A thread's stack size is typically set at creation time, with a default that may be fixed in the threading implementation or set at link time. How much space is available when you allocate that 32 KB buffer depends on how deep your call chain is and how much data each of those frames adds to the stack. If the stack is too small to accommodate the buffer and can't be expanded, you'll get some kind of run-time failure, like a Windows exception or a UNIX signal. Note that stack space is an address-space resource, not (generally) a virtual memory one - that is, stack-space is unlikely to be constrained because the system is running short on virtual memory. It'll happen because most language implementations use contiguous stacks for performance (rather than, say, displays or other non-contiguous structures), and if the stack runs into something else in the process address space, it can't grow any further. So if your process is 64-bit, you should be able to specify ridiculously large thread stacks and not worry about it. If the process is 32-bit, take a look at your thread stack sizes and do a quick estimate on how much space you expect will be there. You can determine this for a specific thread, in a specific run, in a debugger by looking at the address of an automatic variable at the bottom of the thread's stack (in the thread's initial function) and the address of one in your data-receiving function. (Technically comparing those addresses isn't authorized by the language standard, but it's valid on most of the platforms OpenSSL supports.) So I'd say try it in some test runs and see if it looks like stack space might be getting tight; if so, you can likely increase the stack size you specify when creating your threads, since you don't have many of them. > One remaining point leaves me uncertain. Supposing an SSL write gets the > response SSL_ERROR_WANT_READ. Then there is a POLLIN event. I take it > the > first thing that must happen is a retry of the write. Assuming that works, > do I need to assume that there could be data to be read? Or will a further > event occur, so that I should return to looking out for events? I guess the > answer to the last question is probably no, but am unsure. There could be data to be read. Consider this scenario: 1. The peer decides it wants to renegotiate during the conversation. 2. In the middle of the handshake, you call SSL_write. The handshake hasn't completed, and the local side is waiting for a message from the peer, so SSL_write returns SSL_ERROR_WANT_READ. 3. You wait for POLLIN, then call SSL_write again. 4. Before SSL_write returns, the peer has time to respond to the request you just sent. Or it sends something else immediately after completing the handshake, if your application doesn't use a strict switched-duplex request-response protocol. So I'd recommend going ahead and trying a non-blocking SSL_read at that point. The overhead is tiny and you won't miss any inbound-data events. -- Michael Wojcik Technology Specialist, Micro Focus ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Find size of available data prior to ssl_read
Thanks, very helpful. We only support 64 bit. -- View this message in context: http://openssl.6102.n7.nabble.com/Find-size-of-available-data-prior-to-ssl-read-tp61722p61746.html Sent from the OpenSSL - User mailing list archive at Nabble.com. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Changing malloc/debug stuff
> I don't need it so I don't object. But if anyone objects, you could write a > ... Good point! > I guess this is because that interface is not a part of a commercial grade > full > featured SSL/TLS and general purpose crypto library, it is just a means to do > quality assurance on said library. That seems to be the main usage, yes. I think it had more uses in the early days such as on old windows/msdos? > Enjoy and Merry Christmas And a happy new year. Or bah, humbug :) /r$ ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Changing malloc/debug stuff
On 17/12/2015 19:03, Salz, Rich wrote: I don't need it so I don't object. But if anyone objects, you could write a ... Good point! I guess this is because that interface is not a part of a commercial grade full featured SSL/TLS and general purpose crypto library, it is just a means to do quality assurance on said library. That seems to be the main usage, yes. I think it had more uses in the early days such as on old windows/msdos? Old Windows/msdos was the most widespread platform where the compiler provided malloc/free tended to be crap and had to be overridden in a compiler specific way to make it work reasonably. Note that this was a *compiler* specific issue and tended to depend on both the brand, version and sometimes even command line options of the compiler. However even for Win16, MS-DOS and OS/2 16-bit, the ability to redirect all memory calls through a user provided set of malloc/free/realloc/msize functions would do the job. The hard part was writing those override functions, even with a your copies of TAOCP and K (2nd ed) handy. A completely different use for special memory allocation work would be to take advantage of algorithm specific knowledge to optimize allocation and system call patterns, such as keeping all the small allocations for a decoded X.509 certificate or all the intermediaries for an RSA calculation together. Enjoy and Merry Christmas Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] [openssl-dev] Changing malloc/debug stuff
> > https://github.com/openssl/openssl/pull/450 > > This seems much more sane. I'll settle for less insane :) ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] [openssl-dev] Changing malloc/debug stuff
On Thu, Dec 17, 2015 at 08:16:50PM +, Salz, Rich wrote: > > > https://github.com/openssl/openssl/pull/450 > > > > This seems much more sane. > > I'll settle for less insane :) That is, I think, the best you can do. Some allocations might have taken place by the time a wrapper or alternative allocator is installed, in which case something bad will happen. In the case of alternative allocators the something bad is "it blows up", while in the case of a wrapper the something bad is "some state/whatever will be off". A fully sane approach would be to have every allocated object internally point to its destructor, and then always destroy by calling that destructor instead of a global one. (Or call a global one that knows how to find the object's private destructor pointer, and then calls that.) If you wish, something more OO-ish. But for many allocations that's not possible because they aren't "objects" in the sense that matters. You could always wrap allocations so that they always have room at the front for the corresponding destructor, then return the offset of the end of that pointer, but this will be very heavy-duty for many allocations. So, all in all, I like and prefer your approach. Nico -- ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] RSA and FIPS 186-4 in OpenSSL 1.0.1e/fips-2.0.9
Hello, I have read about the use of FIPS_rsa_x931_generate_key_ex() for 186-4 compliance. We are using OpenSSL 1.0.1e with the fips-2.0.9 module. Would it make functional sense using those versions to patch RSA_generate_key_ex() (../crypto/rsa/rsa_gen.c) to have: #ifdef OPENSSL_FIPS if (FIPS_mode()) return FIPS_rsa_x931_generate_key_ex(rsa, bits, e_value, cb); #endif Instead of using FIPS_rsa_generate_key_ex() (and also adding the prototype for FIPS_rsa_x931_generate_key_ex() earlier in rsa_gen.c) Thanks. ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] RSA and FIPS 186-4 in OpenSSL 1.0.1e/fips-2.0.9
On Thu, Dec 17, 2015 at 04:26:21PM -0500, jonetsu wrote: > Hello, > > > I have read about the use of FIPS_rsa_x931_generate_key_ex() for 186-4 > compliance. We are using OpenSSL 1.0.1e with the fips-2.0.9 module. Would > it make functional sense using those versions to patch RSA_generate_key_ex() > (../crypto/rsa/rsa_gen.c) to have: > > > #ifdef OPENSSL_FIPS > if (FIPS_mode()) > return FIPS_rsa_x931_generate_key_ex(rsa, bits, e_value, cb); > #endif > > > Instead of using FIPS_rsa_generate_key_ex() > > > (and also adding the prototype for FIPS_rsa_x931_generate_key_ex() earlier in > rsa_gen.c) I do not think this x931 RSA key generation is 186-4 compliant. Ciao, Marcus ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] Segfault in libcrypto.so
Hello, I have been trying to install a product named wanguard and each time am starting a collector I receive this error message: segfault at efe000 ip 7ffb571e479c sp 7ffced00dcf0 error 4 in libcrypto.so.1.0.0[7ffb57166000+1cb000] And the collector stops immediately. Has anyone encountered this error or can someone help please? Thanks. Regards, Alex ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
