Re: [openssl-users] ssl_pending returns 0 despite having data to read

[Matt Caswell]

On 11/01/17 20:07, Nadia Lapkovskaya wrote:
> Hi,
> 
> We are using openssl-1.0.2j. Noticed, that for http protocol everything is 
> working fine, but when we are using our own binary protocol ssl_pending 
> returns 0 all the time. We are using blocking socket. Tried with 
> SSL_CTX_set_read_ahead set and unset.
> 
> Out test server sends back any info received from the client.
> 
> Test code looks like this:
> bool write(const uint64_t* data, int count)
> {
>   int rc = SSL_write(_ssl, data, count * sizeof(uint64_t));
>   return rc > 0 ? true : false;
> }
> 
> bool read(uint64_t* data, int count)
> {
>   do {
>   int rc = SSL_read(_ssl, data, count * sizeof(uint64_t));
>   if (rc <= 0) {
>   int err = SSL_get_error(_ssl, rc);
>   std::string errs = ERR_error_string(err, nullptr);
>   return false;
>   }
>   } while (SSL_pending(_ssl));
>   return true;
> }
> 
> During first ssl_read we received eight bytes, and after that ssl_pending 
> returns 0. If we continue reading despite having no pending data, ssl_read 
> returns the rest of the data. 
> Could you please suggest what is wrong here.

There are three levels of buffered data that you need to consider:

- Data that is buffered at the network level
- Data that is buffered in OpenSSL but not yet processed (i.e. decrypted)
- Data that is buffered in OpenSSL that has been processed

SSL_pending() only tells you about the last type of data. TLS delivers
blocks of data in records and OpenSSL will decrypt an entire record in
one go. If your application only then reads some of that record then
SSL_pending() will tell you how many bytes of data it still has
available. If you always read an entire record in one go (i.e. if the
size of the buffer that you pass to SSL_read() is equal to or greater
than the amount of data in the record) then SSL_pending() will always
return 0.

Normally OpenSSL will only read one record at a time, so there isn't any
data of the second type. However if you set read_ahead then it will
attempt to read as much data as the network can give it, until the
internal buffer is filled. If that means it has read more than one
record (which could include partial records) then you will get data of
the second type. In 1.0.2 there is no way to get OpenSSL to tell you
whether it has any of that data buffered. In 1.1.0 you can find out
about this data using the new function SSL_has_pending():

https://www.openssl.org/docs/man1.1.0/ssl/SSL_pending.html

For data buffered at the network level you should query this yourself
using something like select() or poll().

Matt
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Erwann Abalea]

ISO/C 2011, clause 6.3.2.3:

An integer constant expression with the value 0, or such an expression cast to 
type void *, is called a null pointer constant. If a null pointer constant is 
converted to a pointer type, the resulting pointer, called a null pointer, is 
guaranteed to compare unequal to any object or function.

Conversion of a null pointer to another pointer type yields a null pointer of 
that type. Any two null pointers shall compare equal.


int *var1 = 0;
int *var2 = (void*)0;

result in var1 and var2 to both be null pointers (the null pointer constant 
being « 0 » or « (void*)0 »).

This doesn’t matter if your specific machine encodes null pointers as 
‘0x'.

On your specific machine, however:

int *var1;
int *var2 = 0;
memset(var1, 0, sizeof(var1));

won’t make var1 be a null pointer, but var2 will internally contain this 
0x, and will be a null pointer.


Cordialement,
Erwann Abalea

> Le 11 janv. 2017 à 17:18, Jeffrey Walton  a écrit :
> 
>> Could someone from the OpenSSL team please explain the rationale for this
>> decision? What is the problem with using assignments with 0 or NULL to
>> initialize pointers?
> 
> I'm not from the team, so take it for what its worth...
> 
> On some systems, NULL is _not_ 0. NULL can be anywhere in memory the
> architecture wants it to be. It can be in a high page in memory, too.
> One of my instructors in college was telling me about a system he
> worked on where NULL was an address in the last page in memory, so it
> took a value like `0x`.
> 
> Jeff
> -- 
> openssl-users mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
> 

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ssl_pending returns 0 despite having data to read

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf
> Of Nadia Lapkovskaya
> Sent: Wednesday, January 11, 2017 15:08
> 
> During first ssl_read we received eight bytes, and after that ssl_pending
> returns 0. If we continue reading despite having no pending data, ssl_read
> returns the rest of the data.

Are you setting SSL_CTRL_SET_READ_AHEAD? SSL_pending doesn't work if read-ahead 
is set. See the comment in the definition of SSL_pending in ssl_lib.c


Did the client send a TLS record with more than 8 bytes of application data?

SSL_pending returns true if there's more application data to be read from the 
current record. (At least that's my interpretation from a quick glance at the 
source.)

TLS is a record-oriented protocol, but the API is not strictly record-oriented. 
TLS segments outbound application data into "fragments", with one fragment for 
each TLS record. If the application makes a single call to SSL_write with a 
data length that fits in a single fragment, that *should* go out as a single 
TLS record (I believe); but if the application makes multiple calls to 
SSL_write or sends a chunk of data that's bigger than the maximum fragment size 
for the connection, then the partitioning of application data into records is 
harder to predict.

If you want to know whether there might be additional records waiting, query 
the socket directly with an API such as select or poll. (If the records haven't 
made it into the socket's receive buffer yet, you're out of luck; there's no 
way for the application to tell that more data might arrive some time in the 
future.)

This isn't an issue for HTTP because HTTP is a self-delimiting protocol. The 
application can continue to issue receives, parsing what it's received so far, 
until it knows that it has the entire message. SSL_pending isn't particularly 
useful for such a protocol, unless it's doing non-blocking I/O - in which case 
the typical pattern is to loop calling SSL_read as long as either SSL_pending 
is true or the socket is readable. (Or until OpenSSL returns SSL_WANT_WRITE, in 
which case you have to wait until the socket is writable instead, because 
you're renegotiating.)

That's all off the top of my head, so I may have gone wrong there somewhere - 
in which case no doubt someone will correct me shortly.

Michael Wojcik 
Distinguished Engineer, Micro Focus 


-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ssl_pending returns 0 despite having data to read

[Salz, Rich]

> During first ssl_read we received eight bytes, and after that ssl_pending
> returns 0. If we continue reading despite having no pending data, ssl_read
> returns the rest of the data.
> Could you please suggest what is wrong here.

Pending is an indication that there is unread data *on the local host.*  It has 
no idea of what the network is doing, buffering or delaying, and so on.

You'll have to look at adding bytecounts or other "framing" techniques to your 
protocol to know when enough data has been read.
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ssl_pending returns 0 despite having data to read

[Ryan Murray]

Situation maybe a security issue

Ryan Murray

On Jan 11, 2017 4:14 PM, "Ryan Murray"  wrote:

> Could you give me a hand on a issue I've seem to of picked up with my
> device . You and the colleagues if possible. My SamsungGalaxy s2 tablet not
> responding.  Power button and display goes black and does not turn on for a
> period of time.  I believe the programs running in background or in a
> rooted format has been making the device malfunction. Is there a remote
> interface we could link up and establish what the heck is happening.  Lol
> Your truly
> Ryan
>
> Ryan Murray
>
> On Jan 11, 2017 4:08 PM, "Nadia Lapkovskaya"  wrote:
>
>> Hi,
>>
>> We are using openssl-1.0.2j. Noticed, that for http protocol everything
>> is working fine, but when we are using our own binary protocol ssl_pending
>> returns 0 all the time. We are using blocking socket. Tried with
>> SSL_CTX_set_read_ahead set and unset.
>>
>> Out test server sends back any info received from the client.
>>
>> Test code looks like this:
>> bool write(const uint64_t* data, int count)
>> {
>>   int rc = SSL_write(_ssl, data, count * sizeof(uint64_t));
>>   return rc > 0 ? true : false;
>> }
>>
>> bool read(uint64_t* data, int count)
>> {
>>   do {
>>   int rc = SSL_read(_ssl, data, count * sizeof(uint64_t));
>>   if (rc <= 0) {
>>   int err = SSL_get_error(_ssl, rc);
>>   std::string errs = ERR_error_string(err, nullptr);
>>   return false;
>>   }
>>   } while (SSL_pending(_ssl));
>>   return true;
>> }
>>
>> During first ssl_read we received eight bytes, and after that ssl_pending
>> returns 0. If we continue reading despite having no pending data, ssl_read
>> returns the rest of the data.
>> Could you please suggest what is wrong here.
>>
>>
>> Best regards,
>> Nadia.
>>
>> --
>> openssl-users mailing list
>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
>>
>
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ssl_pending returns 0 despite having data to read

[Ryan Murray]

Could you give me a hand on a issue I've seem to of picked up with my
device . You and the colleagues if possible. My SamsungGalaxy s2 tablet not
responding.  Power button and display goes black and does not turn on for a
period of time.  I believe the programs running in background or in a
rooted format has been making the device malfunction. Is there a remote
interface we could link up and establish what the heck is happening.  Lol
Your truly
Ryan

Ryan Murray

On Jan 11, 2017 4:08 PM, "Nadia Lapkovskaya"  wrote:

> Hi,
>
> We are using openssl-1.0.2j. Noticed, that for http protocol everything is
> working fine, but when we are using our own binary protocol ssl_pending
> returns 0 all the time. We are using blocking socket. Tried with
> SSL_CTX_set_read_ahead set and unset.
>
> Out test server sends back any info received from the client.
>
> Test code looks like this:
> bool write(const uint64_t* data, int count)
> {
>   int rc = SSL_write(_ssl, data, count * sizeof(uint64_t));
>   return rc > 0 ? true : false;
> }
>
> bool read(uint64_t* data, int count)
> {
>   do {
>   int rc = SSL_read(_ssl, data, count * sizeof(uint64_t));
>   if (rc <= 0) {
>   int err = SSL_get_error(_ssl, rc);
>   std::string errs = ERR_error_string(err, nullptr);
>   return false;
>   }
>   } while (SSL_pending(_ssl));
>   return true;
> }
>
> During first ssl_read we received eight bytes, and after that ssl_pending
> returns 0. If we continue reading despite having no pending data, ssl_read
> returns the rest of the data.
> Could you please suggest what is wrong here.
>
>
> Best regards,
> Nadia.
>
> --
> openssl-users mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
>
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

[openssl-users] ssl_pending returns 0 despite having data to read

[Nadia Lapkovskaya]

Hi,

We are using openssl-1.0.2j. Noticed, that for http protocol everything is 
working fine, but when we are using our own binary protocol ssl_pending returns 
0 all the time. We are using blocking socket. Tried with SSL_CTX_set_read_ahead 
set and unset.

Out test server sends back any info received from the client.

Test code looks like this:
bool write(const uint64_t* data, int count)
{
  int rc = SSL_write(_ssl, data, count * sizeof(uint64_t));
  return rc > 0 ? true : false;
}

bool read(uint64_t* data, int count)
{
  do {
  int rc = SSL_read(_ssl, data, count * sizeof(uint64_t));
  if (rc <= 0) {
  int err = SSL_get_error(_ssl, rc);
  std::string errs = ERR_error_string(err, nullptr);
  return false;
  }
  } while (SSL_pending(_ssl));
  return true;
}

During first ssl_read we received eight bytes, and after that ssl_pending 
returns 0. If we continue reading despite having no pending data, ssl_read 
returns the rest of the data. 
Could you please suggest what is wrong here.


Best regards,
Nadia.

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Viktor Dukhovni]

On Wed, Jan 11, 2017 at 05:27:47PM +, Michael Wojcik wrote:

> Unfortunately writing proper C is a rare skill - relatively few C
> programmers have ever even read the language specification - and much C
> code is saddled with lots of ancient technical debt. Also, of course, it
> often doesn't make economic sense to accommodate rare implementations.

In the case of OpenSSL, the issue was well understood, and upon
consideration a decision was made to not support platforms where
the memory representation of NULL is not zero.  A test was written
to make sure that non-conformant platforms are detected.

By way of contrast, the Postfix project supports non-zero NULLs,
and explicitly initializes pointer-valued member fields in structures.

Neither project simply ignores the issue.

-- 
Viktor.
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Salz, Rich]

> The representation in memory of a null pointer need not be all-bits-zero.
> (The representation in memory of an integer constant with the value zero
> can either be all-bits-zero or, in the unlikely case of sign-magnitude 
> integers,
> a sign bit of 1 followed by all-other-bits-zero.)

And, again, openssl will not work on those platforms and we have a test to 
catch it.

> Doing it properly, incidentally, looks like this:

Look at apps/rehash.c :)

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf
> Of Jeffrey Walton
> Sent: Wednesday, January 11, 2017 11:19
> To: OpenSSL Users
> Subject: Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details
> 
> > Could someone from the OpenSSL team please explain the rationale for
> this
> > decision? What is the problem with using assignments with 0 or NULL to
> > initialize pointers?
> 
> I'm not from the team, so take it for what its worth...
> 
> On some systems, NULL is _not_ 0. NULL can be anywhere in memory the
> architecture wants it to be. It can be in a high page in memory, too.

This comment is misleading (as was Rich Salz's reply). Jeffrey's question, as 
phrased, is correct.

The definition of the NULL macro is mandated by the C standard. It is either an 
integer constant that evaluates to 0, or such a constant cast to void*.

The representation in memory of a null pointer need not be all-bits-zero. (The 
representation in memory of an integer constant with the value zero can either 
be all-bits-zero or, in the unlikely case of sign-magnitude integers, a sign 
bit of 1 followed by all-other-bits-zero.)

In other words, *in C source code* a null pointer is *always* created by 
assigning the NULL macro, or a literal 0, or such a thing cast to an 
appropriate pointer type, to a pointer variable. Even if the representation of 
a null pointer in memory is not all-bits-zero. Similarly, comparing a null 
pointer to a literal 0 evaluates as true, regardless of the representation of 
null pointers. That's required by the C standard; if it doesn't work, you're 
using a language which is almost but not quite entirely unlike C.

However: Operations such as memset(object_pointer, 0, size) may NOT create null 
pointers, because memset simply sets the underlying bits without any knowledge 
of types or representations.

And that's why many programs don't work on implementations where the null 
pointer representation is not all-bits-zero: because they use shortcuts like 
memset and calloc to "clear" pointers (generally as part of structs containing 
pointer fields), rather than doing it properly.

Doing it properly, incidentally, looks like this:

   static const struct foo foo0;/* implicitly equivalent to ... foo0 = 
{0} */
   ...
   struct foo *bar = malloc(sizeof *bar);
   *bar = foo0;

Unfortunately writing proper C is a rare skill - relatively few C programmers 
have ever even read the language specification - and much C code is saddled 
with lots of ancient technical debt. Also, of course, it often doesn't make 
economic sense to accommodate rare implementations. How many C programs work 
correctly on implementations where CHAR_BIT > 8?

Michael Wojcik 
Distinguished Engineer, Micro Focus 



-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details - NULL vs zeros

[Salz, Rich]

> I suspect that it was a shortcut, where they used memset() on an entire
> structure, and it hopefully set pointers to NULL.
> 
> What I pointed out is that if NULL is not all zeros, this breaks.

And OpenSSL does not work on those platforms.  It is part of the test suite to 
check for this.  See test/sanitytest.c

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Jakob Bohm]

On 11/01/2017 16:32, Stephan Mühlstrasser wrote:

Am 03.01.17 um 21:26 schrieb Viktor Dukhovni:



On Jan 3, 2017, at 2:55 PM, Ken Goldman  wrote:

1 - Is this a bit of a bug?

ECDSA_SIG_free() frees the r and s BIGNUMs before is frees the 
structure itself.  However, ECDSA_SIG_new() doesn't set r and s to

NULL.  It calls zalloc, which sets them to 0x00 bytes.

OK, in most platforms, the NULL pointer is an all 0x00 bytes value, 
but it's not guaranteed by the C standard.


E.g., http://c-faq.com/null/confusion4.html


OpenSSL does not support platforms where the memory representation of 
the
NULL pointer contains non-zero bytes. IIRC there are even tests for 
this.


Could someone from the OpenSSL team please explain the rationale for 
this decision? What is the problem with using assignments with 0 or 
NULL to initialize pointers?



I am not from the OpenSSL team either, but the probable thinking is
like this:

The C (and C++) standard allows a number of things that are very rare
in practice, while providing very few guarantees about important
aspects of semantics.  Therefore most major cross-platform projects
will usually take a number of common-sense aspects that are already
common among the desired platforms and elevate them into "project-
specific" requirements, foregoing the option to port the code to
platforms that differ in those aspects.  This provides a more robust
and usable "dialect" of the C/C++ language, allowing code to not do
things that are clumsy, slow, error-prone or otherwise difficult.

For the specific requirement of NULL pointers being encoded as an
all-0 bit pattern (and the similar requirement for 0 integral values),
this allows the use of zero-initializing memory allocators/handlers to
forego the need to explicitly initialize NULL (and 0) field and array
element values.

Note that C++ has a similar, but weaker, requirements that the source
code literal 0 is a valid syntax for null pointer constants and that
many (but not all) uninitialized pointer and numeric fields should be
implicitly initialized to NULL / 0 by compiler generated code, which
doesn't cover the case of memory blocks that don't get to run the
(implicit) C++ constructor.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Jeffrey Walton]

> Could someone from the OpenSSL team please explain the rationale for this
> decision? What is the problem with using assignments with 0 or NULL to
> initialize pointers?

I'm not from the team, so take it for what its worth...

On some systems, NULL is _not_ 0. NULL can be anywhere in memory the
architecture wants it to be. It can be in a high page in memory, too.
One of my instructors in college was telling me about a system he
worked on where NULL was an address in the last page in memory, so it
took a value like `0x`.

Jeff
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Salz, Rich]

> > OpenSSL does not support platforms where the memory representation of
> > the NULL pointer contains non-zero bytes. IIRC there are even tests for
> this.
> 
> Could someone from the OpenSSL team please explain the rationale for this
> decision? What is the problem with using assignments with 0 or NULL to
> initialize pointers?

I think you are confused.

There is no problem with what you posted.  The issue is that if NULL != 0, 
OpenSSL won't necessarily work.

See test/sanitytest.c for what we check.
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] ECDSA_SIG_new and ECDSA_SIG_free details

[Stephan Mühlstrasser]

Am 03.01.17 um 21:26 schrieb Viktor Dukhovni:



On Jan 3, 2017, at 2:55 PM, Ken Goldman  wrote:

1 - Is this a bit of a bug?

ECDSA_SIG_free() frees the r and s BIGNUMs before is frees the structure 
itself.  However, ECDSA_SIG_new() doesn't set r and s to
NULL.  It calls zalloc, which sets them to 0x00 bytes.

OK, in most platforms, the NULL pointer is an all 0x00 bytes value, but it's 
not guaranteed by the C standard.

E.g., http://c-faq.com/null/confusion4.html


OpenSSL does not support platforms where the memory representation of the
NULL pointer contains non-zero bytes. IIRC there are even tests for this.


Could someone from the OpenSSL team please explain the rationale for 
this decision? What is the problem with using assignments with 0 or NULL 
to initialize pointers?


--
Stephan
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users