Build failed in Jenkins: FreeBSD_HEAD-tests2 #1035

[jenkins-admin]

See 

--
[...truncated 2446 lines...]
lib/libnv/dnv_tests:dnvlist_take_string__present  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_add_binary__single_insert  ->  passed  [0.019s]
lib/libnv/nv_tests:nvlist_add_bool__single_insert  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_add_null__single_insert  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_add_number__single_insert  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_add_nvlist__child_with_error  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_add_nvlist__single_insert  ->  passed  [0.162s]
lib/libnv/nv_tests:nvlist_add_string__single_insert  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_clone__empty_nvlist  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_clone__error_nvlist  ->  passed  [0.019s]
lib/libnv/nv_tests:nvlist_clone__nested_nvlist  ->  passed  [0.184s]
lib/libnv/nv_tests:nvlist_clone__nonempty_nvlist  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_create__is_empty  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_free__single_binary  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_free__single_bool  ->  passed  [0.079s]
lib/libnv/nv_tests:nvlist_free__single_null  ->  passed  [0.020s]
lib/libnv/nv_tests:nvlist_free__single_number  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_free__single_nvlist  ->  passed  [0.045s]
lib/libnv/nv_tests:nvlist_free__single_string  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_free_binary__single_binary  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_free_bool__single_bool  ->  passed  [0.021s]
lib/libnv/nv_tests:nvlist_free_null__single_null  ->  passed  [0.021s]
lib/libnv/nv_tests:nvlist_free_number__single_number  ->  passed  [0.020s]
lib/libnv/nv_tests:nvlist_free_nvlist__single_nvlist  ->  passed  [0.025s]
lib/libnv/nv_tests:nvlist_free_string__single_string  ->  passed  [0.026s]
lib/libnv/nv_tests:nvlist_move_binary__single_insert  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_move_nvlist__child_with_error  ->  passed  [0.020s]
lib/libnv/nv_tests:nvlist_move_nvlist__null_child  ->  passed  [0.051s]
lib/libnv/nv_tests:nvlist_move_nvlist__single_insert  ->  passed  [0.189s]
lib/libnv/nv_tests:nvlist_move_string__single_insert  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_pack__empty_nvlist  ->  passed  [0.019s]
lib/libnv/nv_tests:nvlist_pack__error_nvlist  ->  passed  [0.019s]
lib/libnv/nv_tests:nvlist_pack__multiple_values  ->  passed  [0.041s]
lib/libnv/nv_tests:nvlist_take_binary__other_keys_unchanged  ->  passed  
[0.019s]
lib/libnv/nv_tests:nvlist_take_binary__single_remove  ->  passed  [0.061s]
lib/libnv/nv_tests:nvlist_take_bool__other_keys_unchanged  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_take_bool__single_remove  ->  passed  [0.018s]
lib/libnv/nv_tests:nvlist_take_number__other_keys_unchanged  ->  passed  
[0.018s]
lib/libnv/nv_tests:nvlist_take_number__single_remove  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_take_nvlist__other_keys_unchanged  ->  passed  
[0.017s]
lib/libnv/nv_tests:nvlist_take_nvlist__single_remove  ->  passed  [0.020s]
lib/libnv/nv_tests:nvlist_take_string__other_keys_unchanged  ->  passed  
[0.018s]
lib/libnv/nv_tests:nvlist_take_string__single_remove  ->  passed  [0.017s]
lib/libnv/nv_tests:nvlist_unpack__duplicate_key  ->  passed  [0.021s]
lib/libnv/nv_tests:nvlist_unpack__flags_nvlist  ->  passed  [0.018s]
lib/libnv/nvlist_add_test:main  ->  passed  [0.021s]
lib/libnv/nvlist_exists_test:main  ->  passed  [0.025s]
lib/libnv/nvlist_free_test:main  ->  passed  [0.107s]
lib/libnv/nvlist_get_test:main  ->  passed  [0.023s]
lib/libnv/nvlist_move_test:main  ->  passed  [0.024s]
lib/libnv/nvlist_send_recv_test:main  ->  passed  [0.059s]
lib/libpam/t_openpam_ctype:main  ->  passed  [0.016s]
lib/libpam/t_openpam_readlinev:main  ->  passed  [0.090s]
lib/libpam/t_openpam_readword:main  ->  May 15 04:50:35  t_openpam_readword: in 
openpam_readword(): unexpected end of file
passed  [0.124s]
lib/libthr/barrier_test:barrier  ->  passed  [10.257s]
lib/libthr/cond_test:bogus_timedwaits  ->  passed  [0.017s]
lib/libthr/cond_test:broadcast  ->  passed  [2.798s]
lib/libthr/cond_test:cond_timedwait_race  ->  passed  [4.698s]
lib/libthr/cond_test:destroy_after_cancel  ->  passed  [0.018s]
lib/libthr/cond_test:signal_before_unlock  ->  passed  [2.227s]
lib/libthr/cond_test:signal_before_unlock_static_init  ->  passed  [2.115s]
lib/libthr/cond_test:signal_delay_wait  ->  passed  [2.027s]
lib/libthr/cond_test:signal_wait_race  ->  passed  [5.953s]
lib/libthr/condwait_test:cond_wait_mono  ->  passed  [2.120s]
lib/libthr/condwait_test:cond_wait_real  ->  passed  [2.287s]
lib/libthr/detach_test:pthread_detach  ->  May 15 04:50:35  last message 
repeated 2 times
passed  [4.083s]
lib/libthr/equal_test:pthread_equal  ->  passed  [0.165s]
lib/libthr/fork_test:fork  ->  passed  [5.082s]
lib/libthr/fpu_test:fpu  ->  passed  [0.022s]
lib/libthr/join_test:pthread_join  ->  passed  [0.034s]
lib/li

Re: Increase BUFSIZ to 8192

[Slawa Olhovchenkov]

On Thu, May 14, 2015 at 08:53:05AM -0600, Ian Lepore wrote:

> At least I'm inclined to ponder it.  Apparently nobody else is.  People
> running servers with more GB of ram than grains of sand on the beach
> won't care about things like 64k buffers used by /bin/sh to read a line
> of text, and all the world is big servers now, right?

I have setups with servering tens of gigabits pers second from one
server. Default send_lowat (SO_SNDLOWAT) is 2048. Settnig to 128K
increase load. Setting to 16k slightly reduce.
Not so simple.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <5554b8d6.1010...@mu.org>, Alfred Perlstein writes:

>Shouldn't most of these be using st.st_blksize ?

We had a long discussion about that back when GEOM was young and the
conclusionis that st_blksize doesn't tell you anything useful
and generally does the wrong thing, in particular on non-native
filesystems like msdosfs and cd9660.

But the world is more complex than even that.

For instance on a RAID-5 volume, you want to write stripe-width
chunks, properly aligned, no matter what the st_blksize might be
in your filesystem.  Unless your filesystem is guaranteed to lay
out sequentially, you would have to ask before each write.

Other filesystems may have opinions about read-sizes (ie: NFS).

The only sane way to do this properly would be to ask each
individual file with fcntl(2) for preferred read or write
sizes.

You could then have embedded system mount filesystems with
-o iosize=min
and servers instead use
-o iosize=fastest

But for most practical purposes, having a sane constant BUFSIZ is
just fine.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Jenkins build is back to stable : FreeBSD_HEAD-tests2 #1033

[jenkins-admin]

See 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <1431615185.1221.57.ca...@freebsd.org>, Ian Lepore writes:

>I think we've got differing interpretations of what BUFSIZ is for.
>
>IMO, the one correct use of BUFSIZ outside of libc is "if you are going
>to call setbuf() the buffer you pass must be BUFSIZ bytes long."
>
>Over the years, it seems that many people have somehow gotten the
>impression that the intent was "BUFSIZ is the right/ideal/whatever size
>to allocate general purpose IO buffers in any program" 

I don't know when you started, but when I started, on sys-III and
v7 in the mid 1980ies, that was exactly what people told you:
"Do disk-I/O in BUFSIZ units".

I did a quick sampling of src and that seems to be exactly how it is
being used in most of the cases I looked at, including libmd where
I put it there on exactly that reason back in 1994 (5?)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Alfred Perlstein]

On 5/14/15 2:23 AM, Garrett Cooper wrote:

On May 14, 2015, at 1:06, Poul-Henning Kamp  wrote:



In message <20150514075316.gy37...@funkthat.com>, John-Mark Gurney writes:

Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:


In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:


Since you apprently missed my original reply, I said that we shouldn't
abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...

Say what ?

BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
a file into an algorithm.

In fact, posix-2008 references LINE_MAX because:

MDXFileChunk() does not read lines, it reads an entire file.

Being pedantic, technically it’s a portion of a file, which can be the whole thing, 
and it reads it in “sizeof(buffer)” chunks (of which buffer is “hardcoded" to 
BUFSIZ right now).
Cheers!

Shouldn't most of these be using st.st_blksize ?

I recall being part of the move to get rid of PAGE_SIZE, perhaps many 
places should be rid of BUFSIZE as well and BUFSIZE should be something 
we query the system for.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Ian Lepore]

On Thu, 2015-05-14 at 07:42 +, Poul-Henning Kamp wrote:
> 
> In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:
> 
> >Since you apprently missed my original reply, I said that we shouldn't
> >abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...
> 
> Say what ?
> 
> BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
> a file into an algorithm.
> 
> If in stead of open(2), fopen(3) had been used, the exact same thing
> would happen, but using malloc space rather than stack space.
> 
> 

I think we've got differing interpretations of what BUFSIZ is for.

IMO, the one correct use of BUFSIZ outside of libc is "if you are going
to call setbuf() the buffer you pass must be BUFSIZ bytes long."

Over the years, it seems that many people have somehow gotten the
impression that the intent was "BUFSIZ is the right/ideal/whatever size
to allocate general purpose IO buffers in any program" and I don't
believe that was ever the intent, or was ever correct.  All such usage
is erronious and must inevitably lead to the situation we've got now:
it's so widely misused that it can't be changed in the context of its
original purpose without pondering what the wider implications of the
change might be.

At least I'm inclined to ponder it.  Apparently nobody else is.  People
running servers with more GB of ram than grains of sand on the beach
won't care about things like 64k buffers used by /bin/sh to read a line
of text, and all the world is big servers now, right?

-- Ian


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <72720ea2-c251-40b9-9ec0-702c07d5e...@gmail.com>, Garrett Cooper 
writes:

>Until performance has been characterized on 32-bit vs 
>64-bit architectures, blanket changing a value doesn't make sense.

First time I saw benchmarks which showed improved performance
from a larger BUFSIZe was around 1998...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] Replace gnu groff in base by heirloom doctools

[Baptiste Daroussin]

On Thu, May 14, 2015 at 02:31:55AM -0700, Garrett Cooper wrote:
> On May 13, 2015, at 17:02, Baptiste Daroussin  wrote:
> 
> > Hi,
> > 
> > I plan to work in replacing GNU groff for FreeBSD 11.0 in base by heirloom
> > doctools.
> 
> …
> 
> Hi Bapt,
>   Do you have a list of items that require doctools [if groff isn’t 
> present]?
> Thanks!

They are all only build time and all locate in share/docs basically everything
using bsd.doc.mk

man(1) also falls back on gnu groff when not able to read a manpage with mandoc
and I will switch that to use heirloom (and if not present try to use gnu groff
from ports)

Best regards,
Bapt


pgp5F8KDOyC8C.pgp
Description: PGP signature

Re: [RFC] Replace gnu groff in base by heirloom doctools

[Garrett Cooper]

On May 13, 2015, at 17:02, Baptiste Daroussin  wrote:

> Hi,
> 
> I plan to work in replacing GNU groff for FreeBSD 11.0 in base by heirloom
> doctools.

…

Hi Bapt,
Do you have a list of items that require doctools [if groff isn’t 
present]?
Thanks!
-NGie


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Increase BUFSIZ to 8192

[Garrett Cooper]

On May 14, 2015, at 1:01, Poul-Henning Kamp  wrote:

> 
> In message <1431542835.1221.30.ca...@freebsd.org>, Ian Lepore writes:
>> On Wed, 2015-05-13 at 11:13 -0700, John-Mark Gurney wrote:
> 
>> As I've already pointed out, BUFSIZ appears in the
>> base code over 2000 times.  Where is the analysis of the impact an 8x
>> change is going to have on all those uses?
> 
> Not to pick on Ian in particular, but I'm going to call bike-shed
> on this discussion now.
> 
> Please just make it 4K on 32bit archs and 16K on 64 bit archs, and
> get on with your lives.
> 
> If experience in -current (that's why developers run current, right ?!)
> documents that this was the wrong decision, we can revisit it.
> 
> Until then:  Shut up and code.

Baptiste’s recommendation was related to md5 performance, so it might be that 
(as you pointed out with MDXFileChunk), things might be less performant in the 
application than they could be — but that’s an application bug (only helped by 
scaling issues with FreeBSD, potentially). Until performance has been 
characterized on 32-bit vs 64-bit architectures, blanket changing a value 
doesn’t make sense.

I think that changing buffers sized at BUFSIZ for md5/libmd5 probably makes a 
lot more sense as that change is isolated and the end result could be easily 
micro benchmarked. If/when we have an overall characterization we can look at 
increasing the value across the board.

Thanks!
-NGie


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [RFC] Replace gnu groff in base by heirloom doctools

[David Chisnall]

On 14 May 2015, at 10:24, Baptiste Daroussin  wrote:
> 
> On Thu, May 14, 2015 at 10:13:18AM +0100, David Chisnall wrote:
>> On 14 May 2015, at 09:59, Baptiste Daroussin  wrote:
>>> 
>>> On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
 On 14 May 2015, at 01:02, Baptiste Daroussin  wrote:
> 
> - it is partially CDDL partially BSD license.
 
 We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
 build the CDDL parts, what will break?
 
>>> Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX 
>>> aka
>>> you won't have the main part of the toolchain (aka troff/nroff)
>> 
>> But man pages will still work via mandoc?  WITHOUT_GNU is known not to work 
>> (though we’re trying to address that with 11).  WITHOUT_CDDL is generally 
>> expected to give a working system currently.
> 
> Yes since I switched the default manpage renderer to mandoc(1) :) (meaning 
> that
> WITHOUT_GNU is also not blocking manpage rendering)

Sounds good to me.  Probably warrants a release notes entry specifically 
documenting that change, but otherwise sounds like a big improvement!

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] Replace gnu groff in base by heirloom doctools

[Baptiste Daroussin]

On Thu, May 14, 2015 at 10:13:18AM +0100, David Chisnall wrote:
> On 14 May 2015, at 09:59, Baptiste Daroussin  wrote:
> > 
> > On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
> >> On 14 May 2015, at 01:02, Baptiste Daroussin  wrote:
> >>> 
> >>> - it is partially CDDL partially BSD license.
> >> 
> >> We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
> >> build the CDDL parts, what will break?
> >> 
> > Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX 
> > aka
> > you won't have the main part of the toolchain (aka troff/nroff)
> 
> But man pages will still work via mandoc?  WITHOUT_GNU is known not to work 
> (though we’re trying to address that with 11).  WITHOUT_CDDL is generally 
> expected to give a working system currently.

Yes since I switched the default manpage renderer to mandoc(1) :) (meaning that
WITHOUT_GNU is also not blocking manpage rendering)

Best regards,
Bapt


pgp1CQvkz97MO.pgp
Description: PGP signature

Re: Increase BUFSIZ to 8192

[Garrett Cooper]

On May 14, 2015, at 1:06, Poul-Henning Kamp  wrote:

> 
> In message <20150514075316.gy37...@funkthat.com>, John-Mark Gurney writes:
>> Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:
>>> 
>>> In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:
>>> 
 Since you apprently missed my original reply, I said that we shouldn't
 abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...
>>> 
>>> Say what ?
>>> 
>>> BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
>>> a file into an algorithm.
> 
>> In fact, posix-2008 references LINE_MAX because:
> 
> MDXFileChunk() does not read lines, it reads an entire file.

Being pedantic, technically it’s a portion of a file, which can be the whole 
thing, and it reads it in “sizeof(buffer)” chunks (of which buffer is 
“hardcoded" to BUFSIZ right now).
Cheers!


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [RFC] Replace gnu groff in base by heirloom doctools

[David Chisnall]

On 14 May 2015, at 09:59, Baptiste Daroussin  wrote:
> 
> On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
>> On 14 May 2015, at 01:02, Baptiste Daroussin  wrote:
>>> 
>>> - it is partially CDDL partially BSD license.
>> 
>> We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
>> build the CDDL parts, what will break?
>> 
> Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX 
> aka
> you won't have the main part of the toolchain (aka troff/nroff)

But man pages will still work via mandoc?  WITHOUT_GNU is known not to work 
(though we’re trying to address that with 11).  WITHOUT_CDDL is generally 
expected to give a working system currently.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] Replace gnu groff in base by heirloom doctools

[Baptiste Daroussin]

On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
> On 14 May 2015, at 01:02, Baptiste Daroussin  wrote:
> > 
> > - it is partially CDDL partially BSD license.
> 
> We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
> build the CDDL parts, what will break?
> 
Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX aka
you won't have the main part of the toolchain (aka troff/nroff)

Best regards,
Bapt


pgpERse66R4LX.pgp
Description: PGP signature

Re: [RFC] Replace gnu groff in base by heirloom doctools

[David Chisnall]

On 14 May 2015, at 01:02, Baptiste Daroussin  wrote:
> 
> - it is partially CDDL partially BSD license.

We currently have a WITHOUT_CDDL knob that some people use.  If we don’t build 
the CDDL parts, what will break?

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <20150514075316.gy37...@funkthat.com>, John-Mark Gurney writes:
>Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:
>> 
>> In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:
>> 
>> >Since you apprently missed my original reply, I said that we shouldn't
>> >abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...
>> 
>> Say what ?
>> 
>> BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
>> a file into an algorithm.

>In fact, posix-2008 references LINE_MAX because:

MDXFileChunk() does not read lines, it reads an entire file.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <1431542835.1221.30.ca...@freebsd.org>, Ian Lepore writes:
>On Wed, 2015-05-13 at 11:13 -0700, John-Mark Gurney wrote:

>As I've already pointed out, BUFSIZ appears in the
>base code over 2000 times.  Where is the analysis of the impact an 8x
>change is going to have on all those uses?

Not to pick on Ian in particular, but I'm going to call bike-shed
on this discussion now.

Please just make it 4K on 32bit archs and 16K on 64 bit archs, and
get on with your lives.

If experience in -current (that's why developers run current, right ?!)
documents that this was the wrong decision, we can revisit it.

Until then:  Shut up and code.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[John-Mark Gurney]

Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:
> 
> In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:
> 
> >Since you apprently missed my original reply, I said that we shouldn't
> >abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...
> 
> Say what ?
> 
> BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
> a file into an algorithm.

Posix-2008:
BUFSIZ: Size of  buffers.  This shall expand to a positive value.

C99:
BUFSIZ
which expands to an integer constant expression that is the size of
the buffer used by the setbuf function;

In fact, posix-2008 references LINE_MAX because:
Frequently, utility writers selected the UNIX system constant BUFSIZ to
allocate these buffers; therefore, some utilities were limited to 512
bytes for I/O lines, while others achieved 4 096 bytes or greater.

BUFSIZ was already recognized as to small to hold a single line, yet
you're saying it's perfectly fine to use as a buffer for binary data?

> If in stead of open(2), fopen(3) had been used, the exact same thing
> would happen, but using malloc space rather than stack space.

Plus extra overhead.. :)

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[Poul-Henning Kamp]

In message <20150514072155.gt37...@funkthat.com>, John-Mark Gurney writes:

>Since you apprently missed my original reply, I said that we shouldn't
>abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...

Say what ?

BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
a file into an algorithm.

If in stead of open(2), fopen(3) had been used, the exact same thing
would happen, but using malloc space rather than stack space.


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Jenkins build became unstable: FreeBSD_HEAD-tests2 #1032

[jenkins-admin]

See 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[John-Mark Gurney]

David Chisnall wrote this message on Wed, May 13, 2015 at 09:27 +0100:
> On 13 May 2015, at 09:03, John-Mark Gurney  wrote:
> > 
> > Poul-Henning Kamp wrote this message on Tue, May 12, 2015 at 06:31 +:
> >> 
> >> In message <20150512032307.gp37...@funkthat.com>, John-Mark Gurney writes:
> >> 
> >>> Also, you'd probably see even better performance by increasing the
> >>> size to 64k, [...]
> >> 
> >> easy:
> >>8K on 32bit
> >>64k on 64bit
> > 
> > Sounds good to me...  Just for people who care... I did a quick set of
> > benchmarks on sha256.. This is using my preliminary patch to use sse4
> > optimized sha256...  But this should be the same for others...
> > 
> > The numbers in ministat output are the time in seconds it takes my
> > 3.4GHz AMD A10-5700 APU running HEAD to process a 512MB file, so lower
> > numbers are better..  I've processed them into easier to read format:
> > BUFSIZ: 145MB/sec
> > 8k: 193MB/sec
> > 16k:198MB/sec
> > 64k:202MB/sec
> > 128k:   202MB/sec
> > -t: 211MB/sec
> 
> It looks like most of the benefit is gained at 16KB.  Did you try running the 
> benchmark with something else running at the same time to see if there is any 
> advantage in trashing the caches a bit less (simple case, what happens if you 
> run two instances of the same benchmark at once)?
> 
> I suspect that you???re about right anyway - I recently did some tests while 
> playing with JavaScript FFI generation with a multithreaded process 
> JavaScript environment calling out to OpenSSL to do SHA calculations and 
> having each of 8 threads reading in 128KB chunks gave the fastest performance 
> (Core i7, 4 cores + hyperthreading), with only a negligible gain over 64KB.  
> In all cases, the JavaScript implementation was significantly faster than the 
> openssl tool, which used 8KB buffers.

Just in case anyone else wants to know how to run benchmarks
themselves..  Go into /usr/src/lib/libmd, edit mdXhl.c, and change
the occurence of BUFSIZ to what you want to test, say 64*1024, run:
make all && make install

and then you can run programs like sha256 -t, or:
for i in `jot 5 1`; do /usr/bin/time sha256 test.file ; done 2> XXX.times

Where test.file is populated maybe like:
dd if=/dev/urandom of=test.file bs=1m count=512

Then run:
ministat XXX.times YYY.times

to compare multiple results...

Happy benchmarking!

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Increase BUFSIZ to 8192

[John-Mark Gurney]

Ian Lepore wrote this message on Wed, May 13, 2015 at 12:47 -0600:
> On Wed, 2015-05-13 at 11:13 -0700, John-Mark Gurney wrote:
> > Adrian Chadd wrote this message on Wed, May 13, 2015 at 08:34 -0700:
> > > The reason I ask about "why is it faster?" is because for embedded-y
> > > things with low RAM we may not want that to happen due to memory
> > > constraints. However, we may actually want to do some form of
> > > autotuning on some platforms.
> > 
> > If you're already running a program, the difference between 1k and
> > 8k isn't significant... I'll give you 64k can be significant for
> > embedded-y platforms...  But this goes back to the, we need a global
> > knob saying I want low memory usage, and I am willing to pay for it
> > in performance...
> 
> It is NOT just a difference of 1K vs 8K.  It's that much times however
> many BUFSIZ-sized things a program has allocated at once.  It's where
> they are allocated.  As I've already pointed out, BUFSIZ appears in the
> base code over 2000 times.  Where is the analysis of the impact an 8x
> change is going to have on all those uses?

Since you apprently missed my original reply, I said that we shouldn't
abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...

I agree that changing this size to effect all the other files is ill
advised and should not be done...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"