Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Matthew Rezny]

On Wednesday 05 April 2017 19:44:51 Slawa Olhovchenkov wrote:
> On Wed, Apr 05, 2017 at 04:15:41PM +, Alexey Dokuchaev wrote:
> > > I've also tried without WITH_DEBUG= and now. . .
> > > 
> > > # pkg delete llvm40
> > > Checking integrity... done (0 conflicting)
> > > Deinstallation has been requested for the following 1 packages (of 0
> > > packages in the universe):
> > > 
> > > Installed packages to be REMOVED:
> > > llvm40-4.0.0
> > > 
> > > Number of packages to be removed: 1
> > > 
> > > The operation will free 1 GiB.
> > 
> > That 1G looks like a big jump from 259M of llvm39-3.9.1_1.txz to me.
> > I'm surely looking forward modularization of LLVM port; rebuilding it
> > every time becomes a real PITA given that X11 stack requires it. :-(
> 
> What real reason of requiring llvm for X11?
> I am about run time depends:
> 
> # pkg info -r llvm39
> llvm39-3.9.1_4:
> libEGL-13.0.6
> dri-13.0.6,2
> 
> # ldd /usr/local/lib/libXvMCr600.so
> /usr/local/lib/libXvMCr600.so:
> [...]
> libLLVM-3.9.so => /usr/local/llvm39/lib/libLLVM-3.9.so (0x803e0)
> libLTO.so => /usr/local/llvm39/lib/../lib/libLTO.so (0x80820) [...]
> 
> # ls -lh /usr/local/llvm39/lib/libLLVM-3.9.so
> /usr/local/llvm39/lib/../lib/libLTO.so -rwxr-xr-x  1 root  wheel38M Apr
>  2 18:18 /usr/local/llvm39/lib/../lib/libLTO.so -rwxr-xr-x  1 root  wheel  
>  47M Apr  2 18:18 /usr/local/llvm39/lib/libLLVM-3.9.so
> 
> libXvMCr600 realy do run-time llvm interpetation and linker-time
> optimisation?!
> 
> Also, I am don't see any realy dependence libEGL-13.0.6 from llvm.

Yes, Mesa really uses LLVM at runtime for shader compilation/optimization, and 
Xorg depends on Mesa for GLX, etc.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Matthew Rezny]

On Wednesday 05 April 2017 16:15:41 Alexey Dokuchaev wrote:
> On Thu, Mar 30, 2017 at 07:26:43PM +0200, Matthew Rezny wrote:
> > LLVM 3.8 introduced the option to build a shared LLVM library, which is
> > what Mesa needs for use at runtime (for e.g. compiling shaders), separate
> > from linking to it. Previous versions only had one option, if the library
> > was built then all the LLVM binaries were staticly linked to it. [...]
> > 
> > llvm{35,36,37} are statically linked and thus smaller than normal. llvm38
> > switched to dynamic linking, the default, thus the size grew.
> 
> Hmm, I don't quite get it: shouldn't static linking actually increase the
> binaries (and thus the package) size?
> 

I might have reversed static and shared somewhere in the linking explanation, 
or not properly described the situation. Versions prior to 3.8 could either 
build libLLVM as a static library and link all the LLVM binaries to that 
(recommended), or build it as a shared library and link the LLVM binaries to 
that (recommended for development only, but Mesa needs libLLVM.so). LLVM 3.8 
introduced the 3rd option, build the shared library for external use, i.e. 
Mesa, but link the LLVM binaries to the libLLVM*.a bits that were used to 
build the shared library.

llvm37 was built the non-recommended way for the benefit of Mesa, the LLVM 
binaries were linked to the shared library that Mesa used. llvm38 turned 
building/linking of the shared library, so there would be some increase since 
each LLVM binary now had that portion static linked. llvm39 turned on building 
of the shared library but did not enable linking with it so the LLVM binaries 
remain linking that part static, meaning the package grows by the size the 
shared library that is installed but not used by LLVM itself.

Those were changes to a portion, not a complete change between static and 
shared linking for the whole package, so I was somewhat surprised by the size 
difference you noted, but of course I had forgotten that all the parts were 
collapsed into the llvm port. There was a brief period in which llvm39 was 
fully switched to dynamic linking, which made it considerably smaller but 
caused runtime problems (and was also likely to be slower).

> > I assume llvm40 will be a bit bigger, but do not expect to see another
> > jump as you've observed.
> 
> As Mark Millard reports:
> > I've also tried without WITH_DEBUG= and now. . .
> > 
> > # pkg delete llvm40
> > Checking integrity... done (0 conflicting)
> > Deinstallation has been requested for the following 1 packages (of 0
> > packages in the universe):
> > 
> > Installed packages to be REMOVED:
> > llvm40-4.0.0
> > 
> > Number of packages to be removed: 1
> > 
> > The operation will free 1 GiB.
> 
> That 1G looks like a big jump from 259M of llvm39-3.9.1_1.txz to me.
> I'm surely looking forward modularization of LLVM port; rebuilding it
> every time becomes a real PITA given that X11 stack requires it. :-(
> 
> ./danfe

I have both llvm39 and llvm40 installed here (amd64) with all options enabled 
and without any WITH_DEBUG. According to pkg, the flat (installed) size of 
llvm39 is 1.10GB and llvm40 is 1.31GB, so not a huge difference (<20%) but 
still steady growth. The best solution to cut rebuild time for LLVM is ccache.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Matthew Rezny]

On Thursday 30 March 2017 17:06:48 Alexey Dokuchaev wrote:
> On Mon, Mar 27, 2017 at 11:41:40AM +0200, Dimitry Andric wrote:
> > On 26 Mar 2017, at 23:36, Mark Millard  wrote:
> > > ...
> > > Also interesting was:
> > > 
> > > Installed packages to be REMOVED:
> > >   llvm40-4.0.0.r4
> > > 
> > > Number of packages to be removed: 1
> > > 
> > > The operation will free 49 GiB.
> > 
> > Yes, this is big.  But there is no real need to build the llvm ports
> > with debug information, unless you want to hack on llvm itself.
> 
> Cc'ing jmd@ and rezny@.
> 
> I've been watching increasing size of our LLVM packages with increasing
> worry.  This is from my tinderbox cache:
> 
>   $ % env LANG=C ls -lh llvm3*
>   -rw-r--r--  1 root  wheel17M Jan 29  2016 llvm35-3.5.2_1.txz
>   -rw-r--r--  1 root  wheel18M Mar  7  2016 llvm36-3.6.2_2.txz
>   -rw-r--r--  1 root  wheel27M Feb 28 01:05 llvm37-3.7.1_4.txz
>   -rw-r--r--  1 root  wheel   207M Jan 19 18:20 llvm38-3.8.1_5.txz
>   -rw-r--r--  1 root  wheel   244M Mar 23 16:42 llvm39-3.9.1_2.txz
> 
> Dimitry, do you know what had causes such a huge bump in 37 -> 38?
> 
> They take lots of time to build and package.  And given that llvm
> is indirect dependency of any X11-related port, it pessimises their
> build times as well (devel/libclc now requires devel/llvm40 after
> r437268).
> 
> With 49 GiB llvm40, I guess I won't be able to build-test post as my
> hardware would just not be capable enough.
> 
> ./danfe

LLVM 3.8 introduced the option to build a shared LLVM library, which is what 
Mesa needs for use at runtime (for e.g. compiling shaders), separate from 
linking to it. Previous versions only had one option, if the library was built 
then all the LLVM binaries were staticly linked to it.

LLVM devs state that static linking the LLVM binaries is only for developer 
use, users should not do that. Mesa's need was causing distributions to ship 
static linked LLVM binaries against that advice. So, they added a pair of 
switches so that we can separately control whether that library is built 
(required for Mesa) and used to link LLVM binaries (not desired).

llvm{35,36,37} are statically linked and thus smaller than normal. llvm38 
switched to dynamic linking, the default, thus the size grew. llvm39 added the 
library Mesa needs (we didn't turn on the option in llvm38 since Mesa jumped 
from 37 to 39), so it grew a little more. I assume llvm40 will be a bit 
bigger, but do not expect to see another jump as you've observed.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Feature Proposal: Transparent upgrade of crypt() algorithms

[Matthew Rezny]

  Password expiry is an orthogonal issue and should be up to administrator
 
 policy.
 
 Yes, but if you are moving to a different algorithm to improve security, not
 coupling it with an eventual expiration of non-migrated accounts gives a
 false sense of security.  Any admin worth his/her salt is going to want the
 option of enforcing that sort of policy along with the transparent update. 
 They should really be implemented together is all.

Account expiration and password expiration are already present. There is 
absolutely no reason that password algorithm upgrade should be tied in any way 
to expiration. A transparent algorithm upgrade as proposed is *far* better 
than the forced password change method that is commonly employed. If the 
administrator wants to force all accounts to migrate by a set deadline, we 
already have the expiration facilities in place to accomplish that. Expiring 
accounts that have not been used in a long time, regardless of algorithm 
changes, should be part of general housekeeping and may be covered by existing 
policy. Password expiration serves no purpose, EVER. Password expiration 
encourages users to choose bad passwords because they are throwaway items.

Bruce states it well enough I need not elaborate further
https://www.schneier.com/blog/archives/2010/11/changing_passwo.html

Anyone who fails to understand the above should NOT be an administrator.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: BSD XXI Manifesto [mDNS proposal]

[Matthew Rezny]

 On 21 February 2014 20:59, Allan Jude freebsd at allanjude.com wrote:
  I can see the remote controlled installer being especially useful for
  'appliance' type devices, like FreeNAS, pfSense, FUDO, etc.
  
  How would your phone find the address of the machine once it boots off
  the USB, so you could access the web server?
 
 what apple does.
 
 
 
 -a

The solution used by Apple is mDNS and it does work rather well. When 
installing in unstructured environments, I find it quite handy to install 
Apple's mDNSresponder as part of the basic FreeBSD configuration for no reason 
other than the ability to ssh [hostname] from any box to access any other 
box on the network. For completely ad-hoc networks, IPv4LL is also useful in 
conjunction with mDNS.

I think it is time that FreeBSD gains support in base for these ZeroConf 
technologies. Thus, I propose a simple plan to do so.

IPv4LL is supported by dhcpcd in NetBSD. We could use this code in our 
dhclient, or we could simply import their dhcpcd either as replacement for or 
alternate to our dhclient. The choice of approach I leave open to those more 
familiar with our dhclient. The import part is the configuration.

By default, the DHCP client should fallback to IPv4LL if no DHCP server is 
found. This should be the behavior for any interface configured for DHCP i.e. 
ifconfig_[ethif]=DHCP. There also must be a way to disable IPv4LL to enforce 
use of DHCP in environments where it should be present, for which I propose 
the notation ifconfig_[ethif]=DHCP-NOLL. If IPv4LL is in active use, the DHCP 
client should continue to periodically look for a DHCP server and obtain a 
lease without manual user intervention (which is unfortunately required on 
both OS X and Windows, leading to sub-optimal experience in cases of temporary 
unavailability of the DHCP server). When obtaining a lease later, the IPv4LL 
assigned address should still remain as an alias to prevent closure of active 
connections when DHCP becomes available.

Apple's mDNSresponder is currently available in ports. Their implementation 
uses the Apache license so I believe we should be able to import this into 
base without any license issues. The more feature-full Avahi, commonly used on 
Linux system, is also in ports but is LGPL so is not a good choice for base 
due to both size and license.

Use of mDNS should be based on the addressing scheme in use. If there are any 
static addresses, we should assume the admin knows how to reach the box, in 
which case they may enable mDNS. If the box is using purely dynamic addressing 
then we should assume the addresses may be unknown and mDNS will be useful, 
perhaps essential, for locating and accessing the machine. The admin should 
always be able to force mDNS either on or off. Therefore I suggest the rc.conf 
variable MDNS_ENABLE with possible values AUTO, YES, NO. MDNS_ENABLE=AUTO 
should be the default and should implement the above logic; iff all interfaces 
mentioned in rc.conf are not using static addresses then equivalent to YES, 
otherwise NO. The YES and NO settings turn it on and off as expected.

The result is a fresh install that is configured for automatic addressing 
should always have some valid network address and be findable by a known name. 
For headless installation, the installer should have a timeout on the network 
configuration, after which all interfaces attempt DHCP with IPv4LL fallback and 
hostname defaults to FreeBSD-Install. For the case of several machines all 
being installed at once, mDNS resolution should be used to check if that name 
is already in use and if so append a -nnn where nnn is a number to make the 
names unique. On first boot after installation, the configured hostname will be 
used if mDNS in enabled according to the above rules.

Comments?

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: BSD XXI Manifesto [mDNS proposal]

[Matthew Rezny]

On Thursday 27 February 2014 14:55:55 Lyndon Nerenberg wrote:
 On Feb 27, 2014, at 12:20 PM, Matthew Rezny matt...@reztek.cz wrote:
  If IPv4LL is in active use, the DHCP
  client should continue to periodically look for a DHCP server and obtain a
  lease without manual user intervention (which is unfortunately required on
  both OS X and Windows, leading to sub-optimal experience in cases of
  temporary unavailability of the DHCP server).
 
 That's not true for Mac OS.  If you have an interface configured to use
 DHCP, and the Mac is unable to renew the lease, it will automatically
 configure a 169.254.x.y address on the interface.  All the while it
 continues its attempts to renew the DHCP lease and, once successful,
 removes the 169.254.x.y address from the interface.
 
 --lyndon

If they corrected that, it was after I abandoned the platform years ago. I 
remember repeatedly pushing the Renew Lease button in the Network section of 
System Settings in frustration, waiting to see the address change.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: base unzip on 10-STABLE/ 11-HEAD unzip: skipping non-regular entry

[Matthew Rezny]

 Hello,
 
 Unfortunately, this is the case with *zips from Dropbox (Download
 as .zip) directory option.
 
 $ /usr/bin/unzip  file.zip
 Archive:  file.zip
 unzip: skipping non-regular entry ''
 unzip: skipping non-regular entry 'A B C D.pdf'
 
 archivers/unzip manages this case though...
 
 $ /usr/local/bin/unzip  file.zip
 Archive:  file.zip
 warning:  stripped absolute path spec from /
 mapname:  conversion of  failed
 inflating: A B C D.pdf
 
 Tested on 10-STABLE, but should be the same on HEAD.

% /usr/bin/unzip sr003-02-24192.zip 
Archive:  sr003-02-24192.zip
unzip: Mac metadata is too large: 2520404  128K bytes

FAIL

% unzip -t sr003-02-24192.zip
Archive:  sr003-02-24192.zip
unzip: Mac metadata is too large: 2520404  128K bytes

really?

% /usr/local/bin/unzip -t sr003-02-24192.zip 
Archive:  sr003-02-24192.zip
testing: sr003-02-24192.wav   OK
testing: __MACOSX/OK
testing: __MACOSX/._sr003-02-24192.wav   OK

getting better

% /usr/local/bin/unzip sr003-02-24192.zip -x __MACOSX/*
Archive:  sr003-02-24192.zip
  inflating: sr003-02-24192.wav

Flawless victory :)

Base unzip isn't very useful. Having it first in the path is just annoying.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org