Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
And rather than trusting gcc -march=native to do the work for you, you would probably be better off checking `grep ^flags /proc/cpuinfo' anyway. :) Gentoo runs not only on Linux (remember about Gentoo Prefix), and this feature is system-dependend -- Regards, Konstantin
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Mon, 13 Dec 2010 18:41:07 -0600 Ryan Hill dirtye...@gentoo.org wrote: Absolutely, -march=native should be used if you aren't using distcc. This is about setting CPU-based USE flags (mmx, sse, etc.). Exactly, or x86 CPU extensions based USE flags, more exactly. It seems the thread has been somewhat hijacked in favour of automating setting USE flags or at least helping to set them based on information found here and there on the system that happens to be queried for it, something we have never supported since the thankfully forgotten days of profiles/base/use.defaults. And rather than trusting gcc -march=native to do the work for you, you would probably be better off checking `grep ^flags /proc/cpuinfo' anyway. :) jer
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
2010/12/13 Ryan Hill dirtye...@gentoo.org
On Sun, 12 Dec 2010 09:01:13 -0400
Sergio D. Rodríguez Inclan srinc...@gmail.com wrote:
El 12/12/2010 02:46 a.m., Ryan Hill escribió:
I think the fewer sources of magic USE flags the better. Maybe we
could
document how to figure out what instruction sets a processor supports
in the
handbook instead.
A good manual would be greatly appreciated :)
I wrote a guide a couple weeks ago that might be a good starting point.
http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
if I read correctly the article on the wiki it does circa what the script
reported below does.
would be possible to adopt something similar for automatic C*FLAGS selection
if someone step in willing to take the pain to mantain it.
#!/usr/bin/python
# Copyright 2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# Author: Francesco Riosa
# extrapolated from http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS, errors
are mine
# kate: encoding utf-8; eol unix
# kate: indent-width 4; mixedindent off; replace-tabs on;
# kate: remove-trailing-space on; space-indent on
# echo int main() { return 0; } | gcc -march=native -v -E - 21 | grep
march
# echo int main() { return 0; } | gcc -march=core2 -v -Q -x c - 21
example output:
./hw-cflags.py
extrapolating flags for gcc-4.4.5
useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=generic
redundant:-mcx16 -msahf
extrapolating flags for gcc-4.5.1
useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=core2
redundant:-mcx16 -msahf
import os
import time
import fnmatch
from subprocess import Popen, PIPE
GCC_PATH = '/usr/bin/'
GCC_LIST = fnmatch.filter(os.listdir(GCC_PATH), 'gcc-[0-9].*')
GCC_LIST.sort()
def extract_flags(gcccmd, header):
# get output from gcc
buf = ''
devnul = open('/dev/null', 'w')
p = Popen(gcccmd, stdin=PIPE, stdout=devnul, stderr=PIPE)
p.stdin.write(int main() { return 0; })
p.stdin.close()
while p.poll() is None:
t = p.stderr.read()
buf = buf%s % t
time.sleep(0.01)
p.stderr.close()
devnul.close()
# parse it
flags = []
add = False
for line in buf.split('\n'):
if line.startswith(header):
add = True
flags += line.strip().split(' ')
continue
if add:
if line.startswith(' '):
flags += line.strip().split(' ')
else:
break
# extract flags we are interested in
t = []
march = ''
mtune = '-mtune=generic'
for i in xrange(len(flags)):
if flags[i].startswith('-m'):
if flags[i].startswith('-mtune'):
mtune = flags[i]
elif flags[i].startswith('-march'):
march = flags[i]
else:
t.append(flags[i])
elif flags[i] == '--param':
t.append(%s %s % (flags[i], flags[i+1]))
flags = t
return march, mtune, flags
for gcc in GCC_LIST:
print extrapolating flags for %s % gcc
gcccmd = [ GCC_PATH + gcc, '-march=native', '-v', '-E', '-', ]
header='COLLECT_GCC_OPTIONS'
march, mtune, flags_native = extract_flags(gcccmd, header)
gcccmd = [ GCC_PATH + gcc, march, '-v', '-Q', '-x', 'c', '-', ]
header='options enabled:'
t, t, flags_enabled = extract_flags(gcccmd, header)
redundant_flags = []
useful_flags = []
for x in flags_native:
if x in flags_enabled:
redundant_flags.append(x)
else:
useful_flags.append(x)
if gcc gcc-4.5.0:
mtune = '-mtune=generic'
print useful flags: %s %s %s % (march, .join(useful_flags),
mtune)
print redundant:%s % .join(redundant_flags)
print
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
13.12.2010, 18:53, Francesco R viv...@gmail.com:
2010/12/13 Ryan Hill dirtye...@gentoo.org
On Sun, 12 Dec 2010 09:01:13 -0400
Sergio D. Rodríguez Inclan srinc...@gmail.com wrote:
El 12/12/2010 02:46 a.m., Ryan Hill escribió:
I think the fewer sources of magic USE flags the better. Maybe we could
document how to figure out what instruction sets a processor supports in
the
handbook instead.
A good manual would be greatly appreciated :)
I wrote a guide a couple weeks ago that might be a good starting point.
http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
if I read correctly the article on the wiki it does circa what the script
reported below does.
would be possible to adopt something similar for automatic C*FLAGS selection
if someone step in willing to take the pain to mantain it.
#!/usr/bin/python
# Copyright 2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# Author: Francesco Riosa
# extrapolated from http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS, errors
are mine
# kate: encoding utf-8; eol unix
# kate: indent-width 4; mixedindent off; replace-tabs on;
# kate: remove-trailing-space on; space-indent on
# echo int main() { return 0; } | gcc -march=native -v -E - 21 | grep
march
# echo int main() { return 0; } | gcc -march=core2 -v -Q -x c - 21
example output:
./hw-cflags.py
extrapolating flags for gcc-4.4.5
useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=generic
redundant: -mcx16 -msahf
extrapolating flags for gcc-4.5.1
useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=core2
redundant: -mcx16 -msahf
import os
import time
import fnmatch
from subprocess import Popen, PIPE
GCC_PATH = '/usr/bin/'
GCC_LIST = fnmatch.filter(os.listdir(GCC_PATH), 'gcc-[0-9].*')
GCC_LIST.sort()
def extract_flags(gcccmd, header):
# get output from gcc
buf = ''
devnul = open('/dev/null', 'w')
p = Popen(gcccmd, stdin=PIPE, stdout=devnul, stderr=PIPE)
p.stdin.write(int main() { return 0; })
p.stdin.close()
while p.poll() is None:
t = p.stderr.read()
buf = buf%s % t
time.sleep(0.01)
p.stderr.close()
devnul.close()
# parse it
flags = []
add = False
for line in buf.split('\n'):
if line.startswith(header):
add = True
flags += line.strip().split(' ')
continue
if add:
if line.startswith(' '):
flags += line.strip().split(' ')
else:
break
# extract flags we are interested in
t = []
march = ''
mtune = '-mtune=generic'
for i in xrange(len(flags)):
if flags[i].startswith('-m'):
if flags[i].startswith('-mtune'):
mtune = flags[i]
elif flags[i].startswith('-march'):
march = flags[i]
else:
t.append(flags[i])
elif flags[i] == '--param':
t.append(%s %s % (flags[i], flags[i+1]))
flags = t
return march, mtune, flags
for gcc in GCC_LIST:
print extrapolating flags for %s % gcc
gcccmd = [ GCC_PATH + gcc, '-march=native', '-v', '-E', '-', ]
header='COLLECT_GCC_OPTIONS'
march, mtune, flags_native = extract_flags(gcccmd, header)
gcccmd = [ GCC_PATH + gcc, march, '-v', '-Q', '-x', 'c', '-', ]
header='options enabled:'
t, t, flags_enabled = extract_flags(gcccmd, header)
redundant_flags = []
useful_flags = []
for x in flags_native:
if x in flags_enabled:
redundant_flags.append(x)
else:
useful_flags.append(x)
if gcc gcc-4.5.0:
mtune = '-mtune=generic'
print useful flags: %s %s %s % (march, .join(useful_flags), mtune)
print redundant: %s % .join(redundant_flags)
print
Note that for some architectures (e.g, PowerPC) you need to add -mcpu case
(-mcpu has the same meaning as -march on x86)
--
Regards,
Konstantin
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
Hi all, On Mon, Dec 13, 2010 at 07:07:10PM +0100, Francesco R wrote: 2010/12/13 Konstantin Tokarev annu...@yandex.ru 13.12.2010, 18:53, Francesco R viv...@gmail.com: 2010/12/13 Ryan Hill dirtye...@gentoo.org On Sun, 12 Dec 2010 09:01:13 -0400 Sergio D. Rodr?guez Inclan srinc...@gmail.com wrote: El 12/12/2010 02:46 a.m., Ryan Hill escribi?: I think the fewer sources of magic USE flags the better. Maybe we could document how to figure out what instruction sets a processor supports in the handbook instead. A good manual would be greatly appreciated :) I wrote a guide a couple weeks ago that might be a good starting point. http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS if I read correctly the article on the wiki it does circa what the script reported below does. would be possible to adopt something similar for automatic C*FLAGS selection if someone step in willing to take the pain to mantain it. I've been reading this thread, but I don't understand why we need to worry about this since the newer versions of gcc can figure it out automatically by using -march=native? Does -mcpu also have a native option for those who need to use -mcpu? William pgpdtGAkSilGP.pgp Description: PGP signature
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Mon, 13 Dec 2010 18:13:08 -0600 William Hubbs willi...@gentoo.org wrote: I've been reading this thread, but I don't understand why we need to worry about this since the newer versions of gcc can figure it out automatically by using -march=native? Absolutely, -march=native should be used if you aren't using distcc. This is about setting CPU-based USE flags (mmx, sse, etc.). Does -mcpu also have a native option for those who need to use -mcpu? I think native is limited to amd64, x86, alpha, and mips. -- fonts, gcc-porting, it makes no sense how it makes no sense toolchain, wxwidgets but i'll take it free anytime @ gentoo.orgEFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
El 12/12/2010 02:46 a.m., Ryan Hill escribió: On Sat, 11 Dec 2010 19:01:16 + Matt Turnermatts...@gentoo.org wrote: On Sat, Dec 11, 2010 at 5:57 PM, Jeroen Rooversj...@gentoo.org wrote: [snip] I agree that this could be better. To me, most of the problems with this are due to users not knowing which of these should be set for their particular CPU. Instead of having defaults set by a profile, I'd like to figure out a way we can have these flags set by default dependent on the user's CPU. This might require some additional logic in portage; I don't know. I think the fewer sources of magic USE flags the better. Maybe we could document how to figure out what instruction sets a processor supports in the handbook instead. A good manual would be greatly appreciated :) Regards, Sergio.
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Sun, 12 Dec 2010 13:09:08 -0500 ross smith r...@umich.edu wrote: Perhaps something along the lines of: If CPU_FLAGS is empty or not defined, set flags based on the -march and -mtune variables. If CPU_FLAGS is set, respect what has been set there and ignore the other logic for defaults. -march flags are the lowest common denominator for an architecture. For example, core2 enables MMX, SSE, SSE2, SSE3, and SSSE3 while a Core i7 processor also supports SSE4.1, SSE4.2, and AES instructions. It might be useful to use -march as a base and leave it up to the user to set any additional flags, but I don't see how that is any different than what we have now. In addition it would require us to maintain accurate mappings of what -march enables what flags and deal with the fact that these flags may not exist between gcc versions. It also falls apart with -march=native. -- fonts, gcc-porting, it makes no sense how it makes no sense toolchain, wxwidgets but i'll take it free anytime @ gentoo.orgEFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 signature.asc Description: PGP signature
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Sun, 12 Dec 2010 09:01:13 -0400 Sergio D. Rodríguez Inclan srinc...@gmail.com wrote: El 12/12/2010 02:46 a.m., Ryan Hill escribió: I think the fewer sources of magic USE flags the better. Maybe we could document how to figure out what instruction sets a processor supports in the handbook instead. A good manual would be greatly appreciated :) I wrote a guide a couple weeks ago that might be a good starting point. http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS -- fonts, gcc-porting, it makes no sense how it makes no sense toolchain, wxwidgets but i'll take it free anytime @ gentoo.orgEFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Sun, Dec 12, 2010 at 11:09:01PM -0600, Ryan Hill wrote: On Sun, 12 Dec 2010 09:01:13 -0400 Sergio D. Rodríguez Inclan srinc...@gmail.com wrote: El 12/12/2010 02:46 a.m., Ryan Hill escribió: I think the fewer sources of magic USE flags the better. Maybe we could document how to figure out what instruction sets a processor supports in the handbook instead. A good manual would be greatly appreciated :) I wrote a guide a couple weeks ago that might be a good starting point. http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS For older stuff and esp. alternative architectures, dig out the genflags utility that I wrote for the 2004.0 release, at drobbin's request. The data would be the main stuff of use, the application itself not so much. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
Dirkjan Ochtman posted on Sat, 11 Dec 2010 20:03:39 +0100 as excerpted: On Sat, Dec 11, 2010 at 20:01, Matt Turner matts...@gentoo.org wrote: I agree that this could be better. To me, most of the problems with this are due to users not knowing which of these should be set for their particular CPU. Instead of having defaults set by a profile, I'd like to figure out a way we can have these flags set by default dependent on the user's CPU. This might require some additional logic in portage; I don't know. Yeah, I think setting the best possible default is important here. FWIW, I don't think that's a good idea -- at least if you're suggesting real-time build-machine detection (see below for the alternative). Not only will implementing it be a lot of work, but then you'll have people filing bugs because they intended to compile for several different machines and set generic CFLAGS accordingly, but still ended up with broken packages because the default CPU extensions logic looked at the build machine only, and they thought they were done when they set the profile, CFLAGS and CHOST to the level of generic they intended. Unless the logic is based on the -march set in C(XX)FLAGS. That could work, as people should already be setting that as specific or generic as they want. I do like the USE_EXPAND idea, tho. Meanwhile, would it be possible to have altivec in the same USE_EXPAND? It's the same CPU extensions type of thing on PPC, if I'm not mistaken. Does profile-mask work for USE_EXPAND? Because it just seems strange to me to see all those CPU extensions in their own USE_EXPAND and see altivec still in USE. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Sat, 11 Dec 2010 18:57:58 +0100 Jeroen Roovers j...@gentoo.org wrote: Among all CPU extensions USE flags you'll find: 3dnow 3dnowext mmx mmxext sse sse2 sse3 sse4 sse4a sse5 ssse3 I probably missed a few, there. sse4.1, sse4.2, avx sse5 was a draft, it was never implemented. -- fonts, gcc-porting, it makes no sense how it makes no sense toolchain, wxwidgets but i'll take it free anytime @ gentoo.orgEFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 signature.asc Description: PGP signature
[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable
On Sat, 11 Dec 2010 19:01:16 + Matt Turner matts...@gentoo.org wrote: On Sat, Dec 11, 2010 at 5:57 PM, Jeroen Roovers j...@gentoo.org wrote: [snip] I agree that this could be better. To me, most of the problems with this are due to users not knowing which of these should be set for their particular CPU. Instead of having defaults set by a profile, I'd like to figure out a way we can have these flags set by default dependent on the user's CPU. This might require some additional logic in portage; I don't know. I think the fewer sources of magic USE flags the better. Maybe we could document how to figure out what instruction sets a processor supports in the handbook instead. -- fonts, gcc-porting, it makes no sense how it makes no sense toolchain, wxwidgets but i'll take it free anytime @ gentoo.orgEFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 signature.asc Description: PGP signature
