I am sponsoring the following fasttrack for myself --- timeout 8/11/2010.
It adds three new options to the Solaris link-editor (ld).
A copy of the original (ld.1.orig) and new (ld.1.new) ld manpage, as well
as diffs (ld.1.diffs) can be found in the case materials.
Release Binding: Patch/Micro
-z guidance: Committed
-z fatal-warnings/--fatal-warnings Committed
-z nofatal-warnings/--no-fatal-warnings Committed
--------------------------------------------------------------------------
The new options are:
-z guidance
The link-editor has many options that are highly
recommended, but which are not on by default. When
the user enables the -z guidance feature, ld will issue
"guidance" warning (non-fatal) messages to recommend
options or changes that will result in a better object.
-z fatal-warnings, --fatal-warnings
ld warnings are advisory, in that the link-editor continues
on to build the output object. If -z fatal-warnings is set,
warnings are treated as fatal conditions --- the output object
is not built, and a non-zero exit status is returned. This
feature is useful for building code that is held to very
high standards (such as the core Solaris OS/net consolidation),
since it will prevent inadvertent errors from slipping past
developers. It should be noted that this includes the warnings
issued by -z guidance.
--fatal-warnings is the name used by the GNU ld for their version of
this feature. We will accept it as an alias for -z fatal-warnings.
-z nofatal-warnings, --no-fatal-warnings
Undo the effect of -z fatal-warnings, and revert to non-fatal
warnings. This is the default, and so should be rarely needed,
but is useful in the context of the LD_OPTIONS environment
variable to override the use -f -z fatal-warnings from within
a Makefile:
% LD_OPTIONS=-D-znofatal-warnings make
--no-fatal-warnings is the name used by GNU ld for their version of
this feature. We will accept it as an alias for -z nofatal-warnings.
I have successfully built the OS/net with -z guidance, and -z fatal-warnings
turned on. I was able to get a completely clean build with only a modest
set of changes (167 lines changed: 54 ins; 46 del; 67 mod). This reflects the
fact that we already do extensive checking of the objects in the OS/net
(via check_rtime). In general, the changes fall into the following categories:
- Suppress guidance in known cases where the flagged behavior
needs to be retained.
- Some applications pull in dependencies they do not directly require
for the benefit of plugins they later load, and the dependencies for
those plugins are not completely specified. Correcting the plugin
dependencies allows these extras to be dropped.
Although not part of this case, I expect to integrate these changes
in a future project, and to apply -z guidance and -z fatal-warnings to
the OS/net by default. This should make it easier for developers to spot
difficulties earlier in the coding process.
--------------------------------------------------------------------------
Example:
The following example demonstrates how the guidance feature
is intended to work. We will build a shared object that
has a variety of shortcomings:
- Does not specify all it's dependencies
- Specifies dependencies it does not use
- Does not use direct bindings
- Uses a version 1 mapfile
- Contains relocations to the readonly allocable text (not PIC)
This scenario is sadly very common --- many shared objects
have one or more of these issues.
% cat hello.c
#include <stdio.h>
#include <unistd.h>
void
hello(void)
{
printf("hello user %d\n", getpid());
}
% cat mapfile.v1
# This version 1 mapfile will trigger a guidance message
% cc -c hello.c
% ld -o hello.so -G -M mapfile.v1 hello.o -lelf
As you can see, the operation completes without error, resulting
in a usable object. However, turning on guidance reveals a number
of things that could be better:
ld -o hello.so -G -M mapfile.v1 hello.o -lelf -zguidance
ld: guidance: version 2 mapfile syntax recommended: mapfile.v1
ld: guidance: -z lazyload option recommended before first dependency
ld: guidance: -B direct or -z direct option recommended before
first dependency
Undefined first referenced
symbol in file
getpid hello.o (symbol belongs to implicit
dependency /lib/libc.so.1)
printf hello.o (symbol belongs to implicit
dependency /lib/libc.so.1)
ld: warning: symbol referencing errors
ld: guidance: -z defs option recommended for shared objects
ld: guidance: removal of unused dependency recommended: libelf.so.1
warning: Text relocation remains referenced
against symbol offset in file
.rodata1 (section) 0xa hello.o
getpid 0x4 hello.o
printf 0xf hello.o
ld: guidance: position independent (PIC) code recommended for shared objects
ld: guidance: see ld(1) -z guidance for more information
Given the explicit advice in the above guidance messages, it is
relatively easy to modify the example to do the right things:
% cat mapfile.v2
# This version 2 mapfile will not trigger a guidance message
$mapfile_version 2
% cc -c -Kpic hello.c
% ld -o hello.so -G -Bdirect -M mapfile.v2 hello.o -lc -zguidance
Although unlikely, one might imagine a scenario in which it was desired
to use non-PIC code, and accept the price of performing relocations at
runtime on the readonly allocable text segment:
% cc -c hello.c
% ld -o hello.so -G -Bdirect -M mapfile.v2 hello.o -lc -zguidance
warning: Text relocation remains referenced
against symbol offset in file
.rodata1 (section) 0xa hello.o
getpid 0x4 hello.o
printf 0xf hello.o
ld: guidance: position independent (PIC) code recommended for shared objects
ld: guidance: see ld(1) -z guidance for more information
It is easy to disable that specific guidance warning without losing the
overall benefit from allowing the remainder of the guidance feature to
operate:
% ld -o hello.so -G -Bdirect -M mapfile.v2 hello.o -lc -zguidance=notext
--------------------------------------------------------------------------
Rationale:
The information that follows below is not strictly part of this
case, but provides background information that may prove useful
to the reader in understanding the evolution in thinking and design
that led to the implementation of -z guidance and -z fatal-warnings
in their current form.
The Solaris link-editor is one of the older Unix commands. Over
the years, applications have grown in size and complexity, and the ELF
system has evolved to provide them with tools needed to manage their
growing requirements. Features such as lazy loading, and direct bindings
have been added. In an ideal world, many of these options would be defaults,
with rarely used options that allow the user to turn them off. However, the
reality is exactly the reverse: For backward compatibility, these features
are all options that must be explicitly turned on by the user. This has
led to a situation in which most applications do not take advantage of
the many improvements that have been made in linking over the last 20
years. If their code seems to link and run without issue, what motivation
does a developer have to read a complex manpage, absorb the information
provided, choose the features that matter for their application, and apply
them? Experience shows that only the most motivated and diligent programmers
will make that effort. We would like to do something to make it easier
for everyone else.
There have been many conversations over the years regarding this issue,
and how to address it. They break down along the following lines:
Change ld defaults
Since the world would be a better place if these ld features were
defaults, the ls command could simply be changed to make them so.
This idea is simple, elegant, and impossible. Doing so would
break a large number of existing applications, including those of
ISVs, big customers, a plethora of existing open source packages.
In each case, the owner of that code may choose to follow our lead
and fix their code, or they may view it as an invitation to reconsider
their commitment to our platform. Backward compatibility, and our
installed base of working software, is one of our greatest assets,
and not something to be lightly put at risk. Breaking backward
compatibility at this level of the system is likely to do more
harm than good.
New link-editor
One might create a new linker command, not called 'ld', leaving the
old command as it is. The new one could use the same code as 'ld',
but would offer only modern options, with the proper defaults for
features such as direct binding.
The resulting link-editor would be a pleasure to use. However,
the approach is doomed to niche status. There is a vast pile
of exiting code in the world built around the 'ld' command, that
reaches back to the 1970's. ld use is embedded in large and unknown
numbers of makefiles, and is used by name by compilers that
execute it. A Unix link-editor that is not named 'ld' will not find
a majority audience no matter how good it might be.
Finally, a new linker command will eventually cease to be
new, and will accumulate its own burden of backward compatibility
issues.
An option to make 'ld' do the right things automatically
This line of reasoning is best summarized by a CR filed
in 2005, entitled
6239804 make it easier for ld(1) to do what's best
The idea is to have a '-z best' option that unchains ld from
its backward compatibility commitment, and allows it to turn
on the "best" set of features, as determined by the authors
of 'ld'. The specific set of features enabled by -z best would
be subject to change over time, as requirements change.
This idea is more realistic than the other two, but has not been
implemented to date because it has some significant issues.
- The -z best proposal assumes that the user can turn it
on, and trust it to select good options without the user
needing to be aware of the options being applied. This is
a fallacy. Features such as direct bindings require the
user to do some analysis to ensure that the resulting
program will still operate properly.
- A user who is willing to do the work to know what -z best
does is capable of turning on those features directly,
and therefore gains little benefit from -z best.
- The intent is that when a user opts into -z best, that
they understand that z best is subject to sometimes
incompatible evolution. Experience teaches us that
this won't work. People will use this feature, the meaning
of -z best will change, code that used to build will fail,
and then there will be complaints and demands to retract
the change. When (not if) this occurs, we will of course
defend our actions, and point at the disclaimer. We'll win
some of those debates, and lose others. Ultimately, we'll
end up with -z best2 (-z better), or other compromises, and
our goal of simplifying the world will have failed.
- The -z best idea rolls up a set of features that
may or may not be related to each other into a unit
that must be taken wholesale, or not at all. It could
be that only a subset of what it does is compatible
with a given application, in which case the user is expected
to abandon -z best and instead set the options that apply to
their application directly. In doing so, they lose one of
the benefits of -z best, that if you use it, future versions
of ld may choose a different set of options, and automatically
improve the object through the act of rebuilding it.
I draw two conclusions from the above history:
(1) For a link-editor, backward compatibility is vital. If
a given command line linked your application 10 years ago,
you have every reason to expect that it will link today,
assuming that the libraries you're linking against are still
available and compatible with their previous interfaces.
(2) For an application of any size or complexity, there is no
substitute for the work involved in examining the code and
determining which linker options apply and which do not.
These options are largely orthogonal to each other, and
there are reasonable reasons not to use any or all of them,
even in modern applications. It is a mistake to tie them
together.
The idea for -z guidance came from consideration of these points.
By decoupling the advice from the act of taking the advice, we can
retain the good aspects of -z best while avoiding its pitfalls:
- -z guidance gives advice, but the decision to take that advice
remains with the user who must evaluate its merit and make
a decision to take it or not. As such, we are free to change
the specific guidance given in future releases of ld, without
breaking existing applications. The only fallout from this will
be some new warnings in the build output, which can be ignored
or dealt with at the user's convenience.
- It does not couple the various features given into a single
"take it or leave it" option, meaning that there will be no
need to offer "-zguidance2", or other such variants when
things change over time.
- The user is given the flexibility to disable specific
categories of guidance without losing the benefit of
others, including those that might be added to future
versions of the system.
Although -z fatal-warnings stands on its own as a useful feature,
it is of particular interest in combination with -z guidance.
Used together, the guidance turns from advice to hard requirement: The
user must either make the suggested change, or explicitly reject
the advice, in order to get a build. This is valuable in environments
with high coding standards. In addition, it gains us an additional
point of compatibility with the GNU link-editor.
_______________________________________________
opensolaris-arc mailing list
opensolaris-arc@opensolaris.org
