Sorry that it has taken me so long to get back around to this. Holidays
vacation and other work.
On 11/15/23 11:19, Ron Brender wrote:
Ben,
This is a good first proposal for a DWARF Extension Registry.
Following are a few initial reactions...
1. I don't understand why this proposal is an alternative to 230324.1.
That proposal defines a way to get more numbers. This
proposal describes how to manage use of those numbers. Seems like both
proposals are necessary.
That is a fair point. I can see it both ways.
My reasoning is that the reason why we are running out of vendor defined
space is that within in the various vendor spaces the encoding space is
consumed by legacy extensions that:
1) were never implemented publicly
2) were implemented but are no longer in use because the compilers that
generated them have been abandoned
3) were in use but have been incorporated into the standard version of
DWARF.
I feel like clearing those out by drawing a line in the sand and saying
that extensions which existed in previous versions of DWARF do not
necessarily mean the same thing once the new version of DWARF is
released, should clear out the legacy cruft such that there should be
sufficient encoding space for new producer extensions.
I'm personally not a big fan of 230324.1 or at best I'm kind of
ambivalent toward it.
1) As long as they have at least one number left in the encoding space
they can do this already. They just take the last value and define it as
an extension operator. I don't see why this must be in the standard itself.
2) I do not like the idea of having a practically infinite extension
space. I will admit that it is not the strongest argument but I believe
that having a limited producer encoding space encourages the authors of
producers to work through the standardization process and standardize
their extension rather than just accumulating them in the producer
defined encoding space.
3) I also feel like intentionally requiring the producer defined
extensions to be reconsidered with every major version release of the
standard helps keep the standardization process moving along.
2. The proposed text seems to randomly replace "vender-defined" with
"producer-defined", "producer defined" or "producer specific"
(curiously omitting "producer-specific" :-). Use one rendition
consistently, which I think should be "producer-defined".
I will fix this and submit a revised version.
3. Whatever URLs are part of the registration process should be
mentioned in only one place. Section 7.1 is an appropriate place.
References to 7.1 can be included elsewhere if really needed.
I will fix this in a revised version.
4. Re 1.3.13. I can't tell if you actually changed anything in the
first two paragraphs.
I'll try to make the change more obvious in the next version.
5. Re 1.3.13: Blaming "reduced tool compatibility" on the
skipability of unknown constructs seems a huge and unjustified claim.
I don't buy it.
In most cases, I can see your point but in this case I cannot. To me
this seems obvious.
Of course a consumer should be able to skip over something that it
doesn't understand. However, if a consumer wants to achieve full
compatibility with the producer, it needs to be able to interpret those
constructs. It can't skip over them. That seems unquestionably obvious
to me. So that the consumer's author can write the code that allows the
consumer to interpret these producer specific constructs, the text
refers the consumer to the registry.
6. Re 6.1.1.2 and 6.2.4.2 <http://6.2.4.2>: Why single these out? This
is no different than any other producer-defined extension and should
be covered adequately by the general discussion in 7.1
There was specific text in those sections which seemed to need to be
changed.
7. Re 7.1: There seems to be text missing between "where there is a
vendor" and "but it can also".
By tieing a producer-defined extension to a particular version of
DWARF, does that mean that when a new version of DWARF comes out that
producer must re-register the extension to keep "ownership" of the code?
I would say that as part of the new DWARF version release process,
asking the producer developers who have registered extensions if those
extensions are still meaningful and useful in the context of the new
DWARF version.
If so, then it gets carried over. However, If some particular extension
continues to be useful across many versions of DWARF and producers are
still actively using it, then that extension should be considered for
standardization.
If the extension has been standardized as part of the new DWARF version
then that producer specific encoding can be reused in the context of the
new DWARF version. In other words, producers should prefer the
standardized way of expressing a concept over a producer specific way.
Anything short of permanent registration seem contrary to our general
backward compatible mindset. I think associating an extension with a
DWARF version is probably not a good idea.
Backward compatibility is a worthy goal but if no producer is going to
generate that producer specific extension in the context of that DWARF
version, then it just consumes encoding space. Compatibility between
tools and specifically backward compatibility needs to be more than a
theoretical idea. There needs to be an extant producer that will
generate DWARF with those extensions that can the consumer can be tested
against. When you look at:
https://sourceware.org/elfutils/DwarfExtensions There are far too many
things like:
DW_TAG_MIPS_loop 0x4081 Never implemented, mips_extensions
Nothing Constant only
DW_AT_sf_names 0x2101 DWARF1 only? Unknown Constant only
DW_AT_src_info 0x2102 DWARF1 only? Unknown Constant only
DW_AT_mac_info 0x2103 DWARF1 only? Unknown Constant only
DW_AT_src_coords 0x2104 DWARF1 only? Unknown Constant only
DW_AT_body_begin 0x2105 DWARF1 only? Unknown Constant only
DW_AT_body_end 0x2106 DWARF1 only? Unknown Constant only
We don't know what it does, we don't know of any producer which
generates it. No consumer can do anything with it. It is just taking up
space. No producer is going to emit those in the context of DWARF6,
let's reuse the encodings.
And there are things like:
DW_AT_GNU_guarded_by 0x2108 GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded_by 0x2109 GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_guarded 0x210a GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded 0x210b GNU ThreadSafetyAnnotations Not
implemented Constant only
They never got implemented. No producer actually uses them, no consumer
knows what to do with them.
Then there are things like:
DW_AT_GNU_call_site_value 0x2111 GNU call site DWARF5 proposal
GCC Recognized in readelf
DW_AT_GNU_call_site_data_value 0x2112 GNU call site DWARF5
proposal GCC Recognized in readelf
DW_AT_GNU_call_site_target 0x2113 GNU call site DWARF5 proposal
GCC Recognized in readelf
DW_AT_GNU_call_site_target_clobbered 0x2114 GNU call site DWARF5
proposal GCC Recognized in readelf
Presumably, if I do gcc -gdwarf-4 then gcc will use these vendor
specific encodings rather than the DWARF5 standardized versions.
However, GCC should never emit those when it is emitting DWARF5 or
later. This seems like a legitimate case where a consumer may want to
implement backward compatibility. Those attribute encoding can be tied
to DWARF2-4 and then those encodings can be reused in the context of DWARF6.
If extensions are registered, is there any remaining need for
"vender_id"/"vender_id_name" in names. Uniqueness of codes is assured
by the
requirement to register, right? So instead of some kind of
vendor-related text we could just use "ext_" to keep names separate
from the standard stuff.
That is a good point. The registration process can log the producers
which make use of the extension.
8. This is perhaps enough of a proposal to describe changes to the
DWARF document itself. But I would like to see the proposal include
more about the registration contents proper. In particular, I think
there needs to be a requirement that the registration include a
complete specification of the extension. The goal is to allow
(encourage) other consumers to accept the same extension if
appropriate but also for other producers to adopt the same extension
if suitable.
In that connection, there probably needs to be some kind of statement
that any and all information included in a registration must be free
of any restrictions (patent, copyright, trademark, whatever) and
freely reusable by others.
Sounds like a good idea to me. I just think that the registration
process should not be so onerous that it inhibits disclosure and
innovation. For example there was an idea behind:
DW_AT_GNU_guarded_by 0x2108 GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded_by 0x2109 GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_guarded 0x210a GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded 0x210b GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_locks_excluded 0x210c GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_exclusive_locks_required 0x210d GNU
ThreadSafetyAnnotations Not implemented Constant only
DW_AT_GNU_shared_locks_required 0x210e GNU
ThreadSafetyAnnotations Not implemented Constant only
It might have been a good one, it might have been a bad one; I don't
know I wasn't involved. Maybe someone even did a prototype
implementation that never was accepted into a mainline release. I think
it is still worthwhile to allow an eager compiler developer to register
those attributes do their prototype implementation in their own reserved
sandbox. If it catches on, then great; they can provide more information
so other tools can implement it. If the idea gets abandoned then the
encodings only exist for that release of the DWARF version and the
encodings can be reused in the next DWARF cycle.
That is enough for now,
Ron
On Fri, Nov 10, 2023 at 4:52 PM Ben Woodard via Dwarf-discuss
<dwarf-discuss@lists.dwarfstd.org> wrote:
DWARF Extension Registry
Background
The DWARF standard has always had the wisdom to acknowledge the
need for Vendor Extensibility. Section 1.3.13 describes this
policy. For the producers it explicitly reserves some of the valid
values for encoding various constructs and promises not to use
those values in future versions of the standard. Consumers are
given the freedom and expected to skip over data that they do not
recognize.
The original intent of these vendor extensions was that they would
be a private agreement between a particular producer and a
cooperating consumer. However, the range of tools expanded beyond
a toolchain provided by a single vendor and so the concept of a
private agreement between a specific producer and a cooperating
consumer came to be regarded as a registry of vendor extensions
that every tool must be aware of for the sake of compatibility.
Because compatibility between tools was prioritized, the
extensions in these de facto registries were effectively
considered permanent unofficial DWARF encodings. This presumptive
permanence is in spite of the fact that in some cases:
*
Producers that made use of these extensions were never
implemented or publicly released.
*
Producers that made use of these extensions are so obsolete
that not only does the tool chain no longer exist but the
hardware that this toolchain was designed for no longer exists
except as historical artifacts.
*
Official versions of DWARF encodings that accomplish the same
thing have been added to the standard making the vendor
extension obsolete.
The accumulation of these vendor extensions over the years has led
to problems with some DWARF constructs where they are running out
of available encodings that do not overlap with the ones used by
or reserved by the standard. In particular the range of vendor
vendor extension encoding is running particularly short in the
range of DW_OP encoding. This has led to several informal
proposals within the vendor community along the lines of an
official proposal 230324.1 Expression Operation Vendor
Extensibility Opcode https://dwarfstd.org/issues/230324.1.html
<https://dwarfstd.org/issues/230324.1.html>The basic idea in most
of these proposals is to extend the range of available vendor
extension encodings. While DWARF expression operations, Section
2.5, has the most extreme shortage of vendor encodings, other
DWARF constructs are also running out of usable encodings.
Overview
This proposal is an alternative to 230324.1 and similar informal
approaches to increase the available encoding space. It does this
in two ways.
First it makes it explicit that Vendor Extensions can only be
interpreted in the context of a particular producer and a DWARF
standard version. This way obsolete vendor extensions can be
retired when a new version of the standard is released. It also
begins the process of replacing the expectation for consumers that
they can simply ignore encodings that they do not understand with
the expectation they will need to have producer and version
awareness when interpreting encodings. This is because tools
chains are no longer developed by a close knit group of developers
who all work for a single vendor. Tools are now developed by
diverse distributed teams and users need and rightly expect
interoperability.
Because the bulk of tool development is no longer done by vendors
who build a vertically integrated tool chain but rather is done by
looser confederation of projects that need to cooperate to achieve
compatibility, the term “vendor” is replaced with more generic
terms when referring to these extensions.
The second big change is it seeks to foster compatibility and
collaboration by taking the collection of informal registries such
as https://sourceware.org/elfutils/DwarfExtensions
<https://sourceware.org/elfutils/DwarfExtensions>and
https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags
<https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags>and
collects them and brings them under the aegis of the DWARF
standards body. To facilitate innovation and quickly meet
producer’s and consumer’s needs the expectation is that
registering a new producer specific construct and encoding would
be a simple, light weight or even partially automated task. It
would not be at all like amending the standard. The registry would
be published on the DWARF standard’s web site so that consumers
can easily find it. Each extension would be listed along with the
producers that emit it and the range of DWARF standard versions
that it applies to. When new DWARF standard versions are released,
each extension will be reconsidered to see if it continues to add
utility in the context of the new version of the DWARF standard.
Extensions that persist over several versions of the DWARF
standard probably should be considered for inclusion into the
official standard.
Proposed Changes
Section 1.3.13
Section 1.3.13 currently named “Vendor Extensibility” shall be
renamed to “Extensibility” to reflect the fact that cooperating
tools using these extensions are no longer being developed by
vendors who control the entire tool chain. Likewise the following
paragraphs should change to:
This document does not attempt to cover all interesting languages
or even to cover all of the possible debugging information needs
for its primary target languages. Therefore, the document provides
tool developers a way to define their own debugging information
tags, attributes, base type encodings, location operations,
language names, calling conventions and call frame instructions by
reserving a subset of the valid values for these constructs for
tool developer specific additions and defining related naming
conventions. Producers may also use debugging information entries
and attributes defined here in new situations. Future versions of
this document will not use names or values reserved for these
specific additions but the meanings of these encoding may change
from DWARF version to version. Therefore the values of these
extensions must be interpreted in the context of the DWARF
standard versions for which they apply. All names and values not
explicitly reserved for producer specific additions, however, are
reserved for future versions of this document.
Where this specification provides a means for describing the
source language, implementers are expected to adhere to that
specification. For language features that are not supported,
implementers may use existing attributes in novel ways or add
producer-defined attributes.
Implementers who make extensions are strongly encouraged to design
them to be compatible with this specification in the absence of
those extensions. Implementers should also register them at <url>
so that they can be included in the database of known extensions
which can be found at <url>. This will allow consumers that they
are not directly cooperating with to be aware of their extensions
and know how to interpret them.
While the DWARF format is organized so that a consumer can skip
over data which it does not recognize and this may allow a
consumer to read and process files generated according to a later
version of this standard or which contain producer specific
extensions, albeit possibly in a degraded manner, this has been
found to reduce the tool compatibility which users have come to
expect. Consumers should therefore refer to <url> when
encountering a producer specific construct that they are not
familiar with.
Section 6.1.1.2 Structure of the Name Index
The paragraph:
A producer may define additional vendor-specific attributes, and a
consumer will be able to ignore and skip over any attributes it is
not prepared to handle.
Shall be replaced by:
A producer may define additional producer-specific attributes, and
a consumer should be able to ignore and skip over any attributes
it is not prepared to handle. However, to achieve full
compatibility a consumer should refer to <url> where all known
producer specific attributes that apply to the Name Index are
enumerated.
Section 6.2.4 The Line Number Program Header
In the definition of “opcode base”, the phrase “vendor-specific
extensions” should be replaced with “producer-specific extensions”
in all three places where it is used.
Section 6.2.4.2 Vendor-defined Content Descriptions
The name of the section should be changed to “Producer-defined
Content Descriptions” and the first word of the first sentence
should also be changed to match.
The non-normative text following the section should be changed to:
If a consumer encounters a producer-defined content type that it
does not understand, it should skip the content data as though it
were not present. However, to achieve full compatibility the
consumer’s developer should refer to <url> where all known
producer-defined line number content descriptors are enumerated.
Section 6.3.1 Macro Information Header
In the second paragraph of item 4 the phrase “Vendor extension
entry opcodes” should be replaced with “Producer specific entry
opcodes”.
Section 7.1 Vendor Extensibility
The name of the section should be changed from “Vendor
Extensibility” to “Extensibility”.
Throughout section 7.1 the phrase “vendor specific” should be
replaced with “producer specific”.
In the second paragraph, the sentence beginning with “Vendors may
use values” should be replaced with “Producers may use values”.
The paragraph and list of points at the end of the section should
be replaced with:
Producer defined tags, attributes, base type encodings, location
atoms, language names, line number actions, calling conventions
and call frame instructions, historically have used the form
prefix_vendor_id_name, where vendor_id is some identifying
character sequence chosen so as to avoid conflicts with other
vendors. This established convention can continue in the cases
where there is historical precedent or where there is a vendor but
it can also be the name of the producer's project when there is no
specific vendor.
To ensure that extensions added by one producer may be safely
ignored by consumers that do not understand those extensions, the
following rules must be followed:
1.
Producer defined extensions must be evaluated in the context
of a DWARF standard version.
2.
New attributes are added in such a way that a debugger may
recognize the format of a new attribute value without knowing
the content of that attribute value.
3.
The semantics of any new attributes do not alter the semantics
of previously existing attributes.
4.
The semantics of any new tags do not conflict with the
semantics of previously existing tags.
5.
New forms of attribute value are not added.
To ensure full tool intercompatibility, producer defined DWARF
extensions should be registered with the DWARF standards body at
<URL>.
Section 7.32 Type Signature Computation
In the 3rd paragraph of point 4 replace “vendor-specific
attributes” with “producer-specific attributes”.
Appendix A
In the first paragraph, replace “and new, vendor-defined ones”
with “and producer defined ones”
Appendix D.7.3
In the text between the examples replace “vendor-specific” with
“producer-specific” in both cases.
--
Dwarf-discuss mailing list
Dwarf-discuss@lists.dwarfstd.org
https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss
--
Dwarf-discuss mailing list
Dwarf-discuss@lists.dwarfstd.org
https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss
