Re: [edk2-devel] [PATCH v2 1/3] MdeModulePkg: ImagePropertiesRecordLib: Use SectionAlignment for CodeSize

[Marvin Häuser]

Reviewed-by: Marvin Häuser 

On Mon, Mar 11, 2024 at 02:29 PM, Oliver Smith-Denny wrote:

> 
> When an ImageRecord is stored by ImagePropertiesRecordLib, it reports the
> CodeSegmentSize as the SizeOfRawData from the image. However, the image
> as loaded into memory is aligned to the SectionAlignment, so
> SizeOfRawData is under the actual size in memory. This is important,
> because the memory attributes table uses these image records to create
> its entries and it will report that the alignment of an image is
> incorrect, even though the actual image is correct.
> 
> This was discovered on ARM64, which has a 64k runtime page granularity
> alignment, which is backed by a 64k section alignment for
> DXE_RUNTIME_DRIVERs. The runtime code and data was correctly being
> loaded into memory, however the memory attribute table was incorrectly
> reporting misaligned ranges to the OS, causing attributes to be
> ignored for these sections for OSes using greater than 4k pages.
> 
> This patch correctly aligns the CodeSegmentSize to the SectionAlignment
> and the corresponding memory attribute table entries are now correctly
> aligned and pointing to the right places in memory.
> 
> Cc: Liming Gao 
> Cc: Leif Lindholm 
> Cc: Ard Biesheuvel 
> Cc: Sami Mujawar 
> Cc: Taylor Beebe 
> 
> Signed-off-by: Oliver Smith-Denny 
> ---
> MdeModulePkg/Library/ImagePropertiesRecordLib/ImagePropertiesRecordLib.c=
> | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/MdeModulePkg/Library/ImagePropertiesRecordLib/ImagePropertie=
> 
> sRecordLib.c b/MdeModulePkg/Library/ImagePropertiesRecordLib/ImagePropert=
> 
> iesRecordLib.c
> index e53ce086c54c..763a8d65d565 100644
> --- a/MdeModulePkg/Library/ImagePropertiesRecordLib/ImagePropertiesRecord=
> 
> Lib.c
> +++ b/MdeModulePkg/Library/ImagePropertiesRecordLib/ImagePropertiesRecord=
> 
> Lib.c
> @@ -1090,7 +1090,9 @@ CreateImagePropertiesRecord (
> ImageRecordCodeSection->Signature =3D IMAGE_PROPERTIES_RECORD_CODE=
> _SECTION_SIGNATURE;
> =20
> ImageRecordCodeSection->CodeSegmentBase =3D (UINTN)ImageBase + Sec=
> tion[Index].VirtualAddress;
> - ImageRecordCodeSection->CodeSegmentSize =3D Section[Index].SizeOfR=
> awData;
> + // We still need to align the VirtualSize to the SectionAlignment =
> because MSVC does not do
> + // this when creating a PE image. It expects the loader to do this=
> .
> + ImageRecordCodeSection->CodeSegmentSize =3D ALIGN_VALUE (Section[I=
> ndex].Misc.VirtualSize, SectionAlignment);
> =20
> InsertTailList (>CodeSegmentList, =
> tion->Link);
> ImageRecord->CodeSegmentCount++;
> --=20
> 2.40.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#116822): https://edk2.groups.io/g/devel/message/116822
Mute This Topic: https://groups.io/mt/104873193/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel][PATCH v1 1/3] MdeModulePkg: ImagePropertiesRecordLib: Use SectionAlignment for CodeSize

[Marvin Häuser]

> On 15. Mar 2024, at 23:57, Oliver Smith-Denny  
> wrote:
> 
> I don't think this is what I'm saying. What I am trying to say is that
> on MSVC, I see PE images getting created that have VirtualSize set to
> the actual number of initialized bytes in that section (not padded to
> the section alignment). On ElfConverted binaries, I see the VirtualSize
> is padded to the section alignment. I've dropped an example below

Ah, mismatched terminology. Zero-initialized as Ard and I used it refers to 
implicitly or explicitly 0-initialized global variables and such, which is not 
stored in the file, not the padding. So when you mentioned “real data”, I 
assumed you meant strictly the non-0 data from the file. Same misunderstanding 
with SizeOfImage, so that’s all fine. Whew. :)

> No, the specific case where I was researching this was explicitly
> setting /ALIGN:0x1 and /FILEALIGN:0x1000 for DXE_RUNTIME_DRIVERs
> on ARM64 (a UEFI spec requirement). So I would see the SizeOfRawData
> is aligned to the file alignment, as expected, but VirtualSize would
> be the actual size of the data. Again, the troubling thing here for
> me is that the same binary built with gcc has the VirtualSize aligned
> to the section alignment. And I have seen other code that loads PE
> images that relies on VirtualSize not including the padding. The spec
> is vague here, it says VirtualSize is the size of the section as
> loaded in memory (which would lead me to believe this should include
> padding) but it does not explicitly say it should be a multiple of
> the section alignment (as other fields do). But at a minimum I think
> we should have different toolchains doing the same behavior here.

Well, not rounding to pad is somewhat superior in some scenarios. If you round, 
you lose the information on what is section data and what is padding, so you 
might end up treating padding as data for some reason (because it is 
indistinguishable from mentioned 0-initialized data). This shouldn’t matter too 
much for executables and libraries, but MSVC/PE have a lot less of a 
distinction between object file and executable/library concepts (e.g. no 
distinction between sections and segments). That might be why they do it this 
way.

> See below for the VirtualSize examples, I'm confused on your comment on
> SizeOfImage. I agree that SizeOfImage covers everything as loaded into
> memory and I have not seen any issues there.

See first comment.

> Do you mind adding your RB to v2? And certainly if you have any other
> comments that is greatly appreciated.

Will try to remember tomorrow. :)

> Examples of the differences I see between MSVC and gcc binaries:
> 
> I originally noticed this on ARM64 on edk2, but wanted to make sure I
> saw it on x64 too, so this is with binaries from Project Mu's QemuQ35Pkg
> (edk2 doesn't have VS2022 support and I didn't feel like adding it
> or reverting back to VS2019). For reference, this is building the
> current top of tree at a4dd5d9785f48302c95fec852338d6bd26dd961a.
> 
> I dumped ReportStatusCodeRouterRuntimeDxe.efi from both using dumpbin
> (from VS2022) to examine the PE headers.
> 
> MSVC selected header values:
> 
> Main header:
> 0x3200 size of code
> 0x2400 size of initialized data
> 0x0size of uninitialized data
> 0x1000 section alignment
> 0x200  file alignment
> 0xB000 size of image
> 
> 6 sections: .data, .pdata, .rdata, .reloc, .text, .xdata
> 
> .text section:
> 0x30DF virtual size
> 0x3200 size of raw data
> 
> .data section:
> 0x130 virtual size
> 0x200 size of raw data
> 
> 
> GCC ElfConverted selected header values:
> 
> Main header:
> 0x4000 size of code
> 0x1000 size of initialized data
> 0x0size of uninitialized data
> 0x1000 section alignment
> 0x1000 file alignment
> 0x7000 size of image
> 
> 3 sections: .data, .text, .reloc
> 
> .text section:
> 0x4000 virtual size
> 0x4000 size of raw data
> 
> .data section:
> 0x1000 virtual size
> 0x1000 size of raw data
> 
> So my concern here is that ElfConvert takes a
> different view of VirtualSize, that is should be
> section aligned, whereas MSVC binaries take
> VirtualSize to be the actual size without padding.
> I think the correct thing to do would be change
> ElfConvert to do what MSVC does (although the spec
> is vague and so I can understand the confusion).

I don’t think it really matters, but it wouldn’t hurt either. Both kinds of 
binaries are in the wild, so you cannot really leverage any of the choices’ 
advantages either way. Adjusting to MSVC’s behaviour would be right though, as 
you can at least properly distinguish between padding and 0-data with new 
binaries.

> In practice this will tend to work as is, since we
> are using SizeOfImage to allocate with, but as you
> have pointed out there are so many edge cases that
> having a difference here makes me worried we would
> see something weird with only binaries built by one toolchain.

There is plenty of room for that as-is, including that MSVC emits .rdata (and 
others), 

Re: [edk2-devel][PATCH v1 1/3] MdeModulePkg: ImagePropertiesRecordLib: Use SectionAlignment for CodeSize

[Marvin Häuser]

> On 14. Mar 2024, at 15:45, Oliver Smith-Denny  
> wrote:
> 
> This does not appear to be the case with MSVC built binaries. I am
> seeing that VirtualSize is the size of the data-initialized part of the
> section.

What? :( So you are saying modern MSVC generates PEs that either have 
non-contiguous sections, or they have sections that contain zero-initialized 
portions explicitly? Both variants sound badly borked and should be 
investigated, even if only for resource constraints.

Just thinking about it, is there any chance you set ALIGN = FILEALIGN? Because 
then this will be invoked and your file will be converted to XIP: 
https://github.com/tianocore/edk2/blob/5572b43c6767f7cc46b074ae1fc288f6eccdc65d/BaseTools/Source/C/GenFw/GenFw.c#L612

BaseTools has no concept of what should be a XIP and what should not, so if a 
file somewhat qualifies, it will just always emit a XIP: 
https://github.com/tianocore/edk2/blob/5572b43c6767f7cc46b074ae1fc288f6eccdc65d/BaseTools/Source/C/GenFw/GenFw.c#L2150

In AUDK I addressed this by tying XIP to reloc stripping, but that is more of a 
convenient heuristic: 
https://github.com/acidanthera/audk/blob/13f99fa7cbde0829ba98e1e4f2a03891a0791ac6/BaseTools/Source/Python/GenFds/EfiSection.py#L293

> Whereas on our ElfConverted binaries, what you say is true. The
> difference concerns me.

Yes, very concerning. Would you mind sharing dumps (section table, etc.) and/or 
binaries (unless you indeed hit what I outlined above)? I recall MSVC to rather 
be more aggressive with space-savings, whereas ElfConvert would include 
section-alignment padding within the file (but not zero-init data).

> Agreed. I have done further research and v2 uses VirtualSize. I was
> initially led astray here because the ElfConvert tool sets VirtualSize
> and SizeOfRawData to the SectionAlignment universally:
> 
> https://github.com/tianocore/edk2/blob/5572b43c6767f7cc46b074ae1fc288f6eccdc65d/BaseTools/Source/C/GenFw/ElfConvert.c#L134-L136

Yes… plenty of more things in the tool to lead people astray. :)

> Yep, the PE headers are definitely filled with information that can be
> deduced from other sources and the spec is not very verbose, which is
> why I definitely want to tread carefully here. For example, it says
> SizeOfImage is the size of the image loaded into memory, which
> current implementations seem to take to mean the sum of the headers
> plus each section's VirtualSize rounded to the SectionAlignment (which
> the spec does call out that SizeOfImage must be aligned to the
> SectionAlignment). VirtualSize it says is also the size of the section
> as loaded into memory, but current implementations (other than our
> ElfConvert script which I'm currently investigating if we should
> change that) seem to take this to mean the size of the initialized
> data in memory, i.e. the "real" data from the image, not the zeroed
> data to get to the SectionAlignment.

Do you have an example for that? I’m quite certain SizeOfImage should always 
cover the whole thing.

> But the language in the spec
> is almost identical between the two. And again, we have a difference
> in our MSVC and gcc built binaries here.

Do we? :(

> I agree with you there. That being said, the current Image record code
> is, unsurprisingly, broken, and does need to be fixed. Can you review
> the v2 of this? The only difference is that I changed to VirtualSize.

LGTM.

> Do you think the code should be updated for the case of VirtualSize
> == 0? In the current implementation, it will round 0 up to the
> SectionAlignment

It should not. Well, it will technically round, but it will remain 0. It does 
not round up to the strictly next boundary, but if the current value is a 
multiple of the alignment (0 is a multiple of any alignment), it remains 
unchanged.

> , which is what I believe will happen in the PE
> loader, also, but want to get your thoughts on it.
> 
> This would still be a short term solution to fix the active bug
> (ARM64 cannot boot a 64k pagesize Linux with a broken MAT table or
> we will boot without the benefit of the MAT, if we happen to have
> an aligned total region but not each entry, testing shows). I do
> think it is a broken pattern to try to determine the image records
> after we've loaded them into memory, the PE loader should create
> them with the factual data of what was loaded.

There are many more broken patterns everywhere. Did you notice yet that the API 
itself is broken from the ground up? Not to keep “advertising” it, but maybe 
our paper will give you some fun too. :) https://arxiv.org/pdf/2012.05471.pdf

Best regards,
Marvin

> 
> Thanks,
> Oliver



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#116780): https://edk2.groups.io/g/devel/message/116780
Mute This Topic: https://groups.io/mt/104610770/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]

Re: [edk2-devel][PATCH v1 1/3] MdeModulePkg: ImagePropertiesRecordLib: Use SectionAlignment for CodeSize

[Marvin Häuser]

Good day everyone,

Sorry for interjecting, but I’d like to avoid premature changes to PE code that 
would only make the current mess even worse.

> On 4. Mar 2024, at 20:24, Oliver Smith-Denny  wrote:
> 
> I relooked at the spec, you are correct, SizeOfRawData is aligned to
> the FileAlignment. So this is only a bug for a file with
> SectionAlignment != FileAlignment.

Still no, for some reasons you already mentioned, but also reasons you didn’t:
- VirtualSize often is not aligned, so they can still trivially mismatch.
- VirtualSize includes zero-initialized data not part of the file.
- In the old days, VirtualSize = 0 was used to disable loading of debug data, 
but SizeOfRawData remained intact, so it could be loaded on demand.
- SizeOfRawData may be unaligned due to bugs. In fact, there is no reason to 
trust FileAlignment, it has no real meaning in an UEFI context (I am not even 
sure it does on Windows).

> I see, yes I do believe VirtualSize could be used here instead.

*Must*. As Ard said, SizeOfRawData is only interesting to know how many bytes 
to copy from the file and for *nothing* else.

> Two
> things give me pause. One is that the PE spec states that SizeOfRawData
> is rounded and VirtualSize is not, so that SizeOfRawData may be greater
> than the VirtualSize in some cases (which seems incorrect).
> 
> The other is that the image loader partially uses VirtualSize when
> loading:
> 
> https://github.com/tianocore/edk2/blob/918288ab5a7c3abe9c58d576ccc0ae32e2c7dea0/MdePkg/Library/BasePeCoffLib/BasePeCoff.c#L1399-L1400
> 
> However, when determining the size of a loaded image (and therefore the
> number of pages to allocate) it will allocate an extra page:
> 
> https://github.com/tianocore/edk2/blob/918288ab5a7c3abe9c58d576ccc0ae32e2c7dea0/MdeModulePkg/Core/Dxe/Image/Image.c#L646-L652
> 

Sorry, I don’t understand the “however”, this is sound.

> as ImageSize here is:
> 
> https://github.com/tianocore/edk2/blob/918288ab5a7c3abe9c58d576ccc0ae32e2c7dea0/MdePkg/Library/BasePeCoffLib/BasePeCoff.c#L312
> 
> Which according to the spec, SizeOfImage is the size of the image as
> loaded into memory and must be a multiple of section alignment.

Side note, SizeOfImage may be borked, not only due to bugs, but also due to 
malice. There is literally no reason whatsoever to use it - to validate it, you 
need to calculate a sane value, and once you have that, you obviously do not 
need SizeOfImage anymore. You could reject images with malformed SizeOfImage, 
but there likely are various legitimate-bugged ones out there. This is just an 
ancient artifact from before memory safety was a thing. :)

> So, reflecting on this, let me test with VirtualSize here, I think
> that is the right value to use, the only time we would have a
> SizeOfRawData that is greater than the VirtualSize would be if our
> FileAlignment is greater than our SectionAlignment, which would be
> a misconfiguration.

No, it can simply be that FileAlignment = SectionAlignment and SizeOfRawData is 
aligned and VirtualSize is not. Even when both are aligned, the former may 
carry extra optional data (like previously mentioned debug data), or it might 
be some weird I/O performance related padding or something.

> I do think the ImageLoader should also be fixed to only allocate
> ImageSize number of pages (which should be the sum of the section
> VirtualSizes + any headers that aren't included). Might as well not
> waste an extra page for each image and then our image record code is
> simpler as well (ensuring we are protecting the right pages).

NO! :) There are *many* issues surrounding this:
- Sections may overlap (observed with early macOS EFI images, could be ignored, 
but needs to be validated across the landscape).
- Sections may have gaps (observed with “old" Linux EFI Shims iirc, there’ll be 
plenty of broken images in the wild yet).
- The extra page is not pointless - as you can see, it is allocated when 
exceeding EFI page size, which is the only granularity you can page-allocate 
at. So, when you require > 4K alignment, you need the extra page to be able to 
“shift” the image into proper alignment in the allocated region as needed. In 
AUDK, I resolved this by abstracting existing, buried aligned-allocation code 
into a MemoryAllocationLibEx class: 
https://github.com/acidanthera/audk/blob/13f99fa7cbde0829ba98e1e4f2a03891a0791ac6/MdeModulePkg/Library/CommonMemoryAllocationLib/CommonMemoryAllocationLibEx.c#L70
Without aligned-allocation, you cannot get rid of this extra page.

For reference, my image record code is here: 
https://github.com/acidanthera/audk/blob/13f99fa7cbde0829ba98e1e4f2a03891a0791ac6/MdePkg/Library/BaseUefiImageLib/PeSupport.c#L335

> I think this patch series should go in, as it is fixing an active bug,
> but I will also take a look at the image loader creating the image
> records and having a protocol it produces to retrieve the list, to
> attempt to avoid issues like this in the future.

Why does there need to be 

Re: [edk2-devel] [PATCH] Ext4Pkg: Fix CRC16 checksumming on block groups

[Marvin Häuser]

Reviewed-by: Marvin Häuser 

> On Dec 3, 2023, at 23:41, Pedro Falcato  wrote:
> 
> Old filesystems (around 2008 and older) do not use CRC32c
> but rather CRC16-ANSI. Previously, the CalculateCrc16Ansi function was
> broken and gave us wrong checksums. Adapt to the new interface.
> 
> And while we're at it, fix the checksum algorithm itself - the crc16
> algorithm just skips over the bg_checksum, and does not checksum it.
> 
> This problem was found out-of-list when older ext4 filesystems
> (that use crc16 checksums) failed to mount with "corruption".
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4609
> Signed-off-by: Pedro Falcato 
> Cc: Savva Mitrofanov 
> Cc: Marvin Häuser 
> ---
> Features/Ext4Pkg/Ext4Dxe/BlockGroup.c | 6 +-
> 1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/BlockGroup.c 
> b/Features/Ext4Pkg/Ext4Dxe/BlockGroup.c
> index f34cdc5dbad7..d5642a5f155c 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/BlockGroup.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/BlockGroup.c
> @@ -169,14 +169,10 @@ Ext4CalculateBlockGroupDescChecksumGdtCsum (
>   )
> {
>   UINT16  Csum;
> -  UINT16  Dummy;
> 
> -  Dummy = 0;
> -
> -  Csum = CalculateCrc16Ansi (Partition->SuperBlock.s_uuid, 16, 0);
> +  Csum = CalculateCrc16Ansi (Partition->SuperBlock.s_uuid, 16, 
> CRC16ANSI_INIT);
>   Csum = CalculateCrc16Ansi (, sizeof (BlockGroupNum), Csum);
>   Csum = CalculateCrc16Ansi (BlockGroupDesc, OFFSET_OF 
> (EXT4_BLOCK_GROUP_DESC, bg_checksum), Csum);
> -  Csum = CalculateCrc16Ansi (, sizeof (Dummy), Csum);
>   Csum =
> CalculateCrc16Ansi (
>   >bg_block_bitmap_hi,
> --
> 2.43.0
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#111998): https://edk2.groups.io/g/devel/message/111998
Mute This Topic: https://groups.io/mt/102960519/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-




smime.p7s
Description: S/MIME cryptographic signature

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Marvin Häuser]

> On Sep 29, 2023, at 12:58, Michael Brown  wrote:
> 
> On 29/09/2023 10:47, Marvin Häuser wrote:
>> Maybe when Linux starts adhering the spec for file names (the spec clearly 
>> defines e.g. BOOTx64.EFI, while at least some distros/images use 
>> bootx64.efi), this can be discussed. :) Let's not break various GRUB 
>> setups...
> 
> Seems slightly unfair to blame Linux when EDK2 also apparently gets it wrong 
> (in a different way)... :)

I’m perfectly happy with any blame cast on Linux and EDK II, preferably both. 
:) Seems like this is my lucky day…

> 
> //
> // EFI File location to boot from on removable media devices
> //
> ...
> #define EFI_REMOVABLE_MEDIA_FILE_NAME_X64 L"\\EFI\\BOOT\\BOOTX64.EFI"
> 
> Is this something worth fixing in UefiSpec.h?

At this point I’m never sure whether to fix EDK II or fix the spec… I wouldn’t 
make any sudden moves without checking OS behaviours.

Best regards,
Marvin

> 
> Thanks,
> 
> Michael
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109195): https://edk2.groups.io/g/devel/message/109195
Mute This Topic: https://groups.io/mt/101615699/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Marvin Häuser]

> On Sep 28, 2023, at 19:57, Pedro Falcato  wrote:
> 
> On Wed, Sep 27, 2023 at 1:09 PM Gerd Hoffmann  wrote:
>> 
>>  Hi,
>> 
>> I've noticed that the edk2 ext4 driver does case-insensitive filename
>> matching.  I know the fat filesystem is case-insensitive, and the uefi
>> spec describing the fat filesystem also explicitly says it is
>> case-insensitive.  On a quick scan I can't find anything in the uefi
>> spec requiring *all* filesystem drivers being case-insensitive though.
>> 
>> So I'm wondering whenever the ext4 driver behavior is correct.  It
>> certainly is different than the linux kernel's behavior which is (by
>> default) case-sensitive.

Maybe when Linux starts adhering the spec for file names (the spec clearly 
defines e.g. BOOTx64.EFI, while at least some distros/images use bootx64.efi), 
this can be discussed. :) Let's not break various GRUB setups...

> 
> Ok, so this is a funny problem. When I was originally writing it, I
> noticed a good few consumers would expect case insensitiveness, so I
> ended up needing case insensitive string comparison.
> 
> As an example, in my current system:
> Boot0001* ARCHLINUX
> HD(1,GPT,7e1b2e20-3013-4683-b8ab-bef1f9bfb1c8,0x800,0x32000)/File(\EFI\ARCHLINUX\GRUBX64.EFI)
> whereas the GRUB tooling installed itself as:
> /boot/efi/EFI/ARCHLINUX/grubx64.efi
> 
> I treated some good bits of the filesystem driver as "should remain
> similar to FAT" due to compatibility reasons.

Definitely must keep it that way. I think true user-facing case-sensitivity is 
mostly a Linux thing, even recent macOS APFS is still only case-preserving 
(UX-wise). There certainly is no real use-case for UEFI itself beyond things 
like Linux interoperability.

> I really really wouldn't
> be surprised if lots of EFI code out there relied on case
> insensitiveness or other FAT-ish stuff, instead of UNIX semantics.
> (funnily enough, EFI also has the restriction where every filename
> must be valid unicode, which is not the case on most UNIX systems out
> there, that take the file name as "bag of bytes - '/'")
> 
>> 
>> Also note that the linux kernel ext4 driver recently got support for
>> case-insensitive file names, which must be explicitly enabled for both
>> filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
>> (EXT4_CASEFOLD_FL).
> 
> Ugh, this is annoying, they didn't even bother documenting it...
> (https://www.kernel.org/doc/html/latest/filesystems/ext4/globals.html#super-incompat)
> This leaves me in the awkward spot where figuring the behavior out
> would require me to read the fs/ext4 code and thus be legally dubious,
> yay :^)

All hail GPL...

> 
>> 
>> On practical terms I've ran into actual problems due to Fedora mounting
>> the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
>> placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
>> translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
>> So I have both /boot/efi and /boot/EFI ...
> 
> Oh boy, that seems fun. So Ext4Dxe can only open one of the two dirs, right?
> Now that I think of it, there should be fun behavior when doing an EFI
> readdir, where you could find two dirents with "different" names but
> then opening both will lead to the same dirent being open twice...
> yuck

Yes, but this often is not a concern due to case-preservation. There also isn't 
much of a "right or wrong", as supporting case-insensitivity here could change 
the semantics of existing desync'd sensitivity setups. However, 
case-insensitivity would at least be more predictable.

The only real options are:

1) We assume the host OS understands that UEFI is inherently case-sensitive and 
doesn't do weird things, then the current behaviour makes sense. It's 
efficient, doesn't touch unnecessary data, but it's not so predictable.

2) We assume the host OS understands this, but we consider this a risk to 
security. Then it might make sense to reject FSes that feature this. Obviously 
Secure Boot and such should be taking care of only invoking trusted binaries, 
but someone might be trying to invoke ambiguous FS driver behaviour because 
they are aware of an exploit vector. Nevertheless, this seems unnecessary and 
overkill, also it doesn't fix the broken UX. It will actually completely break 
that Linux use-case, which might be a bug or a feature. :) But it's predictable!

3) We assume the host OS does not do its job, but we are being nice anyway. 
Then it might make sense to have opportunistic case-sensitivity, where equality 
is preferred, but insensitive-equality is used as a fallback. This fixes the 
issue when the OS is at least consistent in its insensitivity-violation. 
However, as Pedro showed an example of a desync'd device path and FS name, 
there is yet another loophole to (accidentally) abuse the ambiguity. Not even 
some obviously crazy stuff like ranking by editing distances can save you here. 
And it's also limited in predictability when there is no exact match.

I 

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

Hi Ray,This is handled by GenFw. For ELF-based toolchains, this is done as part of the conversion [1]. For others, a “GenFw post-processing” step takes care of it [2]. Though you are right, this is not obvious and external tools may forget this step (but still produce valid XIP-alike PEs). Consequently, I guess I’ll recommend against my former suggestion to check FileAlign == SectAlign and to instead just check the actual section info [3].While this looks safe for now, GenFv would be a much more logical place for XIP conversion. This way, everything is XIP and when page-aligning all images, the flash storage quickly runs out of space (at least without compression). That GenFv has no logic to grow image files also kept me from implementing a new 2-section approach to XIP where metadata and sections are stored separately, such that the header metadata doesn’t need to be section-aligned.(Completely irrelevant to this discussion, but while we’re at it, the flat storage hierarchy also makes memory protection for XIP a pain, as there will be FFS metadata inbetween 4K-aligned (or maybe even more in the future?) images, thus virtually always yielding padding. If all images were stored consecutively and the FFS metadata rather pointed to their locations, a lot if padding could be avoided. Though this would need some analysis as to how much you actually save, as adding an offset variable will obviously grow the FFS overhead. It would also be nice to somehow encode the offset implicitly, where the first one is explicitly provided and the rest is just adding up the size of the previous one, but that might screw over some existing parsers.)Best regards,Marvin[1] https://github.com/tianocore/edk2/blob/d8e5d35ede7158ccbb9abf600e65b9aa6e043f74/BaseTools/Source/C/GenFw/Elf64Convert.c#L1297[2] https://github.com/tianocore/edk2/blob/d8e5d35ede7158ccbb9abf600e65b9aa6e043f74/BaseTools/Source/C/GenFw/GenFw.c#L2088[3] https://github.com/acidanthera/audk/blob/d9bb10ae3b73134eb434b309cb0db3fa4282a838/MdePkg/Library/BasePeCoffLib2/PeCoffLoad.c#L134On 31. May 2023, at 09:14, Ni, Ray  wrote:







When a PE has a global large variable, there could be a .bss section whose actual size is 0 or very small but the eventual size when loading to memory will be larger.
Can XIP work with this section? 
@Liu, Zhiguang brought this question when discussing with me offline
 
Thanks,
Ray
 


From: devel@edk2.groups.io  
On Behalf Of Marvin Häuser
Sent: Tuesday, May 30, 2023 6:25 PM
To: Ard Biesheuvel 
Cc: edk2-devel-groups-io ; Ni, Ray 
Subject: Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers


 
 



On 30. May 2023, at 12:02, Ard Biesheuvel <a...@kernel.org> wrote:

 

On Tue, 30 May 2023 at 11:53, Marvin Häuser <mhaeu...@posteo.de>
 wrote:






On 30. May 2023, at 11:48, Ard Biesheuvel <a...@kernel.org> wrote:

On Tue, 30 May 2023 at 11:47, Ard Biesheuvel <a...@kernel.org> wrote:


On Tue, 30 May 2023 at 11:42, Marvin Häuser <mhaeu...@posteo.de> wrote:


I took a *very brief* look at the entire series now. Is this just to apply permissions before CpuDxe is loaded


Yes.


Well, actually, the first part of the series gets rid of some
pointless memory copies, which is an improvement in itself.


Sorry, I didn't mean the series, I meant the choice to handle things in DxeIpl over DxeCore in particular.

Is there even a non-FOSS producer of the CpuArch protocol? To my understanding, all (common) Intel platforms use the one in UefiCpuPkg. ARM and friends feel a lot less proprietary to begin with, but I could be wrong. We were quite successful so far with just
 merging CpuArch into DxeCore and setting it up quite early, but of course not at an industry-wide scale. :)


What about the dependencies of CpuArch protocol? On ARM, this is the
GIC driver (for the interrupt controller), which has its own platform
specific dependencies.



 


Hmm, was that for the exception handler? I forgot that was in CpuDxe too, I specifically meant the memory permission related things, sorry!







So this is not tractable in general, and the only options we have (imo) are

- add memory permission attribute handling to DxeCore directly (via a
library with arch specific implementations)



 


Yes, this.






- add a way to dispatch the CpuDxe *and its dependencies* without the
need to manipulate memory permissions



 


That would be awful and I'd prefer your current solution over this.







Clumping everything together into DxeCore does not appear to me as a
sustainable approach for this.



 









_._,_._,_



Groups.io Links:


  
You receive all messages sent to this group.
  
  



View/Reply Online (#105486) |


  

|

  Mute This Topic


| New Topic






Your Subscription |
Contact Group Owner |

Unsubscribe

 [arch...@mail-archive.com]
_._,_._,_

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

> On 30. May 2023, at 12:02, Ard Biesheuvel  wrote:
> 
> On Tue, 30 May 2023 at 11:53, Marvin Häuser  <mailto:mhaeu...@posteo.de>> wrote:
>> 
>> 
>> 
>> On 30. May 2023, at 11:48, Ard Biesheuvel  wrote:
>> 
>> On Tue, 30 May 2023 at 11:47, Ard Biesheuvel  wrote:
>> 
>> 
>> On Tue, 30 May 2023 at 11:42, Marvin Häuser  wrote:
>> 
>> 
>> I took a *very brief* look at the entire series now. Is this just to apply 
>> permissions before CpuDxe is loaded
>> 
>> 
>> Yes.
>> 
>> 
>> Well, actually, the first part of the series gets rid of some
>> pointless memory copies, which is an improvement in itself.
>> 
>> 
>> Sorry, I didn't mean the series, I meant the choice to handle things in 
>> DxeIpl over DxeCore in particular.
>> 
>> Is there even a non-FOSS producer of the CpuArch protocol? To my 
>> understanding, all (common) Intel platforms use the one in UefiCpuPkg. ARM 
>> and friends feel a lot less proprietary to begin with, but I could be wrong. 
>> We were quite successful so far with just merging CpuArch into DxeCore and 
>> setting it up quite early, but of course not at an industry-wide scale. :)
> 
> What about the dependencies of CpuArch protocol? On ARM, this is the
> GIC driver (for the interrupt controller), which has its own platform
> specific dependencies.

Hmm, was that for the exception handler? I forgot that was in CpuDxe too, I 
specifically meant the memory permission related things, sorry!

> 
> So this is not tractable in general, and the only options we have (imo) are
> 
> - add memory permission attribute handling to DxeCore directly (via a
> library with arch specific implementations)

Yes, this.

> - add a way to dispatch the CpuDxe *and its dependencies* without the
> need to manipulate memory permissions

That would be awful and I'd prefer your current solution over this.

> 
> Clumping everything together into DxeCore does not appear to me as a
> sustainable approach for this.



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#105447): https://edk2.groups.io/g/devel/message/105447
Mute This Topic: https://groups.io/mt/99197142/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

> On 30. May 2023, at 11:48, Ard Biesheuvel  wrote:
> 
> On Tue, 30 May 2023 at 11:47, Ard Biesheuvel  <mailto:a...@kernel.org>> wrote:
>> 
>> On Tue, 30 May 2023 at 11:42, Marvin Häuser  wrote:
>>> 
>>> 
>>>> On 30. May 2023, at 11:38, Ard Biesheuvel  wrote:
>>>> 
>>>> On Tue, 30 May 2023 at 11:07, Marvin Häuser  wrote:
>>>>> 
>>>>> Hi Ard,
>>>>> 
>>>>> Native PE toolchains *generally* also generate XIP images (/ALIGN = 
>>>>> /FILEALIGN, but with 32 Byte rather than 64 Byte alignment compared to 
>>>>> GCC49+ / CLANGDWARF) [1]. However, because they are underaligned by 
>>>>> default (even for RT images that run in an OS context and MM drivers... 
>>>>> sigh...), platforms manually override SectionAlignment, but not 
>>>>> necessarily FileAlignment [2], breaking XIP. I don't think what you are 
>>>>> doing is perfectly safe for these, as they will have FileAlignment < 
>>>>> SectionAlignment (and by all chances, BaseTools is borked too). In my 
>>>>> opinion check for FileAlignment == SectionAlignment. I can't vouch for 
>>>>> how likely FileAlignment has a sane value, the AUDK loader does not read 
>>>>> it at all and instead checks PointerToRawData == VirtualAddress, etc.
>>>>> 
>>>>> BaseTools generally has poor support for non-XIP vs XIP, probably due to 
>>>>> notorious underalignment since the very beginning. For PEIM XIPs for 
>>>>> example, which must ship pre-relocated at least for Intel, GenFv just 
>>>>> relocates the image in-memory and then copies the changes back to the FFS 
>>>>> file [3]. There is no concept of changing the image file size within the 
>>>>> procedure and as such, a non-XIP image cannot be converted to XIP on 
>>>>> demand. This would be useful for a distinction between pre-memory and 
>>>>> post-memory PEIMs, the former of which must be XIP (thus aligned), while 
>>>>> the latter can be loaded and relocated in-RAM (thus can be underaligned 
>>>>> w.r.t. FileAlignment), but alas.
>>>>> 
>>>> 
>>>> If XIP for PE images with 4k section alignment is an issue, we could
>>>> always explore loading them into a separate allocation from PEI, just
>>>> like we do with DXE core itself.
>>>> 
>>>> This would actually simplify the loader code quite a lot, as we'd be
>>>> able to use the PEI core image loader directly. However, it means we'd
>>>> have to pass this information (array of  tuples
>>>> describing which images were already loaded by DxeIpl) via a HOB or
>>>> some other method.
>>> 
>>> I took a *very brief* look at the entire series now. Is this just to apply 
>>> permissions before CpuDxe is loaded
>> 
>> Yes.
>> 
> 
> Well, actually, the first part of the series gets rid of some
> pointless memory copies, which is an improvement in itself.

Sorry, I didn't mean the series, I meant the choice to handle things in DxeIpl 
over DxeCore in particular.

Is there even a non-FOSS producer of the CpuArch protocol? To my understanding, 
all (common) Intel platforms use the one in UefiCpuPkg. ARM and friends feel a 
lot less proprietary to begin with, but I could be wrong. We were quite 
successful so far with just merging CpuArch into DxeCore and setting it up 
quite early, but of course not at an industry-wide scale. :)

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#105444): https://edk2.groups.io/g/devel/message/105444
Mute This Topic: https://groups.io/mt/99197142/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

> On 30. May 2023, at 11:38, Ard Biesheuvel  wrote:
> 
> On Tue, 30 May 2023 at 11:07, Marvin Häuser  wrote:
>> 
>> Hi Ard,
>> 
>> Native PE toolchains *generally* also generate XIP images (/ALIGN = 
>> /FILEALIGN, but with 32 Byte rather than 64 Byte alignment compared to 
>> GCC49+ / CLANGDWARF) [1]. However, because they are underaligned by default 
>> (even for RT images that run in an OS context and MM drivers... sigh...), 
>> platforms manually override SectionAlignment, but not necessarily 
>> FileAlignment [2], breaking XIP. I don't think what you are doing is 
>> perfectly safe for these, as they will have FileAlignment < SectionAlignment 
>> (and by all chances, BaseTools is borked too). In my opinion check for 
>> FileAlignment == SectionAlignment. I can't vouch for how likely 
>> FileAlignment has a sane value, the AUDK loader does not read it at all and 
>> instead checks PointerToRawData == VirtualAddress, etc.
>> 
>> BaseTools generally has poor support for non-XIP vs XIP, probably due to 
>> notorious underalignment since the very beginning. For PEIM XIPs for 
>> example, which must ship pre-relocated at least for Intel, GenFv just 
>> relocates the image in-memory and then copies the changes back to the FFS 
>> file [3]. There is no concept of changing the image file size within the 
>> procedure and as such, a non-XIP image cannot be converted to XIP on demand. 
>> This would be useful for a distinction between pre-memory and post-memory 
>> PEIMs, the former of which must be XIP (thus aligned), while the latter can 
>> be loaded and relocated in-RAM (thus can be underaligned w.r.t. 
>> FileAlignment), but alas.
>> 
> 
> If XIP for PE images with 4k section alignment is an issue, we could
> always explore loading them into a separate allocation from PEI, just
> like we do with DXE core itself.
> 
> This would actually simplify the loader code quite a lot, as we'd be
> able to use the PEI core image loader directly. However, it means we'd
> have to pass this information (array of  tuples
> describing which images were already loaded by DxeIpl) via a HOB or
> some other method.

I took a *very brief* look at the entire series now. Is this just to apply 
permissions before CpuDxe is loaded or is there another reason this is not 
handled by DxeCore itself?

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#105440): https://edk2.groups.io/g/devel/message/105440
Mute This Topic: https://groups.io/mt/99197142/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

> On 30. May 2023, at 11:06, Marvin Häuser  wrote:
> 
> Hi Ard,
> 
> Native PE toolchains *generally* also generate XIP images (/ALIGN = 
> /FILEALIGN, but with 32 Byte rather than 64 Byte alignment compared to GCC49+ 
> / CLANGDWARF) [1]. However, because they are underaligned by default (even 
> for RT images that run in an OS context and MM drivers... sigh...), platforms 
> manually override SectionAlignment, but not necessarily FileAlignment [2], 
> breaking XIP. I don't think what you are doing is perfectly safe for these, 
> as they will have FileAlignment < SectionAlignment (and by all chances, 
> BaseTools is borked too). In my opinion check for FileAlignment == 
> SectionAlignment. I can't vouch for how likely FileAlignment has a sane 
> value, the AUDK loader does not read it at all and instead checks 
> PointerToRawData == VirtualAddress, etc.
> 
> BaseTools generally has poor support for non-XIP vs XIP, probably due to 
> notorious underalignment since the very beginning. For PEIM XIPs for example, 
> which must ship pre-relocated at least for Intel, GenFv just relocates the 
> image in-memory and then copies the changes back to the FFS file [3]. There 
> is no concept of changing the image file size within the procedure and as 
> such, a non-XIP image cannot be converted to XIP on demand. This would be 
> useful for a distinction between pre-memory and post-memory PEIMs, the former 
> of which must be XIP (thus aligned), while the latter can be loaded and 
> relocated in-RAM (thus can be underaligned w.r.t. FileAlignment), but alas.
> 
> Are there any docs w.r.t. the use-case for this? Since when are DXE XIPs a 
> thing? The code above is opportunistic (i.e., failures are ignored), how does 
> the dispatcher take this into account? Why is this applied during PEI, how 
> this this work with the notion of payloads?

Sorry, I just realized this was a patch *series*! :(
For the questions that are answered in commit messages or implicitly by the 
code, please disregard, I'll read through them. Not sure the payload situation 
is obvious, as it's not all that obvious to me how they work in the first place 
(though I do know, you can both load a payload from edk2 PEI and have edk2 DXE 
loaded as a payload from something else, so this definitely is a concern in 
some way).

> 
> Thanks!
> 
> Best regards,
> Marvin
> 
> [1] 
> https://github.com/tianocore/edk2/blob/0f9283429dd487deeeb264ee5670551d596fc208/BaseTools/Conf/tools_def.template#L670-L672
> 
> [2] 
> https://github.com/tianocore/edk2-platforms/blob/02fb8459d9c48a8ed4691e9c22f2516c15073835/Silicon/Intel/CoffeelakeSiliconPkg/SiPkgBuildOption.dsc#L129
> 
> [3] 
> https://github.com/tianocore/edk2/blob/0f9283429dd487deeeb264ee5670551d596fc208/BaseTools/Source/C/GenFv/GenFvInternalLib.c#L3881-L3897



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#105438): https://edk2.groups.io/g/devel/message/105438
Mute This Topic: https://groups.io/mt/99197142/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC PATCH 08/11] MdeModulePkg/DxeIpl: Relocate and remap XIP capable DXE drivers

[Marvin Häuser]

Hi Ard,

Native PE toolchains *generally* also generate XIP images (/ALIGN = /FILEALIGN, 
but with 32 Byte rather than 64 Byte alignment compared to GCC49+ / CLANGDWARF) 
[1]. However, because they are underaligned by default (even for RT images that 
run in an OS context and MM drivers... sigh...), platforms manually override 
SectionAlignment, but not necessarily FileAlignment [2], breaking XIP. I don't 
think what you are doing is perfectly safe for these, as they will have 
FileAlignment < SectionAlignment (and by all chances, BaseTools is borked too). 
In my opinion check for FileAlignment == SectionAlignment. I can't vouch for 
how likely FileAlignment has a sane value, the AUDK loader does not read it at 
all and instead checks PointerToRawData == VirtualAddress, etc.

BaseTools generally has poor support for non-XIP vs XIP, probably due to 
notorious underalignment since the very beginning. For PEIM XIPs for example, 
which must ship pre-relocated at least for Intel, GenFv just relocates the 
image in-memory and then copies the changes back to the FFS file [3]. There is 
no concept of changing the image file size within the procedure and as such, a 
non-XIP image cannot be converted to XIP on demand. This would be useful for a 
distinction between pre-memory and post-memory PEIMs, the former of which must 
be XIP (thus aligned), while the latter can be loaded and relocated in-RAM 
(thus can be underaligned w.r.t. FileAlignment), but alas.

Are there any docs w.r.t. the use-case for this? Since when are DXE XIPs a 
thing? The code above is opportunistic (i.e., failures are ignored), how does 
the dispatcher take this into account? Why is this applied during PEI, how this 
this work with the notion of payloads?

Thanks!

Best regards,
Marvin

[1] 
https://github.com/tianocore/edk2/blob/0f9283429dd487deeeb264ee5670551d596fc208/BaseTools/Conf/tools_def.template#L670-L672

[2] 
https://github.com/tianocore/edk2-platforms/blob/02fb8459d9c48a8ed4691e9c22f2516c15073835/Silicon/Intel/CoffeelakeSiliconPkg/SiPkgBuildOption.dsc#L129

[3] 
https://github.com/tianocore/edk2/blob/0f9283429dd487deeeb264ee5670551d596fc208/BaseTools/Source/C/GenFv/GenFvInternalLib.c#L3881-L3897


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#105437): https://edk2.groups.io/g/devel/message/105437
Mute This Topic: https://groups.io/mt/99197142/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2 1/1] MdePkg/Base.h: Simply alignment expressions

[Marvin Häuser]

> On 16. May 2023, at 09:18, Marvin Häuser  wrote:
> 
> 
> 
>> On 16. May 2023, at 04:22, Pedro Falcato  wrote:
>> 
>> On Tue, May 16, 2023 at 2:46 AM gaoliming via groups.io
>>  wrote:
>>> 
>>> Pedro:
>>> 
>>>> -邮件原件-
>>>> 发件人: devel@edk2.groups.io  代表 Pedro Falcato
>>>> 发送时间: 2023年5月15日 23:15
>>>> 收件人: devel@edk2.groups.io
>>>> 抄送: Pedro Falcato ; Michael D Kinney
>>>> ; Liming Gao ;
>>>> Zhiguang Liu ; Marvin Häuser
>>>> 
>>>> 主题: [edk2-devel] [PATCH v2 1/1] MdePkg/Base.h: Simply alignment
>>>> expressions
>>>> 
>>>> Simplify ALIGN_VALUE and ALIGN_VALUE_ADDEND into simpler expressions.
>>>> 
>>>> ALIGN_VALUE can simply be a (value + (align - 1)) & ~align
>>>> expression, which works for any power of 2 alignment and generates
>>>> smaller code sequences. For instance:
>>>> ALIGN_VALUE(15, 16) = (15 + 15) & ~16 = 16
>>>> ALIGN_VALUE(16, 16) = (16 + 15) & ~16 = 16
>>>> 
>>>> Old codegen:
>>>> movq%rdi, %rax
>>>> negq%rax
>>>> andl$15, %eax
>>>> addq%rdi, %rax
>>>> 
>>>> New codegen:
>>>> leaq15(%rdi), %rax
>>>> andq$-16, %rax
>>>> 
>>>> ALIGN_VALUE_ADDEND can simply use a bitwise NOT of Value to get the
>>>> addend for alignment, as, for instance:
>>>> ~15 & (16 - 1) = 1
>>>> 15 + 1 = 16
>>>> 
>>> 
>>>> ~15 & (16 - 1) = 1
>>> Its value should be zero, not 1. I also verify the updated 
>>> ALIGN_VALUE_ADDEND.
>>> Its value is incorrect. Please double check.
>> 
>> Hi Liming, you're 100% right. There was a mixup when we were
>> discussing this optimization, and I got the mental calculations wrong
>> there.
>> Two's complement is definitely what we want, as one's complement is
>> always off by one (from what we want).
>> 
>> So negation (-) works beautifully, as seen in the old codegen (we
>> figured this out from the compiler's output).
> 
> To be clear on the maths side of things:
> 
> “& (Alignment - 1U)” is equivalent to “mod Alignment” for powers of two.
> “-Value” is equivalent to “2^N - Value” once the expression is promoted to an 
> unsigned type, where N is the precision of said type.
> 
> So, the old expression basically was “(Alignment - Value) mod Alignment” and 
> the new expression is “(2^N - Value) mod Alignment”. By modulo laws, we can 
> apply the mod to the operands, which for the left ones gives us “Alignment 
> mod Alignment = 0” and “2^N mod Alignment = 0”, obviously for Alignment being 
> a power of two. They’re trivially equivalent.

Sorry, I have to add a correction here. This assumes that 2^N >= Alignment, 
which somehow I took for granted, but by removing Alignment from the 
expression, -Value will have the precision of Value rather than the higher 
precision out of {Alignment, Value}, which silently changes the semantics of N 
and thus 2^N. For Alignment > 2^(precision of Value), this will indeed not work.

Both current issues would be resolved by some mechanism to type-promote on 
demand. I.e. if for ALIGN_VALUE, the sub-expression (Alignment - 1U) in 
~(Alignment - 1U) was promoted to the precision of Value, and for 
ALIGN_VALUE_ADDEND, the sub-expression Value in -Value was promoted to the 
precision of Alignment, both would be correct.

Best regards,
Marvin.

> 
> If you want a more technical explanation - previously, only the lower 
> “Alignment - 1” Bits of the result were considered. As they are 0 for 
> Alignment, the left operand, basically you get:
> 
> Result = (Alignment - Value) & (Alignment - 1) = (Alignment - Value)[0 : 
> Alignment - 1] = (Alignment[0 : Alignment - 1] - Value[0 : Alignment - 1])[…] 
> = (0 - Value[…])[…]
> 
> As you can see, only the lower Alignment -1 Bits of both operands matter and 
> they are always equal for Alignment and 0.
> 
> Best regards,
> Marvin
> 
>> 
>> Sent a v3.
>> 
>> -- 
>> Pedro



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104918): https://edk2.groups.io/g/devel/message/104918
Mute This Topic: https://groups.io/mt/98918981/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v3 1/1] MdePkg/Base.h: Simplify alignment expressions

[Marvin Häuser]

> On 16. May 2023, at 04:19, Pedro Falcato  wrote:
> 
> Simplify ALIGN_VALUE and ALIGN_VALUE_ADDEND into simpler expressions.
> 
> ALIGN_VALUE can simply be a (value + (align - 1)) & ~align
> expression, which works for any power of 2 alignment and generates
> smaller code sequences. For instance:
>ALIGN_VALUE(15, 16) = (15 + 15) & ~16 = 16
>ALIGN_VALUE(16, 16) = (16 + 15) & ~16 = 16
> 
> Old codegen:
>movq%rdi, %rax
>negq%rax
>andl$15, %eax
>addq%rdi, %rax
> 
> New codegen:
>leaq15(%rdi), %rax
>andq$-16, %rax
> 
> ALIGN_VALUE_ADDEND can simply use the negation of Value to get the
> addend for alignment, as, for instance:
>-15 & (16 - 1) = 1
>15 + 1 = 16
> 
> This change does not necessarily affect the end result, as the GCC and
> clang compilers were already able to see through things and optimize
> them into optimal instruction sequences, in the ALIGN_VALUE_ADDEND case.
> 
> Cc: Michael D Kinney 
> Cc: Liming Gao 
> Cc: Zhiguang Liu 
> Cc: Marvin Häuser 
> Signed-off-by: Pedro Falcato 
> ---
> Changes:
>- Correct the ADDEND macro to use negation and not a binary NOT (thanks 
> Liming!)
> MdePkg/Include/Base.h | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/MdePkg/Include/Base.h b/MdePkg/Include/Base.h
> index 6597e441a6e2..a070593a360d 100644
> --- a/MdePkg/Include/Base.h
> +++ b/MdePkg/Include/Base.h
> @@ -931,7 +931,7 @@ STATIC_ASSERT (ALIGNOF (__VERIFY_INT32_ENUM_SIZE) == 
> sizeof (__VERIFY_INT32_ENUM
> 
>   @return  Addend to round Value up to alignment boundary Alignment.
> **/
> -#define ALIGN_VALUE_ADDEND(Value, Alignment)  (((Alignment) - (Value)) & 
> ((Alignment) - 1U))
> +#define ALIGN_VALUE_ADDEND(Value, Alignment)  ((-(Value)) & ((Alignment) - 
> 1U))
> 
> /**
>   Rounds a value up to the next boundary using a specified alignment.
> @@ -945,7 +945,7 @@ STATIC_ASSERT (ALIGNOF (__VERIFY_INT32_ENUM_SIZE) == 
> sizeof (__VERIFY_INT32_ENUM
>   @return  A value up to the next boundary.
> 
> **/
> -#define ALIGN_VALUE(Value, Alignment)  ((Value) + ALIGN_VALUE_ADDEND (Value, 
> Alignment))
> +#define ALIGN_VALUE(Value, Alignment)  (((Value) + ((Alignment) - 1U)) & 
> ~(Alignment))

~((Alignment) - 1U)

Also, I'm quite sure this breaks for the case typeof(Value) > 
typeof(Alignment),typeof(Value) > unsigned int,typeof(Alignment) >= unsigned 
int (but always unsigned), this breaks. Assume Value is unsigned long long and 
Alignment is unsigned int. Then, the left-hand side will be unsigned long long. 
~((Alignment) - 1U) will be unsigned int (regardless of whether or not we use 1 
or 1U). Now, it will be promoted to unsigned long long for the evaluation of 
the AND operation, but because it is unsigned, it will not "sign-extend", thus 
discarding the high Bits of Value.

Best regards,
Marvin

> 
> /**
>   Adjust a pointer by adding the minimum offset required for it to be aligned 
> on
> -- 
> 2.40.1
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104917): https://edk2.groups.io/g/devel/message/104917
Mute This Topic: https://groups.io/mt/98918938/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2 1/1] MdePkg/Base.h: Simply alignment expressions

[Marvin Häuser]

> On 16. May 2023, at 04:22, Pedro Falcato  wrote:
> 
> On Tue, May 16, 2023 at 2:46 AM gaoliming via groups.io
>  wrote:
>> 
>> Pedro:
>> 
>>> -邮件原件-
>>> 发件人: devel@edk2.groups.io  代表 Pedro Falcato
>>> 发送时间: 2023年5月15日 23:15
>>> 收件人: devel@edk2.groups.io
>>> 抄送: Pedro Falcato ; Michael D Kinney
>>> ; Liming Gao ;
>>> Zhiguang Liu ; Marvin Häuser
>>> 
>>> 主题: [edk2-devel] [PATCH v2 1/1] MdePkg/Base.h: Simply alignment
>>> expressions
>>> 
>>> Simplify ALIGN_VALUE and ALIGN_VALUE_ADDEND into simpler expressions.
>>> 
>>> ALIGN_VALUE can simply be a (value + (align - 1)) & ~align
>>> expression, which works for any power of 2 alignment and generates
>>> smaller code sequences. For instance:
>>>  ALIGN_VALUE(15, 16) = (15 + 15) & ~16 = 16
>>>  ALIGN_VALUE(16, 16) = (16 + 15) & ~16 = 16
>>> 
>>> Old codegen:
>>>  movq%rdi, %rax
>>>  negq%rax
>>>  andl$15, %eax
>>>  addq%rdi, %rax
>>> 
>>> New codegen:
>>>  leaq15(%rdi), %rax
>>>  andq$-16, %rax
>>> 
>>> ALIGN_VALUE_ADDEND can simply use a bitwise NOT of Value to get the
>>> addend for alignment, as, for instance:
>>>  ~15 & (16 - 1) = 1
>>>  15 + 1 = 16
>>> 
>> 
>>>  ~15 & (16 - 1) = 1
>> Its value should be zero, not 1. I also verify the updated 
>> ALIGN_VALUE_ADDEND.
>> Its value is incorrect. Please double check.
> 
> Hi Liming, you're 100% right. There was a mixup when we were
> discussing this optimization, and I got the mental calculations wrong
> there.
> Two's complement is definitely what we want, as one's complement is
> always off by one (from what we want).
> 
> So negation (-) works beautifully, as seen in the old codegen (we
> figured this out from the compiler's output).

To be clear on the maths side of things:

“& (Alignment - 1U)” is equivalent to “mod Alignment” for powers of two.
“-Value” is equivalent to “2^N - Value” once the expression is promoted to an 
unsigned type, where N is the precision of said type.

So, the old expression basically was “(Alignment - Value) mod Alignment” and 
the new expression is “(2^N - Value) mod Alignment”. By modulo laws, we can 
apply the mod to the operands, which for the left ones gives us “Alignment mod 
Alignment = 0” and “2^N mod Alignment = 0”, obviously for Alignment being a 
power of two. They’re trivially equivalent.

If you want a more technical explanation - previously, only the lower 
“Alignment - 1” Bits of the result were considered. As they are 0 for 
Alignment, the left operand, basically you get:

Result = (Alignment - Value) & (Alignment - 1) = (Alignment - Value)[0 : 
Alignment - 1] = (Alignment[0 : Alignment - 1] - Value[0 : Alignment - 1])[…] = 
(0 - Value[…])[…]

As you can see, only the lower Alignment -1 Bits of both operands matter and 
they are always equal for Alignment and 0.

Best regards,
Marvin

> 
> Sent a v3.
> 
> -- 
> Pedro



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104913): https://edk2.groups.io/g/devel/message/104913
Mute This Topic: https://groups.io/mt/98918981/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] MdePkg/Base.h: Simply alignment expressions

[Marvin Häuser]

Well, I explicitly added this macro as a prerequisite to code used in our new 
PE library (remember this patch was initially sent in 2021). We still require 
it downstream, but obviously upstream is not interested in the related 
contributions that were to follow at the time.

Gerd picked it up because he wanted to attempt to re-try contributing the new 
PE library, but I haven't heard from him in weeks.

Design-wise, I agree it could be removed again. However, there first was a 
downstream burden when adding it (as we needed to rewrite our history to drop 
our downstream patch in favour of the upstream solution). Now introducing 
another downstream burden *again* to remove the macro that was added only a few 
weeks back would be a sign of poor management and planning.

Best regards,
Marvin

> On 15. May 2023, at 17:15, Rebecca Cran  wrote:
> 
> On 5/15/23 08:45, Pedro Falcato wrote:
>> -#define ALIGN_VALUE_ADDEND(Value, Alignment)  (((Alignment) - (Value)) & 
>> ((Alignment) - 1U))
>> +#define ALIGN_VALUE_ADDEND(Value, Alignment)  ((~(Value)) & ((Alignment) - 
>> 1U))
>>/**
>>Rounds a value up to the next boundary using a specified alignment.
>> @@ -945,7 +945,7 @@ STATIC_ASSERT (ALIGNOF (__VERIFY_INT32_ENUM_SIZE) == 
>> sizeof (__VERIFY_INT32_ENUM
>>@return  A value up to the next boundary.
>>**/
>> -#define ALIGN_VALUE(Value, Alignment)  ((Value) + ALIGN_VALUE_ADDEND 
>> (Value, Alignment))
>> +#define ALIGN_VALUE(Value, Alignment)  (((Value) + (Alignment - 1)) & 
>> ~(Alignment))
> 
> Since ALIGN_VALUE_ADDEND is only used in ALIGN_VALUE, it should probably be 
> deleted instead of updated.
> 
> 
> -- 
> 
> Rebecca Cran
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104886): https://edk2.groups.io/g/devel/message/104886
Mute This Topic: https://groups.io/mt/98904940/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v3 1/2] Ext4Pkg: Improve extent node validation on the number of entries

[Marvin Häuser]

Reviewed-by: Marvin Häuser 

> On 9. May 2023, at 17:49, Pedro Falcato  wrote:
> 
> Improve the extent tree node validation by validating the number of
> entries the node advertises against the theoretical max (derived from
> the size of on-disk structs and the block size (or i_data, if inline
> extents).
> 
> Previously, we did not validate the number of entries. This could be
> exploited for out-of-bounds reads and crashes.
> 
> Cc: Marvin Häuser 
> Fixes: d9ceedca6c8f ("Ext4Pkg: Add Ext4Dxe driver.")
> Reported-by: Savva Mitrofanov 
> Signed-off-by: Pedro Falcato 
> ---
> Features/Ext4Pkg/Ext4Dxe/Extents.c | 32 ++
> 1 file changed, 28 insertions(+), 4 deletions(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/Extents.c 
> b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> index 9e4364e50d99..2b82417c9e10 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/Extents.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> @@ -1,7 +1,7 @@
> /** @file
>   Extent related routines
> 
> -  Copyright (c) 2021 - 2022 Pedro Falcato All rights reserved.
> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>   SPDX-License-Identifier: BSD-2-Clause-Patent
> **/
> 
> @@ -80,13 +80,15 @@ Ext4GetInoExtentHeader (
> /**
>Checks if an extent header is valid.
>@param[in]  Header Pointer to the EXT4_EXTENT_HEADER structure.
> +   @param[in]  MaxEntries Maximum number of entries possible for 
> this tree node.
> 
>@return TRUE if valid, FALSE if not.
> **/
> STATIC
> BOOLEAN
> Ext4ExtentHeaderValid (
> -  IN CONST EXT4_EXTENT_HEADER  *Header
> +  IN CONST EXT4_EXTENT_HEADER  *Header,
> +  IN UINT16MaxEntries
>   )
> {
>   if (Header->eh_depth > EXT4_EXTENT_TREE_MAX_DEPTH) {
> @@ -99,6 +101,18 @@ Ext4ExtentHeaderValid (
> return FALSE;
>   }
> 
> +  // Note: We do not need to check eh_entries here, as the next branch makes 
> sure max >= entries
> +  if (Header->eh_max > MaxEntries) {
> +DEBUG ((
> +  DEBUG_ERROR,
> +  "[ext4] Invalid extent header max entries (%u eh_max, "
> +  "theoretical max is %u) (larger than permitted)\n",
> +  Header->eh_max,
> +  MaxEntries
> +  ));
> +return FALSE;
> +  }
> +
>   if (Header->eh_max < Header->eh_entries) {
> DEBUG ((
>   DEBUG_ERROR,
> @@ -212,6 +226,9 @@ Ext4ExtentIdxLeafBlock (
>   return LShiftU64 (Index->ei_leaf_hi, 32) | Index->ei_leaf_lo;
> }
> 
> +// Results of sizeof(i_data) / sizeof(extent) - 1 = 4
> +#define EXT4_NR_INLINE_EXTENTS  4
> +
> /**
>Retrieves an extent from an EXT4 inode.
>@param[in]  Partition Pointer to the opened EXT4 partition.
> @@ -237,6 +254,7 @@ Ext4GetExtent (
>   EXT4_EXTENT_HEADER  *ExtHeader;
>   EXT4_EXTENT_INDEX   *Index;
>   EFI_STATUS  Status;
> +  UINT32  MaxExtentsPerNode;
>   EXT4_BLOCK_NR   BlockNumber;
> 
>   Inode  = File->Inode;
> @@ -275,12 +293,17 @@ Ext4GetExtent (
> 
>   ExtHeader = Ext4GetInoExtentHeader (Inode);
> 
> -  if (!Ext4ExtentHeaderValid (ExtHeader)) {
> +  if (!Ext4ExtentHeaderValid (ExtHeader, EXT4_NR_INLINE_EXTENTS)) {
> return EFI_VOLUME_CORRUPTED;
>   }
> 
>   CurrentDepth = ExtHeader->eh_depth;
> 
> +  // A single node fits into a single block, so we can only have (BlockSize 
> / sizeof(EXT4_EXTENT)) - 1
> +  // extents in a single node. Note the -1, because both leaf and internal 
> node headers are 12 bytes,
> +  // and so are individual entries.
> +  MaxExtentsPerNode = (Partition->BlockSize / sizeof (EXT4_EXTENT)) - 1;
> +
>   while (ExtHeader->eh_depth != 0) {
> CurrentDepth--;
> // While depth != 0, we're traversing the tree itself and not any leaves
> @@ -309,6 +332,7 @@ Ext4GetExtent (
> }
> 
> // Read the leaf block onto the previously-allocated buffer.
> +
> Status = Ext4ReadBlocks (Partition, Buffer, 1, BlockNumber);
> if (EFI_ERROR (Status)) {
>   FreePool (Buffer);
> @@ -317,7 +341,7 @@ Ext4GetExtent (
> 
> ExtHeader = Buffer;
> 
> -if (!Ext4ExtentHeaderValid (ExtHeader)) {
> +if (!Ext4ExtentHeaderValid (ExtHeader, MaxExtentsPerNode)) {
>   FreePool (Buffer);
>   return EFI_VOLUME_CORRUPTED;
> }
> -- 
> 2.40.1
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104398): https://edk2.groups.io/g/devel/message/104398
Mute This Topic: https://groups.io/mt/98786841/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2 1/2] Ext4Pkg: Improve extent node validation on the number of entries

[Marvin Häuser]

Sorry, saw V2 too late…

> On 9. May 2023, at 02:55, Pedro Falcato  wrote:
> 
> Improve the extent tree node validation by validating the number of
> entries the node advertises against the theoretical max (derived from
> the size of on-disk structs and the block size (or i_data, if inline
> extents).
> 
> Previously, we did not validate the number of entries. This could be
> exploited for out-of-bounds reads and crashes.
> 
> Cc: Marvin Häuser 
> Fixes: d9ceedca6c8f ("Ext4Pkg: Add Ext4Dxe driver.")
> Reported-by: Savva Mitrofanov 
> Signed-off-by: Pedro Falcato 
> ---
> v2:
> Accidentally based my previous patch on the wrong version of Extents.c, which 
> accidentally undid some previous changes.
> 
> Features/Ext4Pkg/Ext4Dxe/Extents.c | 32 ++
> 1 file changed, 28 insertions(+), 4 deletions(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/Extents.c 
> b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> index 9e4364e50d99..2d86a7abdedb 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/Extents.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> @@ -1,7 +1,7 @@
> /** @file
>   Extent related routines
> 
> -  Copyright (c) 2021 - 2022 Pedro Falcato All rights reserved.
> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>   SPDX-License-Identifier: BSD-2-Clause-Patent
> **/
> 
> @@ -80,13 +80,15 @@ Ext4GetInoExtentHeader (
> /**
>Checks if an extent header is valid.
>@param[in]  Header Pointer to the EXT4_EXTENT_HEADER structure.
> +   @param[in]  MaxEntries Maximum number of entries possible for 
> this tree node.
> 
>@return TRUE if valid, FALSE if not.
> **/
> STATIC
> BOOLEAN
> Ext4ExtentHeaderValid (
> -  IN CONST EXT4_EXTENT_HEADER  *Header
> +  IN CONST EXT4_EXTENT_HEADER  *Header,
> +  IN UINT16MaxEntries
>   )
> {
>   if (Header->eh_depth > EXT4_EXTENT_TREE_MAX_DEPTH) {
> @@ -99,6 +101,18 @@ Ext4ExtentHeaderValid (
> return FALSE;
>   }
> 
> +  if ((Header->eh_max > MaxEntries) || (Header->eh_entries > MaxEntries)) {

My comment on V1 is still valid, this is implicit for eh_entries via the check 
below. The other comment does not apply to V2.

Best regards,
Marvin

> +DEBUG ((
> +  DEBUG_ERROR,
> +  "[ext4] Invalid extent header entries (extent header: %u max,"
> +  " %u entries, theoretical max: %u) (larger than permitted)\n",
> +  Header->eh_max,
> +  Header->eh_entries,
> +  MaxEntries
> +  ));
> +return FALSE;
> +  }
> +
>   if (Header->eh_max < Header->eh_entries) {
> DEBUG ((
>   DEBUG_ERROR,
> @@ -212,6 +226,9 @@ Ext4ExtentIdxLeafBlock (
>   return LShiftU64 (Index->ei_leaf_hi, 32) | Index->ei_leaf_lo;
> }
> 
> +// Results of sizeof(i_data) / sizeof(extent) - 1 = 4
> +#define EXT4_NR_INLINE_EXTENTS  4
> +
> /**
>Retrieves an extent from an EXT4 inode.
>@param[in]  Partition Pointer to the opened EXT4 partition.
> @@ -237,6 +254,7 @@ Ext4GetExtent (
>   EXT4_EXTENT_HEADER  *ExtHeader;
>   EXT4_EXTENT_INDEX   *Index;
>   EFI_STATUS  Status;
> +  UINT32  MaxExtentsPerNode;
>   EXT4_BLOCK_NR   BlockNumber;
> 
>   Inode  = File->Inode;
> @@ -275,12 +293,17 @@ Ext4GetExtent (
> 
>   ExtHeader = Ext4GetInoExtentHeader (Inode);
> 
> -  if (!Ext4ExtentHeaderValid (ExtHeader)) {
> +  if (!Ext4ExtentHeaderValid (ExtHeader, EXT4_NR_INLINE_EXTENTS)) {
> return EFI_VOLUME_CORRUPTED;
>   }
> 
>   CurrentDepth = ExtHeader->eh_depth;
> 
> +  // A single node fits into a single block, so we can only have (BlockSize 
> / sizeof(EXT4_EXTENT)) - 1
> +  // extents in a single node. Note the -1, because both leaf and internal 
> node headers are 12 bytes,
> +  // and so are individual entries.
> +  MaxExtentsPerNode = (Partition->BlockSize / sizeof (EXT4_EXTENT)) - 1;
> +
>   while (ExtHeader->eh_depth != 0) {
> CurrentDepth--;
> // While depth != 0, we're traversing the tree itself and not any leaves
> @@ -309,6 +332,7 @@ Ext4GetExtent (
> }
> 
> // Read the leaf block onto the previously-allocated buffer.
> +
> Status = Ext4ReadBlocks (Partition, Buffer, 1, BlockNumber);
> if (EFI_ERROR (Status)) {
>   FreePool (Buffer);
> @@ -317,7 +341,7 @@ Ext4GetExtent (
> 
> ExtHeader = Buffer;
> 
> -if (!Ext4ExtentHeaderValid (ExtHeader)) {
> +if (!Ext4ExtentHeaderValid (ExtHeader, MaxExtentsPerNode)) {
>   FreePool (Buffer);
>   return EFI_VOLUME_CORRUPTED;
> }
> -- 
> 2.40.1
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104339): https://edk2.groups.io/g/devel/message/104339
Mute This Topic: https://groups.io/mt/98774772/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 2/2] Ext4Pkg: Advertise CSUM_SEED as supported

[Marvin Häuser]

Reviewed-by: Marvin Häuser 

> On 9. May 2023, at 02:42, Pedro Falcato  wrote:
> 
> We had added support for CSUM_SEED but accidentally forgot to advertise
> it in gSupportedIncompatFeat. This made it (erroneously) impossible to
> mount CSUM_SEED filesystems.
> 
> Detected by attempting to mount a relatively new mkfs.ext4'd filesystem.
> 
> Cc: Marvin Häuser 
> Signed-off-by: Pedro Falcato 
> ---
> Features/Ext4Pkg/Ext4Dxe/Superblock.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/Superblock.c 
> b/Features/Ext4Pkg/Ext4Dxe/Superblock.c
> index 3f56de93c105..604925b582c1 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/Superblock.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/Superblock.c
> @@ -1,7 +1,7 @@
> /** @file
>   Superblock managing routines
> 
> -  Copyright (c) 2021 - 2022 Pedro Falcato All rights reserved.
> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>   SPDX-License-Identifier: BSD-2-Clause-Patent
> **/
> 
> @@ -18,7 +18,7 @@ STATIC CONST UINT32  gSupportedIncompatFeat =
>   EXT4_FEATURE_INCOMPAT_64BIT | EXT4_FEATURE_INCOMPAT_DIRDATA |
>   EXT4_FEATURE_INCOMPAT_FLEX_BG | EXT4_FEATURE_INCOMPAT_FILETYPE |
>   EXT4_FEATURE_INCOMPAT_EXTENTS | EXT4_FEATURE_INCOMPAT_LARGEDIR |
> -  EXT4_FEATURE_INCOMPAT_MMP | EXT4_FEATURE_INCOMPAT_RECOVER;
> +  EXT4_FEATURE_INCOMPAT_MMP | EXT4_FEATURE_INCOMPAT_RECOVER | 
> EXT4_FEATURE_INCOMPAT_CSUM_SEED;
> 
> // Future features that may be nice additions in the future:
> // 1) Btree support: Required for write support and would speed up lookups in 
> large directories.
> -- 
> 2.40.1
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104338): https://edk2.groups.io/g/devel/message/104338
Mute This Topic: https://groups.io/mt/98774551/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] Ext4Pkg: Improve extent node validation on the number of entries

[Marvin Häuser]

> On 9. May 2023, at 02:42, Pedro Falcato  wrote:
> 
> Improve the extent tree node validation by validating the number of
> entries the node advertises against the theoretical max (derived from
> the size of on-disk structs and the block size (or i_data, if inline
> extents).
> 
> Previously, we did not validate the number of entries. This could be
> exploited for out-of-bounds reads and crashes.
> 
> Cc: Marvin Häuser 
> Fixes: d9ceedca6c8f ("Ext4Pkg: Add Ext4Dxe driver.")
> Reported-by: Savva Mitrofanov 
> Signed-off-by: Pedro Falcato 
> ---
> Features/Ext4Pkg/Ext4Dxe/Extents.c | 47 +++---
> 1 file changed, 30 insertions(+), 17 deletions(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/Extents.c 
> b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> index 9e4364e50d99..66d085938549 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/Extents.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/Extents.c
> @@ -1,7 +1,7 @@
> /** @file
>   Extent related routines
> 
> -  Copyright (c) 2021 - 2022 Pedro Falcato All rights reserved.
> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>   SPDX-License-Identifier: BSD-2-Clause-Patent
> **/
> 
> @@ -80,13 +80,15 @@ Ext4GetInoExtentHeader (
> /**
>Checks if an extent header is valid.
>@param[in]  Header Pointer to the EXT4_EXTENT_HEADER structure.
> +   @param[in]  MaxEntries Maximum number of entries possible for 
> this tree node.
> 
>@return TRUE if valid, FALSE if not.
> **/
> STATIC
> BOOLEAN
> Ext4ExtentHeaderValid (
> -  IN CONST EXT4_EXTENT_HEADER  *Header
> +  IN CONST EXT4_EXTENT_HEADER  *Header,
> +  IN UINT16MaxEntries
>   )
> {
>   if (Header->eh_depth > EXT4_EXTENT_TREE_MAX_DEPTH) {
> @@ -99,6 +101,18 @@ Ext4ExtentHeaderValid (
> return FALSE;
>   }
> 
> +  if ((Header->eh_max > MaxEntries) || (Header->eh_entries > MaxEntries)) {

For eh_entries, this is implicit via the check below.

> +DEBUG ((
> +  DEBUG_ERROR,
> +  "[ext4] Invalid extent header entries (extent header: %u max,"
> +  " %u entries, theoretical max: %u) (larger than permitted)\n",
> +  Header->eh_max,
> +  Header->eh_entries,
> +  MaxEntries
> +  ));
> +return FALSE;
> +  }
> +
>   if (Header->eh_max < Header->eh_entries) {
> DEBUG ((
>   DEBUG_ERROR,
> @@ -212,6 +226,9 @@ Ext4ExtentIdxLeafBlock (
>   return LShiftU64 (Index->ei_leaf_hi, 32) | Index->ei_leaf_lo;
> }
> 
> +// Results of sizeof(i_data) / sizeof(extent) - 1 = 4
> +#define EXT4_NR_INLINE_EXTENTS  4
> +
> /**
>Retrieves an extent from an EXT4 inode.
>@param[in]  Partition Pointer to the opened EXT4 partition.
> @@ -237,7 +254,7 @@ Ext4GetExtent (
>   EXT4_EXTENT_HEADER  *ExtHeader;
>   EXT4_EXTENT_INDEX   *Index;
>   EFI_STATUS  Status;
> -  EXT4_BLOCK_NR   BlockNumber;
> +  UINT32  MaxExtentsPerNode;
> 
>   Inode  = File->Inode;
>   Ext= NULL;
> @@ -275,12 +292,17 @@ Ext4GetExtent (
> 
>   ExtHeader = Ext4GetInoExtentHeader (Inode);
> 
> -  if (!Ext4ExtentHeaderValid (ExtHeader)) {
> +  if (!Ext4ExtentHeaderValid (ExtHeader, EXT4_NR_INLINE_EXTENTS)) {
> return EFI_VOLUME_CORRUPTED;
>   }
> 
>   CurrentDepth = ExtHeader->eh_depth;
> 
> +  // A single node fits into a single block, so we can only have (BlockSize 
> / sizeof(EXT4_EXTENT)) - 1
> +  // extents in a single node. Note the -1, because both leaf and internal 
> node headers are 12 bytes,
> +  // and so are individual entries.
> +  MaxExtentsPerNode = (Partition->BlockSize / sizeof (EXT4_EXTENT)) - 1;
> +
>   while (ExtHeader->eh_depth != 0) {
> CurrentDepth--;
> // While depth != 0, we're traversing the tree itself and not any leaves
> @@ -289,17 +311,7 @@ Ext4GetExtent (
> // Therefore, we can use binary search, and it's actually the standard 
> for doing so
> // (see FreeBSD).
> 
> -Index   = Ext4BinsearchExtentIndex (ExtHeader, LogicalBlock);
> -BlockNumber = Ext4ExtentIdxLeafBlock (Index);
> -
> -// Check that block isn't file hole
> -if (BlockNumber == EXT4_BLOCK_FILE_HOLE)

Why is this no longer possible?

Best regards,
Marvin

> {
> -  if (Buffer != NULL) {
> -FreePool (Buffer);
> -  }
> -
> -  return EFI_VOLUME_CORRUPTED;
> -}
> +Index = Ext4BinsearchExtentIndex (ExtHeader, LogicalBlock);
> 
> if (Buffer == NULL) {
>   Buffer = AllocatePool (Partition->BlockSize);
> @@ -309,7 +321,8 @@ Ext4GetExtent (
> }
> 
> // Read the leaf block onto the previously-allo

Re: [edk2-devel] [PATCH v5 02/10] MdePkg: don't set visibility to hidden

[Marvin Häuser]

> On 21. Apr 2023, at 09:21, Ard Biesheuvel  wrote:
> 
> On Fri, 21 Apr 2023 at 08:49, Gerd Hoffmann  wrote:
>> 
>>> On Fri, Apr 21, 2023 at 06:01:11AM +, Marvin Häuser wrote:
>>> 
>>>> On 21. Apr 2023, at 06:45, Gerd Hoffmann  wrote:
>>>> 
>>>> Not needed any more on modern toolchains, they are better
>>>> in not creating a GOT without this trick.
>>> 
>>> Hi Gerd,
>>> 
>>> Thanks! Just out of interest, how did you test this and what were the 
>>> results?
>> 
>> Patch #1, adding a linker script assert as suggested by ard, then:
>> 
>> * compile + test on my local workstation (fedora 37, gcc 12).
>> * run CI
>> * compile on some older distros:
>>   - rhel-8 (gcc 8)
>>   - ubuntu-18.04 (gcc 7)
>> 
> 
> I just realized that on x86, GenFw has some code to deal with GOT
> entries if they are emitted. I'm not sure how often that gets
> exercised, given our prior use of hidden visibility, but at least the
> GOT entries should be covered by relocations if they exist.
> 
> *However*, one thing we are not taking into account is the fact that
> relaxations are not usually reflected in the relocations emitted by
> the compiler when using --emit-relocs. So we might end up with
> occurrences like the below (taken from the Linux kernel but the idea
> is the same)
> 
> 82fa59d5:   4c 8d 0d 24 66 88 fflea-0x7799dc(%rip),%r9
>  82fa59d8: R_X86_64_REX_GOTPCRELXlevel4_kernel_pgt-0x4
> 82fa59dc:   49 8d 69 67 lea0x67(%r9),%rbp
> 82fa59e0:   4c 8d 15 19 76 88 fflea-0x7789e7(%rip),%r10
>  82fa59e3: R_X86_64_REX_GOTPCRELXlevel3_kernel_pgt-0x4
> 
> So here, the GOT loads have been relaxed into LEA instructions, but
> GenFw will decode the immediate and assume it points to the GOT entry
> rather than the variable itself, and happily emit a PE relocation for
> it.
> 
> So it would be better to ASSERT() on non-empty GOT, and ignore such
> GOTPCREL relocations instead of attempting to relocate the GOT entries
> they (used to) refer to.

Hmm, we’ve been toying with using only PIE relocs for X64 for a bit and finally 
merged it into master, so far no issues:
https://github.com/acidanthera/audk/commit/92bb32130bcd0c35e48bdc308a18e5bc74cbaa42
https://github.com/acidanthera/audk/commit/42988773a06f9d6bf345fcbe82c1082ff1cfa2af

In fact (I *did not* confirm this, it’s only a report I got), it seems to fix 
something regarding the stack protector. I’d not be surprised if there are 
edge-cases where -q does not get all necessary relocs when PIE is enabled.

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103381): https://edk2.groups.io/g/devel/message/103381
Mute This Topic: https://groups.io/mt/98404595/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v5 02/10] MdePkg: don't set visibility to hidden

[Marvin Häuser]

> On 21. Apr 2023, at 08:49, Gerd Hoffmann  wrote:
> 
> On Fri, Apr 21, 2023 at 06:01:11AM +0000, Marvin Häuser wrote:
>> 
>>>> On 21. Apr 2023, at 06:45, Gerd Hoffmann  wrote:
>>> 
>>> Not needed any more on modern toolchains, they are better
>>> in not creating a GOT without this trick.
>> 
>> Hi Gerd,
>> 
>> Thanks! Just out of interest, how did you test this and what were the 
>> results?
> 
> Patch #1, adding a linker script assert as suggested by ard, then:
> 
> * compile + test on my local workstation (fedora 37, gcc 12).
> * run CI
> * compile on some older distros:
>   - rhel-8 (gcc 8)
>   - ubuntu-18.04 (gcc 7)

Did you include NOOPT? GCC5 specifies USING_LTO for DEBUG and RELEASE, thus the 
visibility would not be changed anyway.

If this works, I hope the series makes it in time for the next stable tag. :)

Thanks!

Best regards,
Marvin

> 
> take care,
>  Gerd
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103376): https://edk2.groups.io/g/devel/message/103376
Mute This Topic: https://groups.io/mt/98404595/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v5 02/10] MdePkg: don't set visibility to hidden

[Marvin Häuser]

> On 21. Apr 2023, at 06:45, Gerd Hoffmann  wrote:
> 
> Not needed any more on modern toolchains, they are better
> in not creating a GOT without this trick.

Hi Gerd,

Thanks! Just out of interest, how did you test this and what were the results?

Best regards,
Marvin

> 
> Signed-off-by: Gerd Hoffmann 
> ---
> MdePkg/Include/X64/ProcessorBind.h | 14 --
> 1 file changed, 14 deletions(-)
> 
> diff --git a/MdePkg/Include/X64/ProcessorBind.h 
> b/MdePkg/Include/X64/ProcessorBind.h
> index f0a4d00142b9..afbb4b6273fb 100644
> --- a/MdePkg/Include/X64/ProcessorBind.h
> +++ b/MdePkg/Include/X64/ProcessorBind.h
> @@ -21,20 +21,6 @@
>   #pragma pack()
> #endif
> 
> -#if defined (__GNUC__) && defined (__pic__) && !defined (USING_LTO)  && 
> !defined (__APPLE__)
> -//
> -// Mark all symbol declarations and references as hidden, meaning they will
> -// not be subject to symbol preemption. This allows the compiler to refer to
> -// symbols directly using relative references rather than via the GOT, which
> -// contains absolute symbol addresses that are subject to runtime relocation.
> -//
> -// The LTO linker will not emit GOT based relocations when all symbol
> -// references can be resolved locally, and so there is no need to set the
> -// pragma in that case (and doing so will cause other issues).
> -//
> -  #pragma GCC visibility push (hidden)
> -#endif
> -
> #if defined (__INTEL_COMPILER)
> //
> // Disable ICC's remark #869: "Parameter" was never referenced warning.
> -- 
> 2.40.0
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103371): https://edk2.groups.io/g/devel/message/103371
Mute This Topic: https://groups.io/mt/98404595/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v3 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

With the current ASM_FUNC() macro, there is no good way to declare an
alignment constraint for a function. As ASM_FUNC() switches sections,
declaring the constraint before the macro invocation applies it to the
current location in the previous section. Declaring the constraint after
the macro invocation lets the function label point to the location prior
to alignment. Depending on toolchain behaviour, this may cause the label
to point to alignment padding preceding the actual function definition.

To address these issues, introduce the ASM_FUNC_ALIGN() macro, which
declares the alignment constraint right before the function label.

Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 ArmPkg/Include/AsmMacroIoLibV8.h | 11 +++
 1 file changed, 11 insertions(+)

diff --git a/ArmPkg/Include/AsmMacroIoLibV8.h b/ArmPkg/Include/AsmMacroIoLibV8.h
index 135aaeca5d0b..81164ea9c9e7 100644
--- a/ArmPkg/Include/AsmMacroIoLibV8.h
+++ b/ArmPkg/Include/AsmMacroIoLibV8.h
@@ -41,8 +41,19 @@
   Name:   ; \
   AARCH64_BTI(c)
 
+#define _ASM_FUNC_ALIGN(Name, Section, Align)   \
+  .global   Name  ; \
+  .section  #Section, "ax"; \
+  .type Name, %function   ; \
+  .balign   Align ; \
+  Name:   ; \
+  AARCH64_BTI(c)
+
 #define ASM_FUNC(Name)  _ASM_FUNC(ASM_PFX(Name), .text. ## Name)
 
+#define ASM_FUNC_ALIGN(Name, Align)  \
+  _ASM_FUNC_ALIGN(ASM_PFX(Name), .text. ## Name, Align)
+
 #define MOV32(Reg, Val)   \
   movz  Reg, (Val) >> 16, lsl #16   ; \
   movk  Reg, (Val) & 0x
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103311): https://edk2.groups.io/g/devel/message/103311
Mute This Topic: https://groups.io/mt/98391308/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v3 2/2] ArmPkg/ArmMmuLib: Fix ArmReplaceLiveTranslationEntry() alignment

[Marvin Häuser]

As the ASM_FUNC() macro performs a section switch, the preceding
.balign directive applies the alignment constraint to the current
location in the previous section. As the linker may not merge the
sections in-order, ArmReplaceLiveTranslationEntry() may be left
unaligned.

Replace the explicit invocation of .balign with the ASM_FUNC_ALIGN()
macro, which guarantees the alignment constraint is applied correctly.
To make sure related issues are reliably caught in the future, align the
end of the function before checking the total occupied size. This
ensures crossing a 0x200 boundary will cause a compilation error.

Reviewed-by: Leif Lindholm 
Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 .../ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S  | 14 --
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S 
b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
index e936a5be4e11..887439bc042f 100644
--- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
+++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
@@ -69,17 +69,16 @@
 .L2_\@:
   .endm
 
-  // Align this routine to a log2 upper bound of its size, so that it is
-  // guaranteed not to cross a page or block boundary.
-  .balign 0x200
-
 //VOID
 //ArmReplaceLiveTranslationEntry (
 //  IN  UINT64  *Entry,
 //  IN  UINT64  Value,
 //  IN  UINT64  Address
 //  )
-ASM_FUNC(ArmReplaceLiveTranslationEntry)
+//
+// Align this routine to a log2 upper bound of its size, so that it is
+// guaranteed not to cross a page or block boundary.
+ASM_FUNC_ALIGN(ArmReplaceLiveTranslationEntry, 0x200)
 
   // disable interrupts
   mrs   x4, daif
@@ -101,5 +100,8 @@ ASM_GLOBAL ASM_PFX(ArmReplaceLiveTranslationEntrySize)
 ASM_PFX(ArmReplaceLiveTranslationEntrySize):
   .long   . - ArmReplaceLiveTranslationEntry
 
-  // Double check that we did not overrun the assumed maximum size
+  // Double check that we did not overrun the assumed maximum size or cross a
+  // 0x200 boundary (and thus implicitly not any larger power of two, including
+  // the page size).
+  .balign 0x200
   .orgArmReplaceLiveTranslationEntry + 0x200
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103312): https://edk2.groups.io/g/devel/message/103312
Mute This Topic: https://groups.io/mt/98391309/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

On 19. Apr 2023, at 23:55, Ard Biesheuvel  wrote:On Wed, 19 Apr 2023 at 22:10, Marvin Häuser  wrote:On 19. Apr 2023, at 21:48, Ard Biesheuvel  wrote:The issue is likely caused by-Wl,--defsym=PECOFF_HEADER_SIZE=0Why are you setting that? It breaks the ELF to PE conversion.Where?It would, but you only appear to be setting that for ASLD_DLINK_FLAGS,right? So that seems unrelated.BaseTools/tools_def AARCH64 ARM: suppres PIE sections via linker script · tianocore/edk2@14ca435github.comThe only thing I am observing is that the store to memory inArmMmuBaseLibConstructor()  Hob = GetFirstGuidHob ();  if (Hob != NULL) {    mReplaceLiveEntryFunc = *(VOID **)GET_GUID_HOB_DATA (Hob);is writing to the emulated NOR flash, and this switches it into NORprogramming mode, causing the firmware to crash immediately as it canno longer fetch instructions.That makes so much more sense now! I expected one of three things to happen:1) The write actually succeeds (after all, this is a VM, this might actually be the case for x86 OVMF)2) The write is silently discarded3) The write causes an exceptionI certainly did not expect *this*. When we initially tried to debug this, we attempted to use watchpoints to no avail, expecting it to be regular memory corruption. As those didn’t fire, we messed with function alignment and discovered the reported bug (which we didn’t really even trigger to begin with, it appears!). I suppose fixing its alignment meant some code that’s important down the line is fetched earlier as part of some flash unit and that’s why it started to work after fixing it. Whew.FYI I am using GDB to step through the code, i.e.,- run gdb (or 'gdb-multiarch' if you are cross-compiling)- start qemu with -s -S- connect using 'target remote :1234'- paste the 'add-symbol-file' line, e.g.,add-symbol-file/home/ard/build/edk2-workspace/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/DxeIplPeim/DxeIpl/DEBUG/DxeIpl.dll0x3- set breakpoint"hb _ModuleEntryPoint"- start executing"c"- use 'ni' to advance to the 'str' instruction that sets mReplaceLiveEntryFunc0x3553c <_ModuleEntryPoint+96>  str x1, [x0, #224]Now, as soon as I step over that instruction (using 's'), the entireview of memory changes into│  > 0x35540 <_ModuleEntryPoint+100> .inst   0x00800080 ; undefined│    0x35544 <_ModuleEntryPoint+104> .inst   0x00800080 ; undefinedetc, and the next step generates an exception, but this cannot behandled either. This is all related to the NOR flash emulation code inQEMU, that stops working as a ROM and switches into programming mode.I cannot explain why this only happens in this case, and why somewrites seem to be ignored. But it does explain why this particularfirmware build is misbehavingNow, if you apply the following patches:ArmPkg/Mmu: Remove handling of NONSECURE memory regionsArmPkg/ArmMmuLib: Introduce region types for RO/XP WB cached memoryArmVirtPkg/ArmVirtQemu: Use read-only memory region type for code flash(from the edk2-devel list), your build still crashes, but it printsone additional lineSynchronous Exception at 0x3553Cwhich is the exception caused by the write to NOR flash, which is nowmapped read-only in the page tables, and so it is caught by thefirmware itself.That’s actually something I proposed to debug the issue early on, but we’re all so-so with ARM experience, so we never got to that with the limited time we could spare. Praise to you!If you subsequently applyArmVirtPkg/ArmVirtQemu: Use PEI flavor of ArmMmuLib for all PEIMsthings work as expected.https://github.com/ardbiesheuvel/edk2/tree/arm_corruption-latest-ardbI‘d love to confirm all this, but I can’t spare the time. I blindly trust you and will try to submit V3 within this week.Best regards,Marvin

_._,_._,_



Groups.io Links:


  
You receive all messages sent to this group.
  
  



View/Reply Online (#103259) |


  

|

  Mute This Topic

| New Topic




Your Subscription |
Contact Group Owner |

Unsubscribe

 [arch...@mail-archive.com]
_._,_._,_

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 19. Apr 2023, at 22:10, Marvin Häuser  wrote:
> 
> 
>> On 19. Apr 2023, at 21:48, Ard Biesheuvel  wrote:
>> 
>> The issue is likely caused by
>> 
>> -Wl,--defsym=PECOFF_HEADER_SIZE=0
>> 
>> Why are you setting that? It breaks the ELF to PE conversion.
> 
> Where?

AUDK doesn’t use that macro, so you must mean my edk2 branch. In that case - I 
don’t know, that’s upstream code that isn’t mine? :D

It’s only done for ASLDLINK, which should not interfere here. Still, ASLC does 
generate PE binaries, so I’m not sure what the deal is with setting it to 0.

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103257): https://edk2.groups.io/g/devel/message/103257
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 19. Apr 2023, at 21:48, Ard Biesheuvel  wrote:
> 
> The issue is likely caused by
> 
> -Wl,--defsym=PECOFF_HEADER_SIZE=0
> 
> Why are you setting that? It breaks the ELF to PE conversion.

Where?


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103254): https://edk2.groups.io/g/devel/message/103254
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 19. Apr 2023, at 20:26, Ard Biesheuvel  wrote:
> 
> On Wed, 19 Apr 2023 at 20:25, Marvin Häuser  wrote:
>> 
>> 
>> On 19. Apr 2023, at 20:03, Ard Biesheuvel  wrote:
>> 
>> Your branch seems to be missing 16e0969ef775b898ac700f3261d76030b8ab9ef0
>> 
>> "ArmVirtPkg/ArmVirtQemu: Use PEI flavor of ArmMmuLib for all PEIMs"
>> 
>> 
>> That's correct (because that commit is after the last commit I managed to 
>> reproduce the issue with), but I don't see how this commit would fix the 
>> issue. As I said, the symptom is that PeiCore memory is badly corrupted and 
>> the stall happens due to executing said corruption, not due to jumping to 
>> NULL. Those broken branches I linked can all be made work by rolling back 
>> the change to MemoryAllocationLib (which changes the code size, thus 
>> misaligns *something*). In fact, using the broken variant only for 
>> MemoryInitPei is sufficient to reproduce the issue, other modules don't seem 
>> to be involved.
>> 
> 
> Applying that commit made your branch work for me.

Yes, that might very well be - applying ae2c904 also "fixes" the issue as per 
https://github.com/mhaeuser/edk2/tree/arm_corruption-earliest-fixed

And technically, so does reverting this line :) 
https://github.com/mhaeuser/edk2/commit/7a96986e024f9c7ccf4774cc6f2ddb47a3abc86e#diff-1edfe01abdf8e4dcac640db4d9436e17b5f15d037714df7f365b58fcfc91e425R409

I don't understand how the changes would *fix* (rather than hide) the issue, so 
I'd attribute it to lucky codegen that doesn't misalign whatever is misaligned. 
I unfortunately have absolutely no time to get back to debugging this. :(

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103238): https://edk2.groups.io/g/devel/message/103238
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 19. Apr 2023, at 20:03, Ard Biesheuvel  wrote:
> 
> Your branch seems to be missing 16e0969ef775b898ac700f3261d76030b8ab9ef0
> 
> "ArmVirtPkg/ArmVirtQemu: Use PEI flavor of ArmMmuLib for all PEIMs"

That's correct (because that commit is after the last commit I managed to 
reproduce the issue with), but I don't see how this commit would fix the issue. 
As I said, the symptom is that PeiCore memory is badly corrupted and the stall 
happens due to executing said corruption, not due to jumping to NULL. Those 
broken branches I linked can all be made work by rolling back the change to 
MemoryAllocationLib (which changes the code size, thus misaligns *something*). 
In fact, using the broken variant only for MemoryInitPei is sufficient to 
reproduce the issue, other modules don't seem to be involved.

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103236): https://edk2.groups.io/g/devel/message/103236
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 19. Apr 2023, at 19:40, Ard Biesheuvel  wrote:
> 
> On Wed, 19 Apr 2023 at 19:14, Marvin Häuser  <mailto:mhaeu...@posteo.de>> wrote:
>> 
>> Hi all,
>> 
>> While testing Ard's suggestion for V3, I noticed I got a broken FD where 
>> ArmReplaceLiveTranslationEntry() is misaligned, but does not cross a 4 KB 
>> boundary.
> 
> Which platform are you building?

ArmVirtPkg / AARCH64 / DEBUG / GCC5 (GCC 12.2.0).

> 
>> To not just hide the issue via this patch, can someone please try to explain 
>> the exact requirements this function has (the comments read like 0x200 was 
>> just the lowest value to guarantee staying within a page)? Why would it be 
>> broken if misaligned, but not crossing a page?
>> 
> 
> 0x200 is a log2 upper bound for the size of the function, so it's just
> the smallest value that fits that requirement, determined manually
> iirc
> 
> And the only reason we have this is that we can cheaply decide whether
> or not unmapping a page will unmap this function or not, but we could
> actually just use the address and size to decide this.
> 
> In any case, if the FD is constructed in a way that violates the
> alignment, there is something wrong with the build tools you are
> using.

The tools are stock edk2, the only changes made are those in the latest commit 
of the linked branch.

> 
>> Is there any chance the FD is somehow misaligned in memory, thus shifting 
>> the function across a page in the process? Or is the FD mapped to a fixed 
>> address like with x86? Is code after ArmReplaceLiveTranslationEntry() 
>> crossing page boundaries the actual issue (and is implicitly fixed by 
>> aligning it)?
>> 
> 
> If you are building ArmVirtQemu.dsc, the FD is mapped at address 0x0
> and the FV is mapped at 0x1000

Then the function simply is not crossing a page boundary... which means the 
patch did fix a valid bug, but it wasn't what actually caused the corruption. 
Any help is appreciated. :)

Best regards,
Marvin

> 
> 
>> For reference, I attached the broken FD. The problematic 
>> ArmReplaceLiveTranslationEntry() is located at MemoryInitPei 0x25C10 - 
>> 0x25D54.
>> This is from my arm_corruption-latest branch: 
>> https://github.com/mhaeuser/edk2/tree/arm_corruption-latest
>> 
>> Best regards,
>> Marvin
>> 
>> 
>> On 18. Apr 2023, at 10:18, Marvin Häuser > <mailto:mhaeu...@posteo.de>> wrote:
>> 
>> 
>> 
>> On 18. Apr 2023, at 10:10, Ard Biesheuvel > <mailto:a...@kernel.org>> wrote:
>> 
>> On Tue, 18 Apr 2023 at 08:40, Marvin Häuser > <mailto:mhaeu...@posteo.de>> wrote:
>> 
>> 
>> 
>> On 17. Apr 2023, at 23:18, Ard Biesheuvel > <mailto:a...@kernel.org>> wrote:
>> 
>> Agree with all of this.
>> 
>> And thanks for tracking this down - must not have been fun :-)
>> 
>> 
>> No worries - it wasn’t. :) It was mere luck Vitaly discovered early it was 
>> an issue that triggered based on the binary layout rather than a bug in the 
>> mapping code itself…
>> 
>> Speaking of not fun to track down, I initially wanted to add ASSERTs (yes, 
>> runtime :( ) to check the alignment guarantee is actually met. Leif 
>> basically declined any form of regression-testing at runtime. Do you happen 
>> to have a simple(!) idea for how it could be checked at build-time? (It’s 
>> less about “which commands do I run?†and more about integration with the 
>> build process / BaseTools, cross-OS compatibility, etc.)
>> 
>> 
>> 
>> I think we should just add another align to the code:
>> 
>> .align xx
>> func:
>> 
>> < code >
>> 
>> .align xx
>> .org func + xx
>> 
>> That way, if the code straddles a xx-aligned boundary, the .org will
>> move backwards and force an error.
>> 
>> 
>> Wow, that's pretty fucking smart... I didn't even know that directive was a 
>> thing. I will try to toy with it soon. Do you want it as a separate series, 
>> separate commit in the current series, or squashed into the fix?
>> 
>> 
>> Attachments:
>> 
>> QEMU_EFI.fd
>> 
>> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103233): https://edk2.groups.io/g/devel/message/103233
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 01/10] BaseTools: add BASETOOLS define

[Marvin Häuser]

> On 18. Apr 2023, at 15:20, Gerd Hoffmann  wrote:
> 
> On Tue, Apr 18, 2023 at 01:59:43PM +0200, Ard Biesheuvel wrote:
>> On Tue, 18 Apr 2023 at 13:52, Gerd Hoffmann  wrote:
>>> 
>>> Seems to work fine on fedora 37, even without adding --relax, maybe this
>>> is enabled by default (there is a --no-relax switch after all).  I'll go
>>> try older distros / compilers / binutils too.
>>> 
>>> What would be the failure mode?  Errors on ELF -> PE conversion because
>>> a GOT is present?  Or will things break at runtime?
>>> 
>> 
>> The problem here is that we rely on --emit-relocs to get at the
>> relocations in the binary, in order to convert the absolute ones into
>> PE/COFF relocations.
>> 
>> However, --emit-relocs did not use to cover the GOT, as those are
>> added at the end by the linker and not by the compiler. So if the GOT
>> is non-empty, the resulting PE executable will be corrupt.
> 
> So no build error.  And at runtime probably random effects, depending on
> whenever the execution path happens to hit a bad relocation or not.  So
> compile + boot testing doesn't cut it.  Lovely.
> 
> So, what can I do instead?  Check **/DEBUG/*.debug and see whenever a
> .got section is present?
> 
>> Of course, the answer here is to dump GenFw altogether for ELF to PE
>> conversion, and implement something that consumes the dynamic
>> relocations generated when linking in PIE mode.
> 
> Marvin's ImageTool is exactly that I think ...

Not quite, we actually didn't change anything about PIE vs non-PIE yet (X64 is 
PIE, rest is non-PIE). Though that was on the "long-term TODO" for sure. There 
are many other advantages though and if you're looking for an alternative to 
GenFw, I'm still happy to discuss things right now. As I develop this as part 
of a thesis and its deadline is in July, any efforts past that heavily depend 
on the workload at whatever job I will end up in. :)

Best regards,
Marvin

> 
> take care,
>  Gerd
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103164): https://edk2.groups.io/g/devel/message/103164
Mute This Topic: https://groups.io/mt/98257896/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 18. Apr 2023, at 10:10, Ard Biesheuvel  wrote:
> 
> On Tue, 18 Apr 2023 at 08:40, Marvin Häuser  wrote:
>> 
>> 
>>> On 17. Apr 2023, at 23:18, Ard Biesheuvel  wrote:
>>> 
>>> Agree with all of this.
>>> 
>>> And thanks for tracking this down - must not have been fun :-)
>> 
>> No worries - it wasn’t. :) It was mere luck Vitaly discovered early it was 
>> an issue that triggered based on the binary layout rather than a bug in the 
>> mapping code itself…
>> 
>> Speaking of not fun to track down, I initially wanted to add ASSERTs (yes, 
>> runtime :( ) to check the alignment guarantee is actually met. Leif 
>> basically declined any form of regression-testing at runtime. Do you happen 
>> to have a simple(!) idea for how it could be checked at build-time? (It’s 
>> less about “which commands do I run?” and more about integration with the 
>> build process / BaseTools, cross-OS compatibility, etc.)
>> 
> 
> 
> I think we should just add another align to the code:
> 
> .align xx
> func:
> 
> < code >
> 
> .align xx
> .org func + xx
> 
> That way, if the code straddles a xx-aligned boundary, the .org will
> move backwards and force an error.

Wow, that's pretty fucking smart... I didn't even know that directive was a 
thing. I will try to toy with it soon. Do you want it as a separate series, 
separate commit in the current series, or squashed into the fix?

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103151): https://edk2.groups.io/g/devel/message/103151
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

> On 17. Apr 2023, at 23:18, Ard Biesheuvel  wrote:
> 
> Agree with all of this.
> 
> And thanks for tracking this down - must not have been fun :-)

No worries - it wasn’t. :) It was mere luck Vitaly discovered early it was an 
issue that triggered based on the binary layout rather than a bug in the 
mapping code itself…

Speaking of not fun to track down, I initially wanted to add ASSERTs (yes, 
runtime :( ) to check the alignment guarantee is actually met. Leif basically 
declined any form of regression-testing at runtime. Do you happen to have a 
simple(!) idea for how it could be checked at build-time? (It’s less about 
“which commands do I run?” and more about integration with the build process / 
BaseTools, cross-OS compatibility, etc.)

Thanks!

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103133): https://edk2.groups.io/g/devel/message/103133
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v2 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

With the current ASM_FUNC() macro, there is no good way to declare an
alignment constraint for a function. As ASM_FUNC() switches sections,
declaring the constraint before the macro invocation applies it to the
current location in the previous section. Declaring the constraint after
the macro invocation lets the function label point to the location prior
to alignment. Depending on toolchain behaviour, this may cause the label
to point to alignment padding preceding the actual function definition.

To address these issues, introduce the ASM_FUNC_ALIGN() macro, which
declares the alignment constraint right before the function label.

Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 ArmPkg/Include/AsmMacroIoLibV8.h | 11 +++
 1 file changed, 11 insertions(+)

diff --git a/ArmPkg/Include/AsmMacroIoLibV8.h b/ArmPkg/Include/AsmMacroIoLibV8.h
index 135aaeca5d0b..81164ea9c9e7 100644
--- a/ArmPkg/Include/AsmMacroIoLibV8.h
+++ b/ArmPkg/Include/AsmMacroIoLibV8.h
@@ -41,8 +41,19 @@
   Name:   ; \
   AARCH64_BTI(c)
 
+#define _ASM_FUNC_ALIGN(Name, Section, Align)   \
+  .global   Name  ; \
+  .section  #Section, "ax"; \
+  .type Name, %function   ; \
+  .balign   Align ; \
+  Name:   ; \
+  AARCH64_BTI(c)
+
 #define ASM_FUNC(Name)  _ASM_FUNC(ASM_PFX(Name), .text. ## Name)
 
+#define ASM_FUNC_ALIGN(Name, Align)  \
+  _ASM_FUNC_ALIGN(ASM_PFX(Name), .text. ## Name, Align)
+
 #define MOV32(Reg, Val)   \
   movz  Reg, (Val) >> 16, lsl #16   ; \
   movk  Reg, (Val) & 0x
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103123): https://edk2.groups.io/g/devel/message/103123
Mute This Topic: https://groups.io/mt/98330366/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v2 2/2] ArmPkg/ArmMmuLib: Fix ArmReplaceLiveTranslationEntry() alignment

[Marvin Häuser]

As the ASM_FUNC() macro performs a section switch, the preceding
.balign directive applies the alignment constraint to the current
location in the previous section. As the linker may not merge the
sections in-order, ArmReplaceLiveTranslationEntry() may be left
unaligned.

Replace the explicit invocation of .balign with the ASM_FUNC_ALIGN()
macro, which guarantees the alignment constraint is applied correctly.

Reviewed-by: Leif Lindholm 
Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S 
b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
index e936a5be4e11..7627aeb95464 100644
--- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
+++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
@@ -69,17 +69,16 @@
 .L2_\@:
   .endm
 
-  // Align this routine to a log2 upper bound of its size, so that it is
-  // guaranteed not to cross a page or block boundary.
-  .balign 0x200
-
 //VOID
 //ArmReplaceLiveTranslationEntry (
 //  IN  UINT64  *Entry,
 //  IN  UINT64  Value,
 //  IN  UINT64  Address
 //  )
-ASM_FUNC(ArmReplaceLiveTranslationEntry)
+//
+// Align this routine to a log2 upper bound of its size, so that it is
+// guaranteed not to cross a page or block boundary.
+ASM_FUNC_ALIGN(ArmReplaceLiveTranslationEntry, 0x200)
 
   // disable interrupts
   mrs   x4, daif
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103122): https://edk2.groups.io/g/devel/message/103122
Mute This Topic: https://groups.io/mt/98330365/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH 2/2] ArmPkg/ArmMmuLib: Fix ArmReplaceLiveTranslationEntry() alignment

[Marvin Häuser]

As the ASM_FUNC() macro performs a section switch, the preceding
.balign directive applies the alignment constraint to the current
location in the previous section. As the linker may not merge the
sections in-order, ArmReplaceLiveTranslationEntry() may be left
unaligned.

Replace the explicit invocation of .balign with the ASM_FUNC_ALIGN()
macro, which guarantees the alignment constraint is applied correctly.

Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S 
b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
index e936a5be4e11..7627aeb95464 100644
--- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
+++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
@@ -69,17 +69,16 @@
 .L2_\@:
   .endm
 
-  // Align this routine to a log2 upper bound of its size, so that it is
-  // guaranteed not to cross a page or block boundary.
-  .balign 0x200
-
 //VOID
 //ArmReplaceLiveTranslationEntry (
 //  IN  UINT64  *Entry,
 //  IN  UINT64  Value,
 //  IN  UINT64  Address
 //  )
-ASM_FUNC(ArmReplaceLiveTranslationEntry)
+//
+// Align this routine to a log2 upper bound of its size, so that it is
+// guaranteed not to cross a page or block boundary.
+ASM_FUNC_ALIGN(ArmReplaceLiveTranslationEntry, 0x200)
 
   // disable interrupts
   mrs   x4, daif
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103118): https://edk2.groups.io/g/devel/message/103118
Mute This Topic: https://groups.io/mt/98325902/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH 1/2] ArmPkg/AsmMacroIoLibV8: Introduce ASM_FUNC_ALIGN()

[Marvin Häuser]

With the current ASM_FUNC() macro, there is no good way to declare an
alignment constraint for a function. As ASM_FUNC() switches sections,
declaring the constraint before the macro invocation applies it to the
current location in the previous section. Declaring the constraint after
the macro invocation lets the function label point to the location prior
to alignment. Depending on toolchain behaviour, this may cause the label
to point to alignment padding preceding the actual function definition.

To address these issues, introduce the ASM_FUNC_ALIGN() macro, which
declares the alignment constraint right before the function label.

Signed-off-by: Marvin Häuser 
Cc: Leif Lindholm 
Cc: Ard Biesheuvel 
Cc: Sami Mujawar 
Cc: Vitaly Cheptsov 
---
 ArmPkg/Include/AsmMacroIoLibV8.h | 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/ArmPkg/Include/AsmMacroIoLibV8.h b/ArmPkg/Include/AsmMacroIoLibV8.h
index 135aaeca5d0b..919edc70384d 100644
--- a/ArmPkg/Include/AsmMacroIoLibV8.h
+++ b/ArmPkg/Include/AsmMacroIoLibV8.h
@@ -34,15 +34,29 @@
 cbnz   SAFE_XREG, 1f;\
 b  .;// We should never get here
 
-#define _ASM_FUNC(Name, Section)\
-  .global   Name  ; \
-  .section  #Section, "ax"; \
-  .type Name, %function   ; \
+#define _ASM_FUNC_HDR(Name, Section) \
+  .global   Name   ; \
+  .section  #Section, "ax" ; \
+  .type Name, %function
+
+#define _ASM_FUNC_FTR(Name) \
   Name:   ; \
   AARCH64_BTI(c)
 
+#define _ASM_FUNC(Name, Section)\
+  _ASM_FUNC_HDR(Name, Section); \
+  _ASM_FUNC_FTR(Name)
+
+#define _ASM_FUNC_ALIGN(Name, Section, Align)   \
+  _ASM_FUNC_HDR(Name, Section); \
+  .balign Align   ; \
+  _ASM_FUNC_FTR(Name)
+
 #define ASM_FUNC(Name)  _ASM_FUNC(ASM_PFX(Name), .text. ## Name)
 
+#define ASM_FUNC_ALIGN(Name, Align)  \
+  _ASM_FUNC_ALIGN(ASM_PFX(Name), .text. ## Name, Align)
+
 #define MOV32(Reg, Val)   \
   movz  Reg, (Val) >> 16, lsl #16   ; \
   movk  Reg, (Val) & 0x
-- 
2.40.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103117): https://edk2.groups.io/g/devel/message/103117
Mute This Topic: https://groups.io/mt/98325898/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Role of DLINK2_FLAGS and link-time option consistency

[Marvin Häuser]

> On 7. Apr 2023, at 14:00, Ard Biesheuvel  wrote:
> 
> (cc Ray)
> 
> On Fri, 7 Apr 2023 at 13:05, Marvin Häuser  wrote:
>> 
>> 
>> On 7. Apr 2023, at 13:02, Ard Biesheuvel  wrote:
>> 
>> IIRC DLINK2_FLAGS is mainly being used (in the GCC case) for passing
>> options that need to come after previous occurrences of the same
>> option with a different value, in order for the DLINK2 one to take
>> precedence.
>> 
>> 
>> Thanks for the quick response! By that rationale, it should probably apply 
>> to the 16-bit ASM linker step as well and the rest can remain as-is (modulo 
>> that weird CLANGDWARF optimization situation)?
>> 
> 
> That build rule seems like dead code to me: we don't have any such
> files in the repo

It appears there still is one file remaining:
https://github.com/tianocore/edk2/blob/797f526ae2a83811b0ccbde0138c65a9f137eba5/IntelFsp2Pkg/FspSecCore/Vtf0/Ia16/ResetVec.asm16

But it is not directly referenced, just included from a file with a different 
extension:
https://github.com/tianocore/edk2/blob/797f526ae2a83811b0ccbde0138c65a9f137eba5/IntelFsp2Pkg/FspSecCore/Vtf0/ResetVectorCode.asm#L11

Being FSP, the most important consumers will probably be downstream, so I will 
not propose to remove it.


> , and no references to the .com output files that it
> produces, afaict
> 
> (The last uses were removed in 1a48fda5315433661c2f3039a30aea5916c22267)
> 
> Maybe Ray can comment?

Ping? :)

Best regards,
Marvin



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103040): https://edk2.groups.io/g/devel/message/103040
Mute This Topic: https://groups.io/mt/98121970/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 01/10] BaseTools: add BASETOOLS define

[Marvin Häuser]

> On 14. Apr 2023, at 16:57, Ard Biesheuvel  wrote:
> 
> On Fri, 14 Apr 2023 at 16:37, Gerd Hoffmann  wrote:
>> 
>>> On Fri, Apr 14, 2023 at 12:29:21PM +, Marvin Häuser wrote:
>>> Hi Gerd,
>>> 
>>> Thanks for your effort!
>>> 
>>> Sorry, but I *really* dislike this “BASETOOLS” notion. There might be
>>> external tools that also want to use the header (like we do with AUDK)
>>> and also edk2 supports host-based unit tests. Imo the macro name
>>> should be generic, like “HOST_OS” or “USERLAND” or something (I’m bad
>>> with names :) ).
>> 
>> I'm certainly open to better naming ideas.
>> 
>> Underlying problem is that x64 ProcessorBind.h sets visibility to
>> hidden (anyone knows why?).
> 
> Yes. Toolchains tend to assume that you are generating code for a
> shared library once you enabled -fpic, and this results in lots of
> pointless indirections via the GOT.

Unless Safari search is borked (this indeed sometimes happens, I swear :-) ), 
-fpic is never passed, only -fpie. Should be mostly preemption, I guess.

> (This has to do with the
> copy-on-write footprint of shared library pages, as a GOT tends to
> gather all relocated quantities in a small region of memory. It is
> also related to ELF symbol preemption, which is another thing we don't
> give a ^$& about).
> 
> *However*, recent linkers do a better job getting rid of these GOT
> references at link time, once you pass the --relax option. On X64 in
> particular, there are special, relaxable GOTPCREL relocations now that
> will not result in to symbol references via the GOT if the linker
> notices that the symbol in question is never preempted and not
> provided externally.
> 
> So it is worth an experiment whether we still need this to begin with.
> 
>> Seems to be fine for freestanding binaries,
>> but when linking host tools against shared libraries things break.  For
>> some reason on ubuntu only, fedora works fine (both gcc and clang).
>> 
>> Automatically detecting freestanding vs. hosted would be even better,
>> anyone knows whenever there is a define which can be used for this?  At
>> least the gcc man page doesn't mention any ...
>> 
> 
> ard@gambale:~/linux-build-x86$ gcc -dM -E -o - -xc - <<< ""|grep HOSTED
> #define __STDC_HOSTED__ 1
> ard@gambale:~/linux-build-x86$ gcc -ffreestanding -dM -E -o - -xc -
> <<< ""|grep HOSTED
> #define __STDC_HOSTED__ 0

You are godsend. :)

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103009): https://edk2.groups.io/g/devel/message/103009
Mute This Topic: https://groups.io/mt/98257896/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 01/10] BaseTools: add BASETOOLS define

[Marvin Häuser]

> On 14. Apr 2023, at 16:37, Gerd Hoffmann  wrote:
> 
> On Fri, Apr 14, 2023 at 12:29:21PM +0000, Marvin Häuser wrote:
>> Hi Gerd,
>> 
>> Thanks for your effort!
>> 
>> Sorry, but I *really* dislike this “BASETOOLS” notion. There might be
>> external tools that also want to use the header (like we do with AUDK)
>> and also edk2 supports host-based unit tests. Imo the macro name
>> should be generic, like “HOST_OS” or “USERLAND” or something (I’m bad
>> with names :) ).
> 
> I'm certainly open to better naming ideas.
> 
> Underlying problem is that x64 ProcessorBind.h sets visibility to
> hidden (anyone knows why?).

It combats some awkward ELF symbol magic (I think preemption?) which would 
otherwise generate a GOT.

> Seems to be fine for freestanding binaries,
> but when linking host tools against shared libraries things break.  For
> some reason on ubuntu only, fedora works fine (both gcc and clang).

Our Ubuntu CI also works with a similar change in AUDK.

> 
> Automatically detecting freestanding vs. hosted would be even better,
> anyone knows whenever there is a define which can be used for this?  At
> least the gcc man page doesn't mention any ...

Don’t think so. :(

> 
>> Speaking of host-based unit tests, why are they not affected by the
>> issue?
> 
> Hmm, good question.  Maybe they are simple enough to not need libc
> symbols.  Guess I should try a full test build on ubuntu, not only
> building basetools.

Tyvm!

Best regards,
Marvin

> 
> take care,
>  Gerd
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103007): https://edk2.groups.io/g/devel/message/103007
Mute This Topic: https://groups.io/mt/98257896/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 01/10] BaseTools: add BASETOOLS define

[Marvin Häuser]

Hi Gerd,

Thanks for your effort!

Sorry, but I *really* dislike this “BASETOOLS” notion. There might be external 
tools that also want to use the header (like we do with AUDK) and also edk2 
supports host-based unit tests. Imo the macro name should be generic, like 
“HOST_OS” or “USERLAND” or something (I’m bad with names :) ).

Speaking of host-based unit tests, why are they not affected by the issue?

Best regards,
Marvin

> On 14. Apr 2023, at 10:03, Gerd Hoffmann  wrote:
> 
> Allows for BaseTools-specific tweaks in include files.
> 
> Signed-off-by: Gerd Hoffmann 
> ---
> BaseTools/Source/C/Makefiles/header.makefile | 3 +++
> BaseTools/Source/C/Makefiles/ms.common   | 4 ++--
> BaseTools/Source/C/VfrCompile/GNUmakefile| 1 +
> 3 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/BaseTools/Source/C/Makefiles/header.makefile 
> b/BaseTools/Source/C/Makefiles/header.makefile
> index 1bf003523baf..6b743fa297b0 100644
> --- a/BaseTools/Source/C/Makefiles/header.makefile
> +++ b/BaseTools/Source/C/Makefiles/header.makefile
> @@ -124,6 +124,9 @@ ifeq ($(DARWIN),Darwin)
> endif
> endif
> 
> +CFLAGS   += -D BASETOOLS=1
> +CXXFLAGS += -D BASETOOLS=1
> +
> # keep BUILD_OPTFLAGS last
> CFLAGS   += $(BUILD_OPTFLAGS)
> CXXFLAGS += $(BUILD_OPTFLAGS)
> diff --git a/BaseTools/Source/C/Makefiles/ms.common 
> b/BaseTools/Source/C/Makefiles/ms.common
> index 8391f10d5dd2..6fc4c6477847 100644
> --- a/BaseTools/Source/C/Makefiles/ms.common
> +++ b/BaseTools/Source/C/Makefiles/ms.common
> @@ -57,5 +57,5 @@ LINKER = $(LD)
> 
> INC = $(INC) -I . -I $(SOURCE_PATH)\Include -I $(ARCH_INCLUDE) -I 
> $(SOURCE_PATH)\Common
> 
> -CFLAGS = $(CFLAGS) /nologo /Z7 /c /O2 /MT /W4 /WX /D 
> _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE
> -CPPFLAGS = $(CPPFLAGS) /EHsc /nologo /Z7 /c /O2 /MT /D 
> _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE
> +CFLAGS = $(CFLAGS) /nologo /Z7 /c /O2 /MT /W4 /WX /D 
> _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D BASETOOLS
> +CPPFLAGS = $(CPPFLAGS) /EHsc /nologo /Z7 /c /O2 /MT /D 
> _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D BASETOOLS
> diff --git a/BaseTools/Source/C/VfrCompile/GNUmakefile 
> b/BaseTools/Source/C/VfrCompile/GNUmakefile
> index 7d5976649dba..f1abed11376e 100644
> --- a/BaseTools/Source/C/VfrCompile/GNUmakefile
> +++ b/BaseTools/Source/C/VfrCompile/GNUmakefile
> @@ -24,6 +24,7 @@ VFR_CPPFLAGS = -DPCCTS_USE_NAMESPACE_STD $(CPPFLAGS)
> endif
> # keep BUILD_OPTFLAGS last
> VFR_CXXFLAGS = $(BUILD_OPTFLAGS)
> +VFR_CXXFLAGS += -D BASETOOLS=1
> 
> # keep EXTRA_LDFLAGS last
> VFR_LFLAGS = $(EXTRA_LDFLAGS)
> -- 
> 2.39.2
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#103000): https://edk2.groups.io/g/devel/message/103000
Mute This Topic: https://groups.io/mt/98257896/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [RFC PATCH] Fix ArmReplaceLiveTranslationEntry alignment

[Marvin Häuser]

Good day everyone,

Sorry, but due to time constraints, I cannot immediately provide a proper 
patch. Would you mind checking this commit and commenting on whether it looks 
about right, so I can submit a proper patch for review some time this or next 
week? 
https://github.com/acidanthera/audk/commit/53f2af3ad5909e177818445cafed7bdb6aae9d97

With the proper patch, I will probably also include an ASSERT to make sure the 
alignment is actually checked.

The symptom is that late PEI may crash due to corrupted memory. This is due to 
the fact that ArmReplaceLiveTranslationEntry() is misaligned despite the 
requirement it may not cross page boundaries. The related .balign directive 
technically belongs to the previous section, as 
ArmReplaceLiveTranslationEntry() is moved to its own section via ASM_FUNC() 
macro *after' the directive appears. The directive also cannot nicely be placed 
after ASM_FUNC(), as that would mean the label may refer to the padding 
inserted to achieve said alignment. Hence, my solution is to introduce a 
separate macro.

Reproducers are here, I tested the last two stable tags:
https://github.com/mhaeuser/edk2/tree/arm_corruption-202211
https://github.com/mhaeuser/edk2/tree/arm_corruption-202302

... and identified the last commit it is reproducible with (my hack does not 
work on master):
https://github.com/mhaeuser/edk2/tree/arm_corruption-latest

The fact that the commits after that last branch work is mere luck, I just 
didn't want to bruteforce a new hack to trigger the issue. :)

To trigger the issue, build ArmVirtQemu/AARCH64 of any of those branches with 
GCC 12 (GCC5) and as DEBUG - GCC 11 and RELEASE/NOOPT do *not* trigger the 
issue as-is for me. If this doesn't work for you, you probably need to find a 
different hack to move the function across a page boundary. When starting the 
generated FD, I get a hang right when jumping to DxeIplPeim (its entry point is 
badly corrupted). Opening PeiCore in IDA, it's obvious said function is 
misaligned.

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102786): https://edk2.groups.io/g/devel/message/102786
Mute This Topic: https://groups.io/mt/98181675/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Role of DLINK2_FLAGS and link-time option consistency

[Marvin Häuser]

> On 7. Apr 2023, at 13:02, Ard Biesheuvel  wrote:
> 
> IIRC DLINK2_FLAGS is mainly being used (in the GCC case) for passing
> options that need to come after previous occurrences of the same
> option with a different value, in order for the DLINK2 one to take
> precedence.

Thanks for the quick response! By that rationale, it should probably apply to 
the 16-bit ASM linker step as well and the rest can remain as-is (modulo that 
weird CLANGDWARF optimization situation)?



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102683): https://edk2.groups.io/g/devel/message/102683
Mute This Topic: https://groups.io/mt/98121970/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] Role of DLINK2_FLAGS and link-time option consistency

[Marvin Häuser]

Good day everyone,

I've been asked to check the following patch: 
https://edk2.groups.io/g/devel/message/102168

One thing I noticed was that -no-pie is added to DLINK2_FLAGS, not DLINK_FLAGS. 
For MSVC toolchains, DLINK2_FLAGS is used for a separate linker step [1] 
introduced by [2], which makes sense to me. For GCC* and CLANG* toolchains, 
however, I haven't really been able to understand its purpose. Generally, 
DLINK_FLAGS and DLINK2_FLAGS are used as part of the same command in the same 
fashion, e.g., [3]. The main difference I've been able to spot is that the 
linker step for 16-bit ASM files omits DLINK2_FLAGS, but passes DLINK_FLAGS 
[4]. As MSVC uses a completely different linker flags macro for these files and 
XCODE* toolchains use the static linker, it appears to be the semantics of 
DLINK2_FLAGS are different between toolchains. That's my confusion for the 
consumer side, I don't understand the exact purpose, or why 16-bit ASM in 
particular is special here. DLINK2_FLAGS unfortunately seems to date back to 
the "Sync BaseTools" SVN commits, so I didn't find much with blame.

For the producer side, I noticed --pie is passed for DLINK_FLAGS [5]. On the 
other hand, the IA32 GCC definitions override this with -no-pie, but in 
DLINK2_FLAGS (which the patch in question replicates). It's my understand that 
this means the linker step for 16-bit ASM files will be the only linker step in 
a IA32 build process to use --pie, while all other steps will have the -no-pie 
override. For the current codebase, there probably isn't much of a difference. 
However, for a well-defined build behaviour, don't we need to pass mostly the 
same linker flags (at the very least PIE-related flags) to all steps in the 
process? Is there a particular reason for how things work the way they do right 
now?

I've also tried to look at which kinds of flags are generally passed as 
DLINK_FLAGS vs DLINK2_FLAGS. For some reason, the linker script and related 
symbol definitions generally seem to be passed via DLINK2_FLAGS [8] as per [9]. 
I could imagine a possible rationale could be selective replacement, where a 
platform can discard-and-reassign DLINK2 to pass its own linker script? I 
didn't find any evidence for this though. With Pedro, I discussed various other 
possibilities for what could be the rationale for DLINK2_FLAGS, but pretty much 
failed. One was, that one of them is meant to be passed to the linker directly, 
while the other was supposed to be passed to the compiler (CC may be DLINK, or 
they may be different). However, there is a very inconsistent usage of "-Wl" 
between the two and LD does not support -Wl.

Another (rather unrelated) oddity we noticed is that CLANGDWARF passes 
different optimization levels to CC and DLINK2 [8], which I'm neither sure 
whether it is safe to do so, nor whether CC's -Oz really does much now, 
considering LTO will be on.

Any help with understanding the current design is appreciated, thanks!

Best regards,
Marvin


[1]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/build_rule.template#L289-L291

[2]
https://github.com/tianocore/edk2/commit/578211b882fa99b1f7c82c3e5fd71510772c

[3]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/build_rule.template#L297

[4]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/build_rule.template#L503

[5]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/tools_def.template#L1858

[6]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/tools_def.template#L2303

[7]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/tools_def.template#L1859

[8]
https://github.com/tianocore/edk2/blob/cdf6ff1719a9453351baec4bd32fcfc30e9ceeac/BaseTools/Conf/tools_def.template#L2937-L2939

[9]
https://github.com/tianocore/edk2/commit/6d732bbbc2b0463f367ceca381cb1861d52cf735



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102679): https://edk2.groups.io/g/devel/message/102679
Mute This Topic: https://groups.io/mt/98121970/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v3 0/4] Enable BTI support in memory attributes table

[Marvin Häuser]

> On 4. Apr 2023, at 18:29, Ard Biesheuvel  wrote:
> On Tue, 4 Apr 2023 at 18:19, Marvin Häuser  wrote:
>> 
>> FWIW, Reviewed-by: Marvin Häuser 
>> 
>> An off-topic remark, but I find it ominous that when adding a hack like the 
>> DllCharacteristicsEx debug entry, the opportunity is not used to turn it 
>> into something that can be expanded in the future without introducing yet 
>> another hack like this (I know 31 more Bits look plenty now, but if an 
>> address, offset, or size will be needed… ouch).
> 
> It *can* be expanded in the future. The debug directory entry includes
> a size field, and once flags get defined that are not present, they
> just default to unset.

Ugh, I should have known this… sorry for my terrible remark! :)

Though doesn’t that mean the size field should ideally be checked in 3/4 (not 
regarding the extensibility point, but well-formedness of the data)?

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102502): https://edk2.groups.io/g/devel/message/102502
Mute This Topic: https://groups.io/mt/98062730/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v3 0/4] Enable BTI support in memory attributes table

[Marvin Häuser]

FWIW, Reviewed-by: Marvin Häuser 

An off-topic remark, but I find it ominous that when adding a hack like the 
DllCharacteristicsEx debug entry, the opportunity is not used to turn it into 
something that can be expanded in the future without introducing yet another 
hack like this (I know 31 more Bits look plenty now, but if an address, offset, 
or size will be needed… ouch).

Best regards,
Marvin

> On 4. Apr 2023, at 17:40, Ard Biesheuvel  wrote:
> Implement version 2 of the memory attributes table, which now contains a
> flag informing the OS whether or not code regions may be mapped with CFI
> mitigations such as IBT or BTI enabled.
> 
> This series covers the remaining parts after the AArch64 specific
> changes were merged:
> 
> - Update the BaseTools to emit the appropriate PE/COFF annotation when a
>  BTI/IBT compatible ELF executable is converted to PE/COFF
> - Take this PE/COFF annotation into account when populating the memory
>  attributes table in the DXE core
> 
> TODO:
> - X64 changes to make the code IBT compatible and emit the ELF note
> - Figure out how to generate such executables with native PE toolchains
> - Implement BTI/IBT enforcement at boot time - this is something I
>  intend to look into next.
> 
> Can be tested with the CLANG38 toolchain (both Clang compiler and LLD
> linker, version 3.8 or newer) with the following build options.
> 
> [BuildOptions]
>  GCC:*_*_AARCH64_PP_FLAGS = -mbranch-protection=bti
>  GCC:*_*_AARCH64_CC_FLAGS = -mbranch-protection=bti
>  GCC:*_*_AARCH64_DLINK_FLAGS = -fuse-ld=lld 
> -Wl,--no-relax,--no-pie,-z,bti-report=error
> 
> Changes since v2:
> - increase DllCharacteristicsEx field to 4 bytes
> - add Oliver's Rb
> 
> If no comments or objections have been raised by the end of the week, I
> will go ahead and merge this - thanks.
> 
> Cc: Michael Kinney 
> Cc: Liming Gao 
> Cc: Jiewen Yao 
> Cc: Michael Kubacki 
> Cc: Sean Brogan 
> Cc: Rebecca Cran 
> Cc: Leif Lindholm 
> Cc: Sami Mujawar 
> Cc: Taylor Beebe 
> Cc: Marvin Häuser 
> Cc: Bob Feng 
> Cc: Oliver Smith-Denny 
> 
> Ard Biesheuvel (4):
>  BaseTools/GenFw: Parse IBT/BTI support status from ELF note
>  BaseTools/GenFw: Add DllCharacteristicsEx field to debug data
>  MdePkg/PeCoffLib: Capture DLL characteristics fields in image context
>  MdeModulePkg: Enable forward edge CFI in mem attributes table
> 
> BaseTools/Source/C/GenFw/Elf64Convert.c   | 104 
> +---
> BaseTools/Source/C/GenFw/GenFw.c  |   3 +-
> BaseTools/Source/C/GenFw/elf_common.h |   9 ++
> BaseTools/Source/C/Include/IndustryStandard/PeImage.h |  13 ++-
> MdeModulePkg/Core/Dxe/DxeMain.h   |   2 +
> MdeModulePkg/Core/Dxe/Image/Image.c   |  10 ++
> MdeModulePkg/Core/Dxe/Misc/MemoryAttributesTable.c|   8 +-
> MdePkg/Include/IndustryStandard/PeImage.h |  13 ++-
> MdePkg/Include/Library/PeCoffLib.h|   6 ++
> MdePkg/Library/BasePeCoffLib/BasePeCoff.c |  46 ++---
> 10 files changed, 186 insertions(+), 28 deletions(-)
> 
> -- 
> 2.39.2



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102498): https://edk2.groups.io/g/devel/message/102498
Mute This Topic: https://groups.io/mt/98062730/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Linker scripts use of "-z common-page-size=0x20" etc.

[Marvin Häuser]

> On 4. Apr 2023, at 16:03, Ard Biesheuvel  wrote:
> 
> On Tue, 4 Apr 2023 at 13:10, Rebecca Cran  > wrote:
>> 
>> On 4/4/23 1:22 AM, Ard Biesheuvel wrote:
>>> On Mon, 3 Apr 2023 at 22:33, Rebecca Cran >> > wrote:
 As part of my work on the toolchain definitions, I've come across a
 situation where ld.lld fails to align sections correctly, due to it
 being invoked via clang with the '-n' option, which causes GenFw to fail
 with "Section address not aligned to its own alignment.".
 
>>> Stupid question: if it breaks stuff, why do you use -n ?
>> 
>> As far as I can see, clang adds it automatically when it invokes ld.lld.
>> 
> 
> Ah right, fair enough.
> 
 The following messages are printed:
 
 
 ld.lld: warning: -z common-page-size set, but paging disabled by omagic
 or nmagic ld.lld: warning: address (0x558) of section .data is not a
 multiple of alignment (16)
 
 I tracked the problem down to GccBase.lds and ClangBase.lds, which have:
 
 /* * The alignment of the .data section should be less than or equal to
 the * alignment of the .text section. This ensures that the relative
 offset * between these sections is the same in the ELF and the PE/COFF
 versions of * this binary. */ .data ALIGN(ALIGNOF(.text)) :
 ALIGN(CONSTANT(COMMONPAGESIZE)) { *(.data .data.* .gnu.linkonce.d.*)
 *(.bss .bss.*) }
 
 I can work around the problem by removing "ALIGN(ALIGNOF(.text)", but
 I'm wondering if our use of COMMONPAGESIZE/MAXPAGESIZE is correct.
 
>>> What do you mean by 'correct'? The intent is clearly to declare the
>>> mapping granule size, and for SEC/PEI binaries that execute in place
>>> from flash, the MMU page size is not the most useful quantum here.
>> 
>> On Discord, I think Marvin was saying that we shouldn't be using
>> COMMONPAGESIZE, but MAXPAGESIZE.
>> 
> 
> I don't have a preference for one over the other, although I fail to
> see how this fixes the -n problem above.
> 
>> Since .text is aligned to COMMONPAGESIZE, I think I can remove the
>> ALIGN(ALIGNOF(.text)) and keep the ALIGN(CONSTANT(COMMONPAGESIZE))?
>> 
> 
> No. The alignment of .text could exceed COMMONPAGESIZE; this happens
> e.g., with AArch64 vector tables which are 2k aligned, when they are
> emitted into a SEC or PEIM module which only has 32 byte alignment by
> default.
> 
> In this case, the PE/COFF section alignment will be 2k, and so .data
> should start at a 2k aligned address as well.
> 
>> 
>>> 'Page size' is highly context specific, and toolchain people are
>>> (imho) usually quite quick to call something abuse if it does not
>>> match their narrow definition of how a compiler or linker should be
>>> used. For the same reason, it has been so difficult to get a compiler
>>> to understand that the desire for position independent code does not
>>> imply that we want GOTs, or care about ELF symbol preemption or
>>> copy-on-write footprint of relocatable segments. In general, the bare
>>> metal use case (which includes EFI) is quite poorly understood by many
>>> people working on toolchains.
>>> 
 I'm wondering what the correct approach is here: should we do something
 similar to how we set PECOFF_HEADER_SIZE and define a SECTION_ALIGNMENT
 symbol?
>>> We cannot, that is the reason for using the page size switches here:
>>> using a symbol to set the location pointer is fine, but using a symbol
>>> to set the alignment of a section is not.
>>> 
 Or, as discussed on Discord should we just use
 CONSTANT(MAXPAGESIZE) and ignore how it's normally used to specify the
 maximum allowable page size for an executable?
 
>>> Note that (when I last checked), the only effect of setting -z
>>> xxx-page-size is that those macros assume the associated value in the
>>> linker script. Nothing else in the linker changes behavior (with the
>>> exception of the warning you are seeing)
>>> 
>>> So claiming abuse because the provided value does not match the page
>>> size of an OS that might also run on the same system is strenuous, and
>>> I think our use of it is fine. AIUI, the reason for having
>>> ClangBase.lds in addition to GccBase.lds is the fact that LLD does not
>>> support common but only max page size, so I think it should be fine to
>>> merge the two, and use max-page-size everywhere.
>> 
>> Thanks for the explanation. This is probably an error on my part, but
>> when I tried to use max-page-size everywhere, the AARCH64 build of
>> ArmVirtQemu failed, with GenFw saying "AARCH64 small code model requires
>> identical ELF and PE/COFF section offsets modulo 4 KB."
>> 
> 
> Let me know if you manage to reproduce that - I'd like to understand
> what exactly is happening there.
> 
>> LLD added support for common-page-size in 2019
>> (https://reviews.llvm.org/D56205), so I think we can keep the current
>> usage of common and max page size and just combine GccBase.lds and
>> 

Re: [edk2-devel] [PATCH v4 0/6] UefiCpuPkg, OvmfPkg: Simplify CpuExceptionHandlerLib

[Marvin Häuser]

FWIW, Reviewed-by: Marvin Häuser 

> On 3. Apr 2023, at 16:29, Ard Biesheuvel  wrote:
> 
> We have a special version of CpuExceptionHandlerLib for XCODE5, whose
> linker (LLD) does not permit absolute symbol references in read-only
> sections.
> 
> Instead of fixing this up at runtime for all toolchains (which is done
> by writing the fixed up values to the .text section, which we'd prefer
> to avoid), tweak the SEC/PEI version so it does not need this, and
> update the remaining versions to only incorporate this logic when using
> the XCODE toolchain.
> 
> Changes since v3:
> - Add another patch to move PeiCpuExceptionHandlerLib to SEC/PEI version
>  of the NASM source file
> - Improve commit logs
> 
> Changes since v2:
> -  As Marvin points out, using '-read_only_relocs suppress' with the X64
>   XCODE linker is a terrible idea, as it corrupts the resulting PE
>   binaries, so instead, let's do the following:
>   . tweak the SEC/PEI version of the library so the relocs are emitted
> into .data when using XCODE;
>   . tweak the other versions so the runtime fixups are only done when
> using XCODE
> - add acks from Jiewen and Ray
> 
> Changes since v1:
> - pass linker switches to permit absolute relocations in read-only
>  regions, and keep all code in .text
> 
> Cc: "Ni, Ray" 
> Cc: Andrew Fish 
> Cc: "Kinney, Michael D" 
> Cc: "Liu, Zhiguang" 
> Cc: Rebecca Cran 
> Cc: Tom Lendacky 
> Cc: Marvin Häuser 
> 
> Ard Biesheuvel (6):
>  BaseTools/tools_def CLANGDWARF: Permit text relocations
>  UefiCpuPkg/CpuExceptionHandlerLib: Use single SEC/PEI version
>  UefiCpuPkg/PeiCpuExceptionHandlerLib: Use SEC/PEI specific asm
>component
>  UefiCpuPkg/CpuExceptionHandlerLib: Make runtime fixups XCODE-only
>  OvmfPkg: Drop special Xcode5 version of exception handler library
>  UefiCpuPkg/CpuExceptionHandlerLib: Drop special XCODE5 version
> 
> BaseTools/Conf/tools_def.template 
>|   2 +-
> OvmfPkg/AmdSev/AmdSevX64.dsc  
>|   4 -
> OvmfPkg/CloudHv/CloudHvX64.dsc
>|   4 -
> OvmfPkg/IntelTdx/IntelTdxX64.dsc  
>|   4 -
> OvmfPkg/Microvm/MicrovmX64.dsc
>|   4 -
> OvmfPkg/OvmfPkgIa32.dsc   
>|   4 -
> OvmfPkg/OvmfPkgIa32X64.dsc
>|   4 -
> OvmfPkg/OvmfPkgX64.dsc
>|   4 -
> OvmfPkg/OvmfXen.dsc   
>|   4 -
> UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.inf   
>|   5 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandlerLib.inf   
>|   4 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuExceptionHandlerLib.inf
>|   4 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandlerLib.inf   
>|   4 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHandlerAsm.nasm
>| 116 +---
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/{Xcode5ExceptionHandlerAsm.nasm 
> => SecPeiExceptionHandlerAsm.nasm} | 108 +++---
> UefiCpuPkg/Library/CpuExceptionHandlerLib/Xcode5SecPeiCpuExceptionHandlerLib.inf
>  |  65 ---
> UefiCpuPkg/Library/CpuExceptionHandlerLib/Xcode5SecPeiCpuExceptionHandlerLib.uni
>  |  18 ---
> UefiCpuPkg/UefiCpuPkg.dsc 
>|   7 --
> 18 files changed, 133 insertions(+), 232 deletions(-)
> rename 
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/{Xcode5ExceptionHandlerAsm.nasm 
> => SecPeiExceptionHandlerAsm.nasm} (70%)
> delete mode 100644 
> UefiCpuPkg/Library/CpuExceptionHandlerLib/Xcode5SecPeiCpuExceptionHandlerLib.inf
> delete mode 100644 
> UefiCpuPkg/Library/CpuExceptionHandlerLib/Xcode5SecPeiCpuExceptionHandlerLib.uni
> 
> -- 
> 2.39.2
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102471): https://edk2.groups.io/g/devel/message/102471
Mute This Topic: https://groups.io/mt/98036435/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Linker scripts use of "-z common-page-size=0x20" etc.

[Marvin Häuser]

On 4. Apr 2023, at 09:23, Ard Biesheuvel  wrote:On Mon, 3 Apr 2023 at 22:33, Rebecca Cran  wrote:As part of my work on the toolchain definitions, I've come across asituation where ld.lld fails to align sections correctly, due to itbeing invoked via clang with the '-n' option, which causes GenFw to failwith "Section address not aligned to its own alignment.".Stupid question: if it breaks stuff, why do you use -n ?The following messages are printed:ld.lld: warning: -z common-page-size set, but paging disabled by omagicor nmagic ld.lld: warning: address (0x558) of section .data is not amultiple of alignment (16)I tracked the problem down to GccBase.lds and ClangBase.lds, which have:/* * The alignment of the .data section should be less than or equal tothe * alignment of the .text section. This ensures that the relativeoffset * between these sections is the same in the ELF and the PE/COFFversions of * this binary. */ .data ALIGN(ALIGNOF(.text)) :ALIGN(CONSTANT(COMMONPAGESIZE)) { *(.data .data.* .gnu.linkonce.d.*)*(.bss .bss.*) }I can work around the problem by removing "ALIGN(ALIGNOF(.text)", butI'm wondering if our use of COMMONPAGESIZE/MAXPAGESIZE is correct.What do you mean by 'correct'? The intent is clearly to declare themapping granule size, and for SEC/PEI binaries that execute in placefrom flash, the MMU page size is not the most useful quantum here.We pass in a value of 0x20, 0x40 or 0x1000 to "-z common-page-size" with0x20 being the most common value.Given the page size of the target will never be 32 bytes, the followingcomment on https://reviews.llvm.org/D61688 makes sense:"There is at least one linkerscript in Tianocore edk2 that (ab)uses -n-z common-page-size=0x20 to use CONSTANT(COMMONPAGESIZE) as if it were apreprocessor macro set with -D in the compiler. The usual approach tothis is to pre-process the linkerscript."'Page size' is highly context specific, and toolchain people are(imho) usually quite quick to call something abuse if it does notmatch their narrow definition of how a compiler or linker should beused. For the same reason, it has been so difficult to get a compilerto understand that the desire for position independent code does notimply that we want GOTs, or care about ELF symbol preemption orcopy-on-write footprint of relocatable segments. In general, the baremetal use case (which includes EFI) is quite poorly understood by manypeople working on toolchains.I'm wondering what the correct approach is here: should we do somethingsimilar to how we set PECOFF_HEADER_SIZE and define a SECTION_ALIGNMENTsymbol?We cannot, that is the reason for using the page size switches here:using a symbol to set the location pointer is fine, but using a symbolto set the alignment of a section is not.Or, as discussed on Discord should we just useCONSTANT(MAXPAGESIZE) and ignore how it's normally used to specify themaximum allowable page size for an executable?Note that (when I last checked), the only effect of setting -zxxx-page-size is that those macros assume the associated value in thelinker script. Nothing else in the linker changes behavior (with theexception of the warning you are seeing)Probably, but they still have conventional meanings (which is why LLVM implemented one but not the other). common-page-size is supposed to be the page size you are most likely to encounter with your product [1]. Apparently, it’s supposed to be used for link-time optimisations, such as internal data alignment (e.g., DATA_SEGMENT_ALIGN [1]). Thus using max-page-size for both GCC and LLVM makes sense to me.[1] Untitleddocs.adacore.comSo claiming abuse because the provided value does not match the pagesize of an OS that might also run on the same system is strenuous, andI think our use of it is fine. AIUI, the reason for havingClangBase.lds in addition to GccBase.lds is the fact that LLD does notsupport common but only max page size, so I think it should be fine tomerge the two, and use max-page-size everywhere.If that works, +1Best regards,Marvin


_._,_._,_



Groups.io Links:


  
You receive all messages sent to this group.
  
  



View/Reply Online (#102464) |


  

|

  Mute This Topic

| New Topic




Your Subscription |
Contact Group Owner |

Unsubscribe

 [arch...@mail-archive.com]
_._,_._,_

Re: [edk2-devel] Linker scripts use of "-z common-page-size=0x20" etc.

[Marvin Häuser]

> On 3. Apr 2023, at 22:33, Rebecca Cran  wrote:
> 
> As part of my work on the toolchain definitions, I've come across a 
> situation where ld.lld fails to align sections correctly, due to it being 
> invoked via clang with the '-n' option, which causes GenFw to fail with 
> "Section address not aligned to its own alignment.".
> 
> The following messages are printed:
> 
> 
> ld.lld: warning: -z common-page-size set, but paging disabled by omagic or 
> nmagic ld.lld: warning: address (0x558) of section .data is not a multiple of 
> alignment (16)
> 
> I tracked the problem down to GccBase.lds and ClangBase.lds, which have:
> 
> /* * The alignment of the .data section should be less than or equal to the * 
> alignment of the .text section. This ensures that the relative offset * 
> between these sections is the same in the ELF and the PE/COFF versions of * 
> this binary. */ .data ALIGN(ALIGNOF(.text)) : ALIGN(CONSTANT(COMMONPAGESIZE)) 
> { *(.data .data.* .gnu.linkonce.d.*) *(.bss .bss.*) }
> 
> I can work around the problem by removing "ALIGN(ALIGNOF(.text)", but I'm 
> wondering if our use of COMMONPAGESIZE/MAXPAGESIZE is correct.
> 
> 
> We pass in a value of 0x20, 0x40 or 0x1000 to "-z common-page-size" with 0x20 
> being the most common value.
> 
> Given the page size of the target will never be 32 bytes, the following 
> comment on https://reviews.llvm.org/D61688 makes sense:
> 
> 
> "There is at least one linkerscript in Tianocore edk2 that (ab)uses -n -z 
> common-page-size=0x20 to use CONSTANT(COMMONPAGESIZE) as if it were a 
> preprocessor macro set with -D in the compiler. The usual approach to this is 
> to pre-process the linkerscript."
> 
> 
> I'm wondering what the correct approach is here: should we do something 
> similar to how we set PECOFF_HEADER_SIZE and define a SECTION_ALIGNMENT 
> symbol? Or, as discussed on Discord should we just use CONSTANT(MAXPAGESIZE) 
> and ignore how it's normally used to specify the maximum allowable page size 
> for an executable?

That last part is actually not ignoring the use-case, that *is* our use-case. 
The terminology again is very OS-oriented, it’s important to know that 
generally OSes will fail to load binaries that are aligned less than the 
platform page size, as they cannot apply permissions (and probably also some 
implementation details of mmap / VM / whatever). That’s why the maximum page 
size you’re realistic to encounter is exactly the image segment alignment (by 
hardware convention, this is a power of two, thus a strict alignment satisfies 
all less strict alignment constraints).

The common page size on the other hand appears to be an optimization, for which 
you specify the most common page size (e.g., you may target AARCH64 which may 
require 16 KB alignment, but most of your targets will have only 4 KB pages), 
which the compiler will use to optimize the binary for the common targets. I 
have no idea why this is even used. There also were discussions on LLVM 
platforms that it should be avoided.

The naive approach would be to just use max-page-size, drop all references to 
common-page-size, and align all ELF sections that will be converted to PE 
sections by max-page-size. But I’m sure there’s some ancient workaround / 
compiler bug / edge use case / portability or whatever reason why 
common-page-size was used. :)
(CC Leif for related experience.)

edk2 generally sets this to a low value to save SPI (and possibly RAM) space, 
as nothing in the stack enforces memory protection ( :( ). I’m not sure why 
there’s both 32 and 64 Bytes, but I could imagine it’s some GNU ABI thing where 
some type of some arch actually requires this alignment, or maybe there’s 
different rules for global variables. A text segment must at least satisfy the 
maximum instruction alignment constraint (this may be a thing with, e.g., 
normalised instruction lengths), while data segments must satisfy at least the 
maximum data alignment constraint (which might usually be some large float? Not 
sure). 4 KB is used when memory protection is needed (usually RT drivers, as 
they’re mapped into the OS environment), but AARCH64 may actually require 16 KB 
(e.g. Apple A chips didn’t even support less for a while).

Best regards,
Marvin

> 
> 
> -- 
> Rebecca Cran
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102441): https://edk2.groups.io/g/devel/message/102441
Mute This Topic: https://groups.io/mt/98045342/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

> On 31. Mar 2023, at 16:58, Rebecca Cran  wrote:
> 
> On 3/31/23 2:29 AM, Marvin Häuser wrote:
>> I agree if there’s an actual plan on doing that. We depend on XCODE5 
>> downstream, but I think it would literally be easier for us if the upstream 
>> version was dropped than rebasing against hacks that our slightly modded 
>> variant does not require.
>> 
>> Cc Andrew and Rebecca. I don’t know anyone else who might still be using 
>> XCODE5. Any objections to dropping it? If so, any plans to pick up my 
>> proposed changes instead?
> 
> I've only been using XCODE5 to test if it still works. I do all of my 
> development work on Linux or FreeBSD.

I checked the list and most of the traffic regarding XCODE5 is basically Andrew 
and you. So if you don't need this toolchain...

@Andrew Would you agree it's fair for Apple to maintain XCODE5 and its oddities 
downstream? I know someone mentioned the ARM transition, but I think you still 
support UEFI for HV things, don't you? The alternative would be we revamp the 
upstream XCODE5 toolchain and mtoc with our changes from AUDK and ocmtoc, but 
this isn't feasible without strong support from Apple (our previous patches to 
mtoc were ignored). I don't think the burden easily fixable XCODE5 oddities put 
on the general codebase are acceptable going forward. Thanks!

Best regards,
Marvin

> 
> 
> -- 
> 
> Rebecca Cran
> 
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102320): https://edk2.groups.io/g/devel/message/102320
Mute This Topic: https://groups.io/mt/97960758/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix CLANGDWARF_IA32_X64

[Marvin Häuser]

> On 31. Mar 2023, at 16:41, Ni, Ray  wrote:
> 
> Why ELF header overflows into .text section?

That's a good question, isn't it? :)

>From what I can see, these binaries don't pass post-processing like GenFw or 
>such. GCC (and I think thus CLANGDWARF?) gets an extra objcopy step as part of 
>linking [2], but the arguments are empty [3] and thus should be no-op (I 
>hope?).

I suppose potential candidates are:

1) A bug in the LLD linker used by CLANGDWARF for IA32 and X64. That would be 
very surprising to me, especially as no other platform reported issues and LLD 
is well-established. But who knows, generally ELFs will have large alignment 
values compared to the 64 Bytes used by edk2.

2) A bug in llvm-objcopy used by UniversalPayloadBuild.py [1]. I'm honestly 
unfamiliar with objcopy variants and their quality/reliability.

3) A bug in the llvm-objcopy or CLANGDWARF tools_def commands on the edk2 side 
of things.

Some may disagree, but I would reduce 3) to either 1) or 2). I think even if 
the commands malformed and this causes the overflow, I believe LLD or objcopy 
should issue a warning regardless.

As I have no way to reproduce the issue, I cannot really help further, sorry.

Best regards,
Marvin

[1]
https://github.com/tianocore/edk2/blob/b08a19eae28e76fb5a296a604c27d06fab29b08a/UefiPayloadPkg/UniversalPayloadBuild.py#L163-L183

[2]
https://github.com/tianocore/edk2/blob/b08a19eae28e76fb5a296a604c27d06fab29b08a/BaseTools/Conf/build_rule.template#L298

[3]
https://github.com/tianocore/edk2/blob/b08a19eae28e76fb5a296a604c27d06fab29b08a/BaseTools/Conf/tools_def.template#L2895
https://github.com/tianocore/edk2/blob/b08a19eae28e76fb5a296a604c27d06fab29b08a/BaseTools/Conf/tools_def.template#L2931

> 
>> -Original Message-
>> From: Patrick Rudolph 
>> Sent: Friday, March 17, 2023 10:06 PM
>> Cc: devel@edk2.groups.io; Dong, Guo ; Guo, Gua
>> ; Lu, James ; Ni, Ray
>> ; mhaeu...@posteo.de; a...@kernel.org
>> Subject: [PATCH 2/3] BaseTools/Conf/tools_def: Fix
>> CLANGDWARF_IA32_X64
>> 
>> Drop the "-z max-page-size=0x40" option as it causes the ELF
>> header to overflow into the .text section, causing undefined
>> behaviour.
>> 
>> With high optimization level it corrupts essential code and
>> the binary would crash. It might work with low optimization
>> level though. As the default is to use Oz and LTO, it always
>> crashes.
>> 
>> Test:
>> The ELF generated by
>> 'python UefiPayloadPkg/UniversalPayloadBuild.py -a IA32' boots.
>> 
>> Signed-off-by: Patrick Rudolph 
>> Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=4357
>> ---
>> BaseTools/Conf/tools_def.template | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/BaseTools/Conf/tools_def.template
>> b/BaseTools/Conf/tools_def.template
>> index 9b59bd75c3..0c584ab390 100755
>> --- a/BaseTools/Conf/tools_def.template
>> +++ b/BaseTools/Conf/tools_def.template
>> @@ -2866,7 +2866,7 @@ DEFINE CLANGDWARF_X64_PREFIX=
>> ENV(CLANG_BIN)
>> 
>> 
>> # LLVM/CLANG doesn't support -n link option. So, it can't share the same
>> IA32_X64_DLINK_COMMON flag.
>> 
>> # LLVM/CLANG doesn't support common page size. So, it can't share the
>> same GccBase.lds script.
>> 
>> -DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib -Wl,-q,--gc-
>> sections -z max-page-size=0x40
>> 
>> +DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib -Wl,-q,--
>> gc-sections
>> 
>> DEFINE CLANGDWARF_DLINK2_FLAGS_COMMON = -Wl,--
>> script=$(EDK_TOOLS_PATH)/Scripts/ClangBase.lds
>> 
>> DEFINE CLANGDWARF_IA32_X64_ASLDLINK_FLAGS =
>> DEF(CLANGDWARF_IA32_X64_DLINK_COMMON) -Wl,--
>> defsym=PECOFF_HEADER_SIZE=0
>> DEF(CLANGDWARF_DLINK2_FLAGS_COMMON) -Wl,--
>> entry,ReferenceAcpiTable -u ReferenceAcpiTable
>> 
>> DEFINE CLANGDWARF_IA32_X64_DLINK_FLAGS=
>> DEF(CLANGDWARF_IA32_X64_DLINK_COMMON) -Wl,--
>> entry,$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT) -Wl,-
>> Map,$(DEST_DIR_DEBUG)/$(BASE_NAME).map,--whole-archive
>> 
>> --
>> 2.39.1
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102318): https://edk2.groups.io/g/devel/message/102318
Mute This Topic: https://groups.io/mt/97673649/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v3 0/5] UefiCpuPkg, OvmfPkg: Simplify CpuExceptionHandlerLib

[Marvin Häuser]

> On 31. Mar 2023, at 16:39, Ni, Ray  wrote:
> 
> 
> 
>> -Original Message-
>> From: devel@edk2.groups.io  On Behalf Of Marvin
>> Häuser
>> Sent: Friday, March 31, 2023 7:10 PM
>> To: Ard Biesheuvel 
>> Cc: Ni, Ray ; devel@edk2.groups.io; Andrew Fish
>> ; Kinney, Michael D ; Liu,
>> Zhiguang ; Rebecca Cran ;
>> Tom Lendacky 
>> Subject: Re: [edk2-devel] [RFT PATCH v3 0/5] UefiCpuPkg, OvmfPkg: Simplify
>> CpuExceptionHandlerLib
>> 
>> 
>>> On 31. Mar 2023, at 13:03, Ard Biesheuvel  wrote:
>>> 
>>> On Fri, 31 Mar 2023 at 12:41, Marvin Häuser  wrote:
>>>> 
>>>> Hi Ray,
>>>> 
>>>>>> On 31. Mar 2023, at 12:09, Ni, Ray  wrote:
>>>>> 
>>>>> Ard,
>>>>> What does "-read_only_relocs suppress" control?
>>>> 
>>>> It controls whether relocs that target read-only segments yield a build
>> error or not. I think lld uses “-z notext”.
>>>> 
>>>>> Linker doesn't produce relocation entries that modifies .text section
>> silently
>>>>> so the final .text just cannot run at all?
>>>> 
>>>> Could you please rephrase? I’m not sure I understand, but I think it’s
>> important everyone understands the issues at play to make a good judgment
>> call.
>>>> 
>>> 
>>> As *I* understood it, it means suppress the *warning* not suppress the
>>> *relocation*
> 
> What the meaning of "suppress relocation"?

The option naming is just a bit odd, it suppresses the warning about 
relocations to read-only segments, not the relocations themselves.

> Why the final binaries are not executable?

I explained that here: https://edk2.groups.io/g/devel/message/102219

TL;dr: Relocations are relative to the first writable segment (thus usually 
__DATA), so relocations to preceding segments (usually __TEXT) will underflow 
and thus get corrupted offsets.

Best regards,
Marvin

> 
>> 
>> Correct.
>> 
>>> 
>>> But the resulting binaries are broken, so it doesn't really matter.
>> 
>> 
>> 
>> 
>> 
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102315): https://edk2.groups.io/g/devel/message/102315
Mute This Topic: https://groups.io/mt/97969646/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v3 0/5] UefiCpuPkg, OvmfPkg: Simplify CpuExceptionHandlerLib

[Marvin Häuser]

> On 31. Mar 2023, at 13:03, Ard Biesheuvel  wrote:
> 
> On Fri, 31 Mar 2023 at 12:41, Marvin Häuser  wrote:
>> 
>> Hi Ray,
>> 
>>>> On 31. Mar 2023, at 12:09, Ni, Ray  wrote:
>>> 
>>> Ard,
>>> What does "-read_only_relocs suppress" control?
>> 
>> It controls whether relocs that target read-only segments yield a build 
>> error or not. I think lld uses “-z notext”.
>> 
>>> Linker doesn't produce relocation entries that modifies .text section 
>>> silently
>>> so the final .text just cannot run at all?
>> 
>> Could you please rephrase? I’m not sure I understand, but I think it’s 
>> important everyone understands the issues at play to make a good judgment 
>> call.
>> 
> 
> As *I* understood it, it means suppress the *warning* not suppress the
> *relocation*

Correct.

> 
> But the resulting binaries are broken, so it doesn't really matter.



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102300): https://edk2.groups.io/g/devel/message/102300
Mute This Topic: https://groups.io/mt/97969646/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

Hi Gerd,

> On 31. Mar 2023, at 12:53, Gerd Hoffmann  wrote:
> 
>   Hi,
> 
>>> However, those issues might have been fixed and it’s not impossible
>>> Vitaly will give it another try eventually. In any case, I think our
>>> downstream variant of XCODE5 doesn’t require any level of special
>>> care, so it doesn’t really matter to us.
>>> 
>>> (Another thing to consider is despite the bugs are fixed, mtoc has a
>>> much higher overall code quality and more safety checks than GenFw,
>>> which is used for CLANGDWARF.)
>>> 
>>> The upstream toolchain has no future in my opinion, as mtoc has been
>>> deprecated and already failed to compile certain things (like it
>>> lacked Standalone MM types). The reason it still “worked” was
>>> because homebrew silently shipped a variant with a subset of our
>>> ocmtoc patches. So as I see it, taking our changes or dropping it
>>> entirely are the only sane options, even regardless of this
>>> particular issue you’re trying to fix. Personally, I have no
>>> preference.
>> 
>> I think both GenFw and mtoc are horrible hacks that should be phased
>> out once we can - with good cross-architecture Clang support for
>> native PE binaries, I'd hope macOS could move to CLANGPDB for all
>> targets.
> 
> What is the difference between CLANGPDB and CLANGDWARF?  Just the debug
> info format?

DWARF generates an ELF (which is utilized by UefiPayloadPkg) and PDB generates 
a PE directly.

Best regards,
Marvin

> 
> What is the support status?  Is CLANGDWARF expected to build edk2 on all
> platforms?  Including cross-builds?  Or will that work only after
> Rebecca's toolchain fix/cleanup series being merged?
> 
> Should we eventually switch from gcc to clang on linux too?
> 
> take care,
>  Gerd
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102298): https://edk2.groups.io/g/devel/message/102298
Mute This Topic: https://groups.io/mt/97960758/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix CLANGDWARF_IA32_X64

[Marvin Häuser]

Liming,

Platform maintainers can decide whether or not they want to combat *binary 
corruption*? Excuse me, but what the bloody hell? This needs a root cause 
analysis for which part of the stack silently borks us and not an “oh, if 
something fails, well, copy and paste this workaround… maybe”. If you give me 
an efficient way to reproduce it, I’ll do it.

Best regards,
Marvin

> On 31. Mar 2023, at 06:54, gaoliming  wrote:
> 
> Marvin:
> Platform developer can decide how to configure this option in their DSC file 
> to resolve their problem. This is one option for the platform developer. 
> 
> Thanks
> Liming
>> -邮件原件-
>> 发件人: devel@edk2.groups.io  代表 Marvin
>> H?user
>> 发送时间: 2023年3月28日 19:26
>> 收件人: gaoliming 
>> 抄送: devel@edk2.groups.io; patrick.rudo...@9elements.com;
>> guo.d...@intel.com; gua@intel.com; james...@intel.com;
>> ray...@intel.com; a...@kernel.org
>> 主题: Re: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix
>> CLANGDWARF_IA32_X64
>> 
>> Hi all,
>> 
 On 28. Mar 2023, at 07:38, gaoliming  wrote:
>>> Patrick:
>>> I prefer to override this option in DSC instead of the change in
>>> tools_def.txt.
>> 
>> A DSC override to fix *binary corruption* of an unknown cause? It is 
>> ridiculous
>> this can even happen silently, even though it’s unclear which component is at
>> fault.
>> 
>>> Normal EFI image needs to set its page size for the smaller
>>> image size.
>>> 
>>> You can see GCC DLINK option. It also sets page-size as 0x40.
>>> 
>>> DEFINE GCC49_IA32_X64_DLINK_COMMON   = -nostdlib
>> -Wl,-n,-q,--gc-sections -z
>>> common-page-size=0x40
>> 
>> Side note, the correct way to do this is setting max-page-size, not
>> common-page-size.
>> 
>>> 
>>> Thanks
>>> Liming
 -邮件原件-
 发件人: devel@edk2.groups.io  代表 Patrick
 Rudolph
 发送时间: 2023年3月17日 22:06
 抄送: devel@edk2.groups.io; guo.d...@intel.com; gua@intel.com;
 james...@intel.com; ray...@intel.com; mhaeu...@posteo.de;
 a...@kernel.org
 主题: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix
 CLANGDWARF_IA32_X64
 
 Drop the "-z max-page-size=0x40" option as it causes the ELF
 header to overflow into the .text section, causing undefined
 behaviour.
>> 
>> That *definitely* is not a fix. Not only does this regress binary size for
>> platforms that have tight SPI space constraints, it also only masks the 
>> issue. In
>> the (frankly near-impossible) case the ELF header dramatically grows in size,
>> who knows whether it can overflow again?
>> 
>> Sorry, but the overall description is pretty vague. Which OS / compiler 
>> version
>> are you using? Do you have any trivial way to detect the corruption? I never
>> really touched UefiPayloadPkg and have nothing set up to boot it to reproduce
>> the issue.
>> 
>> Best regards,
>> Marvin
>> 
 
 With high optimization level it corrupts essential code and
 the binary would crash. It might work with low optimization
 level though. As the default is to use Oz and LTO, it always
 crashes.
 
 Test:
 The ELF generated by
 'python UefiPayloadPkg/UniversalPayloadBuild.py -a IA32' boots.
 
 Signed-off-by: Patrick Rudolph 
 Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=4357
 ---
 BaseTools/Conf/tools_def.template | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 
 diff --git a/BaseTools/Conf/tools_def.template
 b/BaseTools/Conf/tools_def.template
 index 9b59bd75c3..0c584ab390 100755
 --- a/BaseTools/Conf/tools_def.template
 +++ b/BaseTools/Conf/tools_def.template
 @@ -2866,7 +2866,7 @@ DEFINE CLANGDWARF_X64_PREFIX=
 ENV(CLANG_BIN)
 
 
 # LLVM/CLANG doesn't support -n link option. So, it can't share the same
 IA32_X64_DLINK_COMMON flag.
 
 # LLVM/CLANG doesn't support common page size. So, it can't share the
 same GccBase.lds script.
 
 -DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib
 -Wl,-q,--gc-sections -z max-page-size=0x40
 
 +DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib
 -Wl,-q,--gc-sections
 
 DEFINE CLANGDWARF_DLINK2_FLAGS_COMMON =
 -Wl,--script=$(EDK_TOOLS_PATH)/Scripts/ClangBase.lds
 
 DEFINE CLANGDWARF_IA32_X64_ASLDLINK_FLAGS =
 DEF(CLANGDWARF_IA32_X64_DLINK_COMMON)
 -Wl,--defsym=PECOFF_HEADER_SIZE=0
 DEF(CLANGDWARF_DLINK2_FLAGS_COMMON)
 -Wl,--entry,ReferenceAcpiTable -u ReferenceAcpiTable
 
 DEFINE CLANGDWARF_IA32_X64_DLINK_FLAGS=
 DEF(CLANGDWARF_IA32_X64_DLINK_COMMON)
 -Wl,--entry,$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT)
 -Wl,-Map,$(DEST_DIR_DEBUG)/$(BASE_NAME).map,--whole-archive
 
 --
 2.39.1
 
 
 
 -=-=-=-=-=-=
 Groups.io Links: You receive all messages sent to this group.
 View/Reply Online (#101341):
 https://edk2.groups.io/g/devel/message/101341
 Mute This Topic: https://groups.io/mt/97673649/4905953
 Group Owner: 

Re: [edk2-devel] [RFT PATCH v3 0/5] UefiCpuPkg, OvmfPkg: Simplify CpuExceptionHandlerLib

[Marvin Häuser]

Hi Ray,

> On 31. Mar 2023, at 12:09, Ni, Ray  wrote:
> 
> Ard,
> What does "-read_only_relocs suppress" control?

It controls whether relocs that target read-only segments yield a build error 
or not. I think lld uses “-z notext”.

> Linker doesn't produce relocation entries that modifies .text section silently
> so the final .text just cannot run at all?

Could you please rephrase? I’m not sure I understand, but I think it’s 
important everyone understands the issues at play to make a good judgment call.

Best regards,
Marvin

> 
> Thanks,
> Ray
> 
>> -Original Message-
>> From: devel@edk2.groups.io  On Behalf Of Ard
>> Biesheuvel
>> Sent: Friday, March 31, 2023 5:15 PM
>> To: devel@edk2.groups.io
>> Cc: Ard Biesheuvel ; Ni, Ray ; Andrew
>> Fish ; Kinney, Michael D ;
>> Liu, Zhiguang ; Rebecca Cran
>> ; Tom Lendacky ;
>> Marvin Häuser 
>> Subject: [edk2-devel] [RFT PATCH v3 0/5] UefiCpuPkg, OvmfPkg: Simplify
>> CpuExceptionHandlerLib
>> 
>> We have a special version of CpuExceptionHandlerLib for XCODE5, whose
>> 
>> linker (LLD) does not permit absolute symbol references in read-only
>> 
>> sections.
>> 
>> 
>> 
>> Instead of fixing this up at runtime for all toolchains (which is done
>> 
>> by writing the fixed up values to the .text section, which we'd prefer
>> 
>> to avoid), tweak the SEC/PEI version so it does not need this, and
>> 
>> update the remaining versions to only incorporate this logic when using
>> 
>> the XCODE toolchain.
>> 
>> 
>> 
>> Changes since v3:
>> 
>> -  As Marvin points out, using '-read_only_relocs suppress' with the X64
>> 
>>   XCODE linker is a terrible idea, as it corrupts the resulting PE
>> 
>>   binaries, so instead, let's do the following:
>> 
>>   . tweak the SEC/PEI version of the library so the relocs are emitted
>> 
>> into .data when using XCODE;
>> 
>>   . tweak the other versions so the runtime fixups are only done when
>> 
>> using XCODE
>> 
>> - add acks from Jiewen and Ray
>> 
>> 
>> 
>> Changes since v2:
>> 
>> - pass linker switches to permit absolute relocations in read-only
>> 
>>  regions, and keep all code in .text
>> 
>> 
>> 
>> Cc: "Ni, Ray" 
>> 
>> Cc: Andrew Fish 
>> 
>> Cc: "Kinney, Michael D" 
>> 
>> Cc: "Liu, Zhiguang" 
>> 
>> Cc: Rebecca Cran 
>> 
>> Cc: Tom Lendacky 
>> 
>> Cc: Marvin Häuser 
>> 
>> 
>> 
>> Ard Biesheuvel (5):
>> 
>>  BaseTools/tools_def CLANGDWARF: Permit text relocations
>> 
>>  UefiCpuPkg/CpuExceptionHandlerLib: Use single SEC/PEI version
>> 
>>  UefiCpuPkg/CpuExceptionHandlerLib: Make runtime fixups XCODE-only
>> 
>>  OvmfPkg: Drop special Xcode5 version of exception handler library
>> 
>>  UefiCpuPkg/CpuExceptionHandlerLib: Drop special XCODE5 version
>> 
>> 
>> 
>> BaseTools/Conf/tools_def.template
>> |   2 +-
>> 
>> OvmfPkg/AmdSev/AmdSevX64.dsc
>> |   4 -
>> 
>> OvmfPkg/CloudHv/CloudHvX64.dsc
>> |   4 -
>> 
>> OvmfPkg/IntelTdx/IntelTdxX64.dsc 
>> |
>> 4 -
>> 
>> OvmfPkg/Microvm/MicrovmX64.dsc
>> |   4 -
>> 
>> OvmfPkg/OvmfPkgIa32.dsc  
>> |   4
>> -
>> 
>> OvmfPkg/OvmfPkgIa32X64.dsc   
>> |
>> 4 -
>> 
>> OvmfPkg/OvmfPkgX64.dsc   
>> |   4
>> -
>> 
>> OvmfPkg/OvmfXen.dsc  
>> |   4 -
>> 
>> 
>> UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.in
>> f  |   5 +-
>> 
>> 
>> UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandlerLib.inf
>> |   4 +-
>> 
>> 
>> UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuExceptionHandlerLib
>> .inf   |   4 +-
>> 
>> 
>> UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandlerLib.i
>> nf  |   

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

> On 31. Mar 2023, at 11:36, Ard Biesheuvel  wrote:
> 
> On Fri, 31 Mar 2023 at 11:27, Marvin Häuser  wrote:
>> 
>> 
>>>> On 31. Mar 2023, at 10:59, Ard Biesheuvel  wrote:
>>> 
>>> On Fri, 31 Mar 2023 at 10:29, Marvin Häuser  wrote:
>>>> 
>>>> 
>>>>>> On 31. Mar 2023, at 09:39, Ard Biesheuvel  wrote:
>>>>> Hi Marvin,
>>>>> 
>>>>> Thanks for the context.
>>>>> 
>>>>> 
>>>>> On Thu, 30 Mar 2023 at 23:54, Marvin Häuser  wrote:
>>>>>> 
>>>>>> Hi Ard,
>>>>>> 
>>>>>> Sorry, I cannot preserve the CC list as the groups.io interface doesn't 
>>>>>> seem to allow it. Can you please CC me on future revisions?
>>>>>> 
>>>>>> This patch will badly corrupt binaries. I cannot cite a source right now 
>>>>>> (if you want me to, please remind me in your response, so I can look it 
>>>>>> up tomorrow), but for X64 (but not IA32, which is why this is enabled 
>>>>>> there), relocs are relative to the first *writable* segment. In other 
>>>>>> words, any relocation to __TEXT will badly corrupt binaries this way.
>>>>> 
>>>>> OMG.
>>>>> 
>>>>> I can't believe how buggy all this stuff is. But I can confirm that
>>>>> the resulting binaries don't look right, even though they appear to
>>>>> boot fine.
>>>> 
>>>> Codegen does not change from the suppress flag, so there will be no 
>>>> additional text relocs beyond those you introduced. As they target the 
>>>> exception handler, I guess you’d need to actively provoke the broken code 
>>>> paths (and may end up with a nice recursion :) ).
>>>> 
>>> 
>>> I understand that the codegen is the same. I was specifically talking
>>> about the PE relocations, which seem to be lacking entirely.
>> 
>> Sure, I was just elaborating on the “appear to boot fine” part, which does 
>> make sense. When I last tried, the relocs were not absent but underflown. 
>> Might be mtoc drops them somehow, I think I inspected the Mach-O directly. 
>> Whatever, you reproduce the issue. :)
>> 
> 
> Fair enough.
> 
>>> 
>>>>> In particular, when I dump the PE relocations using
>>>>> llvm-readobj --coff-basereloc, I don't see any relocations referring
>>>>> to the .text section.
>>>>> 
>>>>>> In AUDK, we support this with two essential changes. The first is that 
>>>>>> we always generate a writable dummy segment at the beginning of the 
>>>>>> address space [1], making the relocs relative to the image base. The 
>>>>>> second is that in ocmtoc, our fork of the abandoned (and pretty 
>>>>>> badly-bugged) Apple mtoc, we explicitly require this segment to be 
>>>>>> present and verify its virtual address is the minimum virtual address 
>>>>>> [2]. It is then omitted from the conversion process [3]. I suggest you 
>>>>>> replicate these changes and fully switch to ocmtoc for XCODE5 builds.
>>>>> 
>>>>> I'm not going to do any of that. Instead, I am going to drop this
>>>>> change, and do the following:
>>>>> 
>>>>> - modify the SecPei version of CpuExceptionHandlerLib to put the
>>>>> vector templates in .data, as I proposed before. This works around the
>>>>> issue, and given that SEC/PEI is assumed to be read-only anyway (as it
>>>>> may execute in place from flash) and does not use page alignment for
>>>>> the sections due to size constraints, it is reasonable to assume that
>>>>> .text and .data will be mapped executable anyway.
>>>> 
>>>> Well, that assumption is more than fair to make for the status quo 
>>>> platforms, but this is just another rock in the way of doing things the 
>>>> right way (even if it’s just VMs).
>>>> 
>>> 
>>> How so? SEC and PEI could be mapped read-only today, it's just that we
>>> never bother.
>> 
>> I’m not concerned about read-only but NX.
>> 
> 
> We don't have writable data in SEC or PEI, so this would require SEC,
> PEI_CORE and every PEIM to have split .text and .rodata, and round
> them up to page size. Not sure this is worth it, especially given the
> fact that CoCo targets seems to be

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

> On 31. Mar 2023, at 10:59, Ard Biesheuvel  wrote:
> 
> On Fri, 31 Mar 2023 at 10:29, Marvin Häuser  wrote:
>> 
>> 
>>>> On 31. Mar 2023, at 09:39, Ard Biesheuvel  wrote:
>>> Hi Marvin,
>>> 
>>> Thanks for the context.
>>> 
>>> 
>>> On Thu, 30 Mar 2023 at 23:54, Marvin Häuser  wrote:
>>>> 
>>>> Hi Ard,
>>>> 
>>>> Sorry, I cannot preserve the CC list as the groups.io interface doesn't 
>>>> seem to allow it. Can you please CC me on future revisions?
>>>> 
>>>> This patch will badly corrupt binaries. I cannot cite a source right now 
>>>> (if you want me to, please remind me in your response, so I can look it up 
>>>> tomorrow), but for X64 (but not IA32, which is why this is enabled there), 
>>>> relocs are relative to the first *writable* segment. In other words, any 
>>>> relocation to __TEXT will badly corrupt binaries this way.
>>> 
>>> OMG.
>>> 
>>> I can't believe how buggy all this stuff is. But I can confirm that
>>> the resulting binaries don't look right, even though they appear to
>>> boot fine.
>> 
>> Codegen does not change from the suppress flag, so there will be no 
>> additional text relocs beyond those you introduced. As they target the 
>> exception handler, I guess you’d need to actively provoke the broken code 
>> paths (and may end up with a nice recursion :) ).
>> 
> 
> I understand that the codegen is the same. I was specifically talking
> about the PE relocations, which seem to be lacking entirely.

Sure, I was just elaborating on the “appear to boot fine” part, which does make 
sense. When I last tried, the relocs were not absent but underflown. Might be 
mtoc drops them somehow, I think I inspected the Mach-O directly. Whatever, you 
reproduce the issue. :)

> 
>>> In particular, when I dump the PE relocations using
>>> llvm-readobj --coff-basereloc, I don't see any relocations referring
>>> to the .text section.
>>> 
>>>> In AUDK, we support this with two essential changes. The first is that we 
>>>> always generate a writable dummy segment at the beginning of the address 
>>>> space [1], making the relocs relative to the image base. The second is 
>>>> that in ocmtoc, our fork of the abandoned (and pretty badly-bugged) Apple 
>>>> mtoc, we explicitly require this segment to be present and verify its 
>>>> virtual address is the minimum virtual address [2]. It is then omitted 
>>>> from the conversion process [3]. I suggest you replicate these changes and 
>>>> fully switch to ocmtoc for XCODE5 builds.
>>> 
>>> I'm not going to do any of that. Instead, I am going to drop this
>>> change, and do the following:
>>> 
>>> - modify the SecPei version of CpuExceptionHandlerLib to put the
>>> vector templates in .data, as I proposed before. This works around the
>>> issue, and given that SEC/PEI is assumed to be read-only anyway (as it
>>> may execute in place from flash) and does not use page alignment for
>>> the sections due to size constraints, it is reasonable to assume that
>>> .text and .data will be mapped executable anyway.
>> 
>> Well, that assumption is more than fair to make for the status quo 
>> platforms, but this is just another rock in the way of doing things the 
>> right way (even if it’s just VMs).
>> 
> 
> How so? SEC and PEI could be mapped read-only today, it's just that we
> never bother.

I’m not concerned about read-only but NX.

> 
>> Cc Gerd for an OVMF security perspective. Is PEI-time memory protection 
>> something you’d be interested in in the future?
>> 
> 
> My WXN series for ARM maps all PEIMs read-only, and turns off
> shadowing entirely (which makes no sense for VMs). So we have most of
> what we need to do that, and this change has no bearing on that.

Well yes, if everything is read-only, you guarantee W^X implicitly, but 
downstream we have plans for the full deal including NX data. It’s however 
shelved for the distant future, so as long as this is changed with the 
intention of reverting it once XCODE5 is fixed or dropped, that sounds fine to 
me. I just don’t like the notion of intentionally breaking the memory 
permission model as a hack. I rather hope we’ll make some swift progress on 
removing XCODE5 as a source of frustration. :)

> 
>>> 
>>> - update the version that performs the runtime fixups to only do so
>>> when using the XCODE toolchain - we can phase that out once we drop
>>> XCODE sup

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

> On 31. Mar 2023, at 09:39, Ard Biesheuvel  wrote:
> Hi Marvin,
> 
> Thanks for the context.
> 
> 
> On Thu, 30 Mar 2023 at 23:54, Marvin Häuser  wrote:
>> 
>> Hi Ard,
>> 
>> Sorry, I cannot preserve the CC list as the groups.io interface doesn't seem 
>> to allow it. Can you please CC me on future revisions?
>> 
>> This patch will badly corrupt binaries. I cannot cite a source right now (if 
>> you want me to, please remind me in your response, so I can look it up 
>> tomorrow), but for X64 (but not IA32, which is why this is enabled there), 
>> relocs are relative to the first *writable* segment. In other words, any 
>> relocation to __TEXT will badly corrupt binaries this way.
> 
> OMG.
> 
> I can't believe how buggy all this stuff is. But I can confirm that
> the resulting binaries don't look right, even though they appear to
> boot fine.

Codegen does not change from the suppress flag, so there will be no additional 
text relocs beyond those you introduced. As they target the exception handler, 
I guess you’d need to actively provoke the broken code paths (and may end up 
with a nice recursion :) ).

> In particular, when I dump the PE relocations using
> llvm-readobj --coff-basereloc, I don't see any relocations referring
> to the .text section.
> 
>> In AUDK, we support this with two essential changes. The first is that we 
>> always generate a writable dummy segment at the beginning of the address 
>> space [1], making the relocs relative to the image base. The second is that 
>> in ocmtoc, our fork of the abandoned (and pretty badly-bugged) Apple mtoc, 
>> we explicitly require this segment to be present and verify its virtual 
>> address is the minimum virtual address [2]. It is then omitted from the 
>> conversion process [3]. I suggest you replicate these changes and fully 
>> switch to ocmtoc for XCODE5 builds.
> 
> I'm not going to do any of that. Instead, I am going to drop this
> change, and do the following:
> 
> - modify the SecPei version of CpuExceptionHandlerLib to put the
> vector templates in .data, as I proposed before. This works around the
> issue, and given that SEC/PEI is assumed to be read-only anyway (as it
> may execute in place from flash) and does not use page alignment for
> the sections due to size constraints, it is reasonable to assume that
> .text and .data will be mapped executable anyway.

Well, that assumption is more than fair to make for the status quo platforms, 
but this is just another rock in the way of doing things the right way (even if 
it’s just VMs).

Cc Gerd for an OVMF security perspective. Is PEI-time memory protection 
something you’d be interested in in the future?

> 
> - update the version that performs the runtime fixups to only do so
> when using the XCODE toolchain - we can phase that out once we drop
> XCODE support.

I agree if there’s an actual plan on doing that. We depend on XCODE5 
downstream, but I think it would literally be easier for us if the upstream 
version was dropped than rebasing against hacks that our slightly modded 
variant does not require.

Cc Andrew and Rebecca. I don’t know anyone else who might still be using 
XCODE5. Any objections to dropping it? If so, any plans to pick up my proposed 
changes instead?

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102252): https://edk2.groups.io/g/devel/message/102252
Mute This Topic: https://groups.io/mt/97960758/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v2 4/6] UefiCpuPkg/CpuExceptionHandlerLib: Remove needless runtime fixups

[Marvin Häuser]

Hi Ray,

Definitely - this is just an optimisation “while we’re at it“, nothing urgent.

Best regards,
Marvin

> On 31. Mar 2023, at 07:09, Ni, Ray  wrote:
> 
> 
> 
> diff --git a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionH=
> andlerAsm.nasm b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHan=
> dlerAsm.nasm
> similarity index 92%
> rename from UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHa=
> ndlerAsm.nasm
> rename to UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHandlerAsm=
> .nasm
> index 957478574253e619..10af4cfcdb6b1ea2 100644
> --- a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerA=
> sm.nasm
> +++ b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHandlerAsm.nasm
> @@ -59,7 +59,7 @@ AsmIdtVectorBegin:
> %rep 256=0D
> push strict dword %[Vector] ; This instruction pushes sign-extended=
> 8-byte value on stack=0D
> push rax=0D
> - mov rax, strict qword 0 ; mov rax, ASM_PFX(CommonInterruptE=
> ntry)=0D
> + mov rax, ASM_PFX(CommonInterruptEntry)=0D
> I'm fairly certain this can be a relative reference, as the code doesn't seem 
> to be copied away (as opposed to HookAfterStubHeaderBegin). If true, this 
> would save 256 relocs, which sounds quite nice. Would you mind verifying? 
> Thanks!
> 
> Marvin,
> 
> You are right the mov can be replaced with “lea”. But, we need to analyze if 
> using lea, the size of each idt stub is fixed or variant (today the size is 
> fixed, so StubSize = TotalSize / 256).
> 
> Another thing we need to evaluate is the impact to “Hook after” feature 
> because “hook after” stub is very like the idt stub.
> 
> I think we can firstly use Ard’s solution. Later to evaluate to eliminate 
> unnecessary absolute reference.
> 
> Thanks,
> 
> Ray


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102250): https://edk2.groups.io/g/devel/message/102250
Mute This Topic: https://groups.io/mt/97960766/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFT PATCH v2 4/6] UefiCpuPkg/CpuExceptionHandlerLib: Remove needless runtime fixups

[Marvin Häuser]

Hi Ard,

Thanks for these important updates!

On Thu, Mar 30, 2023 at 02:21 PM, Ard Biesheuvel wrote:

> 
> Recent versions of the XCODE linker can be instructed to permit text
> relocations, so we no longer have to work around this, which is
> especially nice as our workaround assumes that the .text section is
> mapped both writable and executable at the same time.
> 
> So remove the runtime fixups and instead, just emit the absolute
> references into the .text section.
> 
> While at it, rename the Xcode5ExceptionHandlerAsm.nasm source file and
> drop the Xcode5 prefix: this code is used by other toolchains too.
> 
> Signed-off-by: Ard Biesheuvel 
> ---
> UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.inf =
> | 2 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandlerLib.inf =
> | 2 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandlerLib.inf =
> | 2 +-
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/{Xcode5ExceptionHandlerAsm.n=
> 
> asm =3D> ExceptionHandlerAsm.nasm} | 18 ++
> 4 files changed, 5 insertions(+), 19 deletions(-)
> 
> diff --git
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandl=
> erLib.inf
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandle=
> rLib.inf
> index d0f82095cf926e99..1b2dde746d154706 100644
> ---
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.i=
> nf
> +++
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.i=
> nf
> @@ -28,7 +28,7 @@ [Sources.Ia32]
> Ia32/ArchInterruptDefs.h=0D
> =0D
> [Sources.X64]=0D
> - X64/Xcode5ExceptionHandlerAsm.nasm=0D
> + X64/ExceptionHandlerAsm.nasm=0D
> X64/ArchExceptionHandler.c=0D
> X64/ArchInterruptDefs.h=0D
> =0D
> diff --git
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandl=
> erLib.inf
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandle=
> rLib.inf
> index 5339f8e604045801..86248cea3e97cedb 100644
> ---
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandlerLib.i=
> nf
> +++
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/PeiCpuExceptionHandlerLib.i=
> nf
> @@ -28,7 +28,7 @@ [Sources.Ia32]
> Ia32/ArchInterruptDefs.h=0D
> =0D
> [Sources.X64]=0D
> - X64/Xcode5ExceptionHandlerAsm.nasm=0D
> + X64/ExceptionHandlerAsm.nasm=0D
> X64/ArchExceptionHandler.c=0D
> X64/ArchInterruptDefs.h=0D
> =0D
> diff --git
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandl=
> erLib.inf
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandle=
> rLib.inf
> index 8f8a5dab79303f87..0eed594be8660302 100644
> ---
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandlerLib.i=
> nf
> +++
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/SmmCpuExceptionHandlerLib.i=
> nf
> @@ -28,7 +28,7 @@ [Sources.Ia32]
> Ia32/ArchInterruptDefs.h=0D
> =0D
> [Sources.X64]=0D
> - X64/Xcode5ExceptionHandlerAsm.nasm=0D
> + X64/ExceptionHandlerAsm.nasm=0D
> X64/ArchExceptionHandler.c=0D
> X64/ArchInterruptDefs.h=0D
> =0D
> diff --git
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionH=
> andlerAsm.nasm
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHan=
> dlerAsm.nasm
> similarity index 92%
> rename from
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHa=
> ndlerAsm.nasm
> rename to
> UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHandlerAsm=
> .nasm
> index 957478574253e619..10af4cfcdb6b1ea2 100644
> ---
> a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerA=
> sm.nasm
> +++
> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ExceptionHandlerAsm.nasm
> @@ -59,7 +59,7 @@ AsmIdtVectorBegin:
> %rep 256=0D
> push strict dword %[Vector] ; This instruction pushes sign-extended=
> 8-byte value on stack=0D
> push rax=0D
> - mov rax, strict qword 0 ; mov rax, ASM_PFX(CommonInterruptE=
> ntry)=0D
> + mov rax, ASM_PFX(CommonInterruptEntry)=0D

I'm fairly certain this can be a relative reference, as the code doesn't seem 
to be copied away (as opposed to HookAfterStubHeaderBegin). If true, this would 
save 256 relocs, which sounds quite nice. Would you mind verifying? Thanks!

If you want to take things a step further, this is how we merged the SEC/PEI 
and DXE/SMM variants: 
https://github.com/acidanthera/audk/commit/9646f2c4bc0475e0635b60b7c7828999a1d40dcb
 ( 
https://github.com/acidanthera/audk/commit/9646f2c4bc0475e0635b60b7c7828999a1d40dcb
 )
(There is a bug visible in the changes, which was fixed only later, so take 
this only as a PoC).

Best regards,
Marvin

> 
> jmp rax=0D
> %assign Vector Vector+1=0D
> %endrep=0D
> @@ -69,8 +69,7 @@ HookAfterStubHeaderBegin:
> push strict dword 0 ; 0 will be fixed=0D
> VectorNum:=0D
> push rax=0D
> - mov rax, strict qword 0 ; mov rax, HookAfterStubHeaderEnd=
> =0D
> -JmpAbsoluteAddress:=0D
> + mov rax, HookAfterStubHeaderEnd=0D
> jmp rax=0D
> HookAfterStubHeaderEnd:=0D
> mov rax, rsp=0D
> @@ -456,19 +455,6 @@ ASM_PFX(AsmGetTemplateAddressMap):
> mov qword [rcx + 0x8], 

Re: [edk2-devel] [RFT PATCH v2 1/6] BaseTools/tools_def XCODE: Link X64 with -read_only_relocs suppress

[Marvin Häuser]

Hi Ard,

Sorry, I cannot preserve the CC list as the groups.io interface doesn't seem to 
allow it. Can you please CC me on future revisions?

This patch will badly corrupt binaries. I cannot cite a source right now (if 
you want me to, please remind me in your response, so I can look it up 
tomorrow), but for X64 (but not IA32, which is why this is enabled there), 
relocs are relative to the first *writable* segment. In other words, any 
relocation to __TEXT will badly corrupt binaries this way.

In AUDK, we support this with two essential changes. The first is that we 
always generate a writable dummy segment at the beginning of the address space 
[1], making the relocs relative to the image base. The second is that in 
ocmtoc, our fork of the abandoned (and pretty badly-bugged) Apple mtoc, we 
explicitly require this segment to be present and verify its virtual address is 
the minimum virtual address [2]. It is then omitted from the conversion process 
[3]. I suggest you replicate these changes and fully switch to ocmtoc for 
XCODE5 builds.

Best regards,
Marvin

[1]
https://github.com/acidanthera/audk/blob/c382e9c571c7d5f39ba53b46a0c723c7943f33c5/BaseTools/Conf/tools_def.template#L2976-L2988

[2]
https://github.com/acidanthera/ocmtoc/blob/b0152c51beae264770c3faf0d213f9594ee043be/efitools/mtoc.c#L1097-L1123
https://github.com/acidanthera/ocmtoc/blob/b0152c51beae264770c3faf0d213f9594ee043be/efitools/mtoc.c#L1204-L1214

[3]
https://github.com/acidanthera/ocmtoc/blob/b0152c51beae264770c3faf0d213f9594ee043be/efitools/mtoc.c#L1307-L1311


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102219): https://edk2.groups.io/g/devel/message/102219
Mute This Topic: https://groups.io/mt/97960758/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/3] BaseTools/Conf/tools_def: Fix linking using CLANGDWARF_IA32

[Marvin Häuser]

Hi Lean,I don’t think individual patches from a series are merged generally. Your 2/3 has open concerns from both Liming and myself.Best regards,MarvinOn 30. Mar 2023, at 09:31, Lean Sheng Tan  wrote:HI Liming,If no further concern, would you mind to help get this patch merged?Thanks!Best Regards,Lean Sheng TanOn Tue, 28 Mar 2023 at 07:42, gaoliming via groups.io byosoft.com...@groups.io> wrote:Reviewed-by: Liming Gao 

> -邮件原件-
> 发件人: devel@edk2.groups.io  代表 Patrick
> Rudolph
> 发送时间: 2023年3月17日 22:06
> 抄送: devel@edk2.groups.io; guo.d...@intel.com; gua@intel.com;
> james...@intel.com; ray...@intel.com; mhaeu...@posteo.de;
> a...@kernel.org
> 主题: [edk2-devel] [PATCH 1/3] BaseTools/Conf/tools_def: Fix linking using
> CLANGDWARF_IA32
> 
> The clang toolchain might default to fPIE/fPIC, which prevents
> lld from linking the objects into a binary.
> 
> Specify -fno-pie -fno-pic as done on GCC to fix linking.
> 
> Test:
> Building the Universal Payload using the command
> 'python UefiPayloadPkg/UniversalPayloadBuild.py -a IA32' actually
> works.
> 
> Signed-off-by: Patrick Rudolph 
> Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=4356
> ---
>  BaseTools/Conf/tools_def.template | 14 +++---
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/BaseTools/Conf/tools_def.template
> b/BaseTools/Conf/tools_def.template
> index 471eb67c0c..9b59bd75c3 100755
> --- a/BaseTools/Conf/tools_def.template
> +++ b/BaseTools/Conf/tools_def.template
> @@ -2888,7 +2888,7 @@ DEFINE CLANGDWARF_X64_DLINK2_FLAGS
> = -Wl,--defsym=PECOFF_HEADER_SIZE=0x22
>  *_CLANGDWARF_IA32_RC_PATH              =
> DEF(CLANGDWARF_IA32_PREFIX)llvm-rc
> 
> 
> 
>  *_CLANGDWARF_IA32_ASLCC_FLAGS          = DEF(GCC_ASLCC_FLAGS)
> -m32 -fno-lto DEF(CLANG38_IA32_TARGET)
> 
> -*_CLANGDWARF_IA32_ASLDLINK_FLAGS       =
> DEF(CLANGDWARF_IA32_X64_ASLDLINK_FLAGS) -Wl,-m,elf_i386 -fuse-ld=lld
> 
> +*_CLANGDWARF_IA32_ASLDLINK_FLAGS       =
> DEF(CLANGDWARF_IA32_X64_ASLDLINK_FLAGS) -Wl,-m,elf_i386 -fuse-ld=lld
> -no-pie
> 
>  *_CLANGDWARF_IA32_ASM_FLAGS            = DEF(GCC5_ASM_FLAGS)
> -m32 -march=i386 DEF(CLANG38_IA32_TARGET)
> 
>  *_CLANGDWARF_IA32_RC_FLAGS             =
> DEF(GCC_IA32_RC_FLAGS)
> 
>  *_CLANGDWARF_IA32_OBJCOPY_FLAGS        =
> 
> @@ -2897,17 +2897,17 @@ DEFINE CLANGDWARF_X64_DLINK2_FLAGS
> = -Wl,--defsym=PECOFF_HEADER_SIZE=0x22
>  *_CLANGDWARF_IA32_ASLPP_FLAGS          = DEF(GCC_ASLPP_FLAGS)
> DEF(CLANG38_IA32_TARGET)
> 
>  *_CLANGDWARF_IA32_VFRPP_FLAGS          = DEF(GCC_VFRPP_FLAGS)
> DEF(CLANG38_IA32_TARGET)
> 
> 
> 
> -DEBUG_CLANGDWARF_IA32_CC_FLAGS         =
> DEF(CLANG38_ALL_CC_FLAGS) -m32 -Oz -flto -march=i586
> DEF(CLANG38_IA32_TARGET) -g -malign-double
> 
> +DEBUG_CLANGDWARF_IA32_CC_FLAGS         =
> DEF(CLANG38_ALL_CC_FLAGS) -fno-pic -fno-pie -m32 -Oz -flto -march=i586
> DEF(CLANG38_IA32_TARGET) -g -malign-double
> 
>  DEBUG_CLANGDWARF_IA32_DLINK_FLAGS      =
> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -flto -Wl,-O3 -Wl,-melf_i386
> -Wl,--oformat,elf32-i386
> 
> -DEBUG_CLANGDWARF_IA32_DLINK2_FLAGS     =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O3 -fuse-ld=lld
> 
> +DEBUG_CLANGDWARF_IA32_DLINK2_FLAGS     =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O3 -fuse-ld=lld -no-pie
> 
> 
> 
> -RELEASE_CLANGDWARF_IA32_CC_FLAGS       =
> DEF(CLANG38_ALL_CC_FLAGS) -m32 -Oz -flto -march=i586
> DEF(CLANG38_IA32_TARGET) -malign-double
> 
> +RELEASE_CLANGDWARF_IA32_CC_FLAGS       =
> DEF(CLANG38_ALL_CC_FLAGS) -fno-pic -fno-pie -m32 -Oz -flto -march=i586
> DEF(CLANG38_IA32_TARGET) -malign-double
> 
>  RELEASE_CLANGDWARF_IA32_DLINK_FLAGS    =
> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -flto -Wl,-O3 -Wl,-melf_i386
> -Wl,--oformat,elf32-i386
> 
> -RELEASE_CLANGDWARF_IA32_DLINK2_FLAGS   =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O3 -fuse-ld=lld
> 
> +RELEASE_CLANGDWARF_IA32_DLINK2_FLAGS   =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O3 -fuse-ld=lld -no-pie
> 
> 
> 
> -NOOPT_CLANGDWARF_IA32_CC_FLAGS         =
> DEF(CLANG38_ALL_CC_FLAGS) -m32 -O0 -march=i586
> DEF(CLANG38_IA32_TARGET) -g -malign-double
> 
> +NOOPT_CLANGDWARF_IA32_CC_FLAGS         =
> DEF(CLANG38_ALL_CC_FLAGS) -fno-pic -fno-pie -m32 -O0 -march=i586
> DEF(CLANG38_IA32_TARGET) -g -malign-double
> 
>  NOOPT_CLANGDWARF_IA32_DLINK_FLAGS      =
> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -Wl,-O0 -Wl,-melf_i386
> -Wl,--oformat,elf32-i386
> 
> -NOOPT_CLANGDWARF_IA32_DLINK2_FLAGS     =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O0 -fuse-ld=lld
> 
> +NOOPT_CLANGDWARF_IA32_DLINK2_FLAGS     =
> DEF(CLANGDWARF_IA32_DLINK2_FLAGS) -O0 -fuse-ld=lld -no-pie
> 
> 
> 
>  ##
> 
>  # CLANGDWARF X64 definitions
> 
> --
> 2.39.1
> 
> 
> 
> -=-=-=-=-=-=
> Groups.io Links: You receive all messages sent to this group.
> View/Reply Online (#101340):
> https://edk2.groups.io/g/devel/message/101340
> Mute This Topic: https://groups.io/mt/97673648/4905953
> Group Owner: devel+ow...@edk2.groups.io
> Unsubscribe: 

Re: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix CLANGDWARF_IA32_X64

[Marvin Häuser]

Hi all,

> On 28. Mar 2023, at 07:38, gaoliming  wrote:
> Patrick:
>  I prefer to override this option in DSC instead of the change in
> tools_def.txt.

A DSC override to fix *binary corruption* of an unknown cause? It is ridiculous 
this can even happen silently, even though it’s unclear which component is at 
fault.

> Normal EFI image needs to set its page size for the smaller
> image size. 
> 
>  You can see GCC DLINK option. It also sets page-size as 0x40.
> 
> DEFINE GCC49_IA32_X64_DLINK_COMMON   = -nostdlib -Wl,-n,-q,--gc-sections -z
> common-page-size=0x40

Side note, the correct way to do this is setting max-page-size, not 
common-page-size.

> 
> Thanks
> Liming
>> -邮件原件-
>> 发件人: devel@edk2.groups.io  代表 Patrick
>> Rudolph
>> 发送时间: 2023年3月17日 22:06
>> 抄送: devel@edk2.groups.io; guo.d...@intel.com; gua@intel.com;
>> james...@intel.com; ray...@intel.com; mhaeu...@posteo.de;
>> a...@kernel.org
>> 主题: [edk2-devel] [PATCH 2/3] BaseTools/Conf/tools_def: Fix
>> CLANGDWARF_IA32_X64
>> 
>> Drop the "-z max-page-size=0x40" option as it causes the ELF
>> header to overflow into the .text section, causing undefined
>> behaviour.

That *definitely* is not a fix. Not only does this regress binary size for 
platforms that have tight SPI space constraints, it also only masks the issue. 
In the (frankly near-impossible) case the ELF header dramatically grows in 
size, who knows whether it can overflow again?

Sorry, but the overall description is pretty vague. Which OS / compiler version 
are you using? Do you have any trivial way to detect the corruption? I never 
really touched UefiPayloadPkg and have nothing set up to boot it to reproduce 
the issue.

Best regards,
Marvin

>> 
>> With high optimization level it corrupts essential code and
>> the binary would crash. It might work with low optimization
>> level though. As the default is to use Oz and LTO, it always
>> crashes.
>> 
>> Test:
>> The ELF generated by
>> 'python UefiPayloadPkg/UniversalPayloadBuild.py -a IA32' boots.
>> 
>> Signed-off-by: Patrick Rudolph 
>> Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=4357
>> ---
>> BaseTools/Conf/tools_def.template | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/BaseTools/Conf/tools_def.template
>> b/BaseTools/Conf/tools_def.template
>> index 9b59bd75c3..0c584ab390 100755
>> --- a/BaseTools/Conf/tools_def.template
>> +++ b/BaseTools/Conf/tools_def.template
>> @@ -2866,7 +2866,7 @@ DEFINE CLANGDWARF_X64_PREFIX=
>> ENV(CLANG_BIN)
>> 
>> 
>> # LLVM/CLANG doesn't support -n link option. So, it can't share the same
>> IA32_X64_DLINK_COMMON flag.
>> 
>> # LLVM/CLANG doesn't support common page size. So, it can't share the
>> same GccBase.lds script.
>> 
>> -DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib
>> -Wl,-q,--gc-sections -z max-page-size=0x40
>> 
>> +DEFINE CLANGDWARF_IA32_X64_DLINK_COMMON   = -nostdlib
>> -Wl,-q,--gc-sections
>> 
>> DEFINE CLANGDWARF_DLINK2_FLAGS_COMMON =
>> -Wl,--script=$(EDK_TOOLS_PATH)/Scripts/ClangBase.lds
>> 
>> DEFINE CLANGDWARF_IA32_X64_ASLDLINK_FLAGS =
>> DEF(CLANGDWARF_IA32_X64_DLINK_COMMON)
>> -Wl,--defsym=PECOFF_HEADER_SIZE=0
>> DEF(CLANGDWARF_DLINK2_FLAGS_COMMON)
>> -Wl,--entry,ReferenceAcpiTable -u ReferenceAcpiTable
>> 
>> DEFINE CLANGDWARF_IA32_X64_DLINK_FLAGS=
>> DEF(CLANGDWARF_IA32_X64_DLINK_COMMON)
>> -Wl,--entry,$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT)
>> -Wl,-Map,$(DEST_DIR_DEBUG)/$(BASE_NAME).map,--whole-archive
>> 
>> --
>> 2.39.1
>> 
>> 
>> 
>> -=-=-=-=-=-=
>> Groups.io Links: You receive all messages sent to this group.
>> View/Reply Online (#101341):
>> https://edk2.groups.io/g/devel/message/101341
>> Mute This Topic: https://groups.io/mt/97673649/4905953
>> Group Owner: devel+ow...@edk2.groups.io
>> Unsubscribe: https://edk2.groups.io/g/devel/unsub
>> [gaolim...@byosoft.com.cn]
>> -=-=-=-=-=-=



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#102024): https://edk2.groups.io/g/devel/message/102024
Mute This Topic: https://groups.io/mt/97902889/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2 14/17] BaseTools/GenFw: Add DllCharacteristicsEx field to debug data

[Marvin Häuser]

Hi Ard,

> On 27. Mar 2023, at 13:01, Ard Biesheuvel  wrote:
> 
> The PE/COFF spec describes an additional DllCharacteristics field
> implemented as a debug directory entry, which carries flags related to
> which control flow integrity (CFI) features are supported by the binary.

Out of mere personal interest, is there any reference for this yet? The "PE 
format" page [1] doesn't seem to have this yet.

[1] 
https://learn.microsoft.com/en-us/windows/win32/debug/pe-format#the-debug-section

> 
> So let's add this entry when doing ELF to PE/COFF conversion - we will
> add support for setting the flags in a subsequent patch.
> 
> Signed-off-by: Ard Biesheuvel 
> ---
> BaseTools/Source/C/GenFw/Elf64Convert.c   | 54 
> +++-
> BaseTools/Source/C/GenFw/GenFw.c  |  3 +-
> BaseTools/Source/C/Include/IndustryStandard/PeImage.h | 13 -
> 3 files changed, 55 insertions(+), 15 deletions(-)
> 
> diff --git a/BaseTools/Source/C/GenFw/Elf64Convert.c 
> b/BaseTools/Source/C/GenFw/Elf64Convert.c
> index 2a810e835d4a4a66..9c17c90b16602421 100644
> --- a/BaseTools/Source/C/GenFw/Elf64Convert.c
> +++ b/BaseTools/Source/C/GenFw/Elf64Convert.c
> @@ -992,6 +992,16 @@ ScanSections64 (
> sizeof(EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY) +
> strlen(mInImageName) + 1;
> 
> +  //
> +  // Add more space in the .debug data region for the DllCharacteristicsEx
> +  // field.
> +  //
> +  if (mDllCharacteristicsEx != 0) {
> +mCoffOffset = DebugRvaAlign(mCoffOffset) +
> +  sizeof (EFI_IMAGE_DEBUG_DIRECTORY_ENTRY) +
> +  sizeof (EFI_IMAGE_DEBUG_EX_DLLCHARACTERISTICS_ENTRY);
> +  }
> +
>   mCoffOffset = CoffAlign(mCoffOffset);
>   if (SectionCount == 0) {
> mDataOffset = mCoffOffset;
> @@ -2244,29 +2254,47 @@ WriteDebug64 (
>   VOID
>   )
> {
> -  UINT32  Len;
> -  EFI_IMAGE_OPTIONAL_HEADER_UNION *NtHdr;
> -  EFI_IMAGE_DATA_DIRECTORY*DataDir;
> -  EFI_IMAGE_DEBUG_DIRECTORY_ENTRY *Dir;
> -  EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY *Nb10;
> +  UINT32  Len;
> +  EFI_IMAGE_OPTIONAL_HEADER_UNION *NtHdr;
> +  EFI_IMAGE_DATA_DIRECTORY*DataDir;
> +  EFI_IMAGE_DEBUG_DIRECTORY_ENTRY *Dir;
> +  EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY *Nb10;
> +  EFI_IMAGE_DEBUG_EX_DLLCHARACTERISTICS_ENTRY *DllEntry;
> 
>   Len = strlen(mInImageName) + 1;
> 
> +  NtHdr = (EFI_IMAGE_OPTIONAL_HEADER_UNION *)(mCoffFile + mNtHdrOffset);
> +  DataDir = 
> >Pe32Plus.OptionalHeader.DataDirectory[EFI_IMAGE_DIRECTORY_ENTRY_DEBUG];
> +  DataDir->VirtualAddress = mDebugOffset;
> +  DataDir->Size = sizeof (EFI_IMAGE_DEBUG_DIRECTORY_ENTRY);
> +
>   Dir = (EFI_IMAGE_DEBUG_DIRECTORY_ENTRY*)(mCoffFile + mDebugOffset);
> +
> +  if (mDllCharacteristicsEx != 0) {
> +DataDir->Size  += sizeof (EFI_IMAGE_DEBUG_DIRECTORY_ENTRY);
> +
> +Dir->Type   = EFI_IMAGE_DEBUG_TYPE_EX_DLLCHARACTERISTICS;
> +Dir->SizeOfData = sizeof (EFI_IMAGE_DEBUG_EX_DLLCHARACTERISTICS_ENTRY);
> +Dir->FileOffset = mDebugOffset + DataDir->Size +
> +  sizeof (EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY) +
> +  DebugRvaAlign(Len);
> +Dir->RVA= Dir->FileOffset;
> +
> +DllEntry = (VOID *)(mCoffFile + Dir->FileOffset);
> +
> +DllEntry->DllCharacteristicsEx = mDllCharacteristicsEx;
> +
> +Dir++;
> +  }
> +
>   Dir->Type = EFI_IMAGE_DEBUG_TYPE_CODEVIEW;
>   Dir->SizeOfData = sizeof(EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY) + Len;
> -  Dir->RVA = mDebugOffset + sizeof(EFI_IMAGE_DEBUG_DIRECTORY_ENTRY);
> -  Dir->FileOffset = mDebugOffset + sizeof(EFI_IMAGE_DEBUG_DIRECTORY_ENTRY);
> +  Dir->RVA = mDebugOffset + DataDir->Size;
> +  Dir->FileOffset = mDebugOffset + DataDir->Size;
> 
>   Nb10 = (EFI_IMAGE_DEBUG_CODEVIEW_NB10_ENTRY*)(Dir + 1);
>   Nb10->Signature = CODEVIEW_SIGNATURE_NB10;
>   strcpy ((char *)(Nb10 + 1), mInImageName);
> -
> -
> -  NtHdr = (EFI_IMAGE_OPTIONAL_HEADER_UNION *)(mCoffFile + mNtHdrOffset);
> -  DataDir = 
> >Pe32Plus.OptionalHeader.DataDirectory[EFI_IMAGE_DIRECTORY_ENTRY_DEBUG];
> -  DataDir->VirtualAddress = mDebugOffset;
> -  DataDir->Size = sizeof(EFI_IMAGE_DEBUG_DIRECTORY_ENTRY);
> }
> 
> STATIC
> diff --git a/BaseTools/Source/C/GenFw/GenFw.c 
> b/BaseTools/Source/C/GenFw/GenFw.c
> index 6f61f16788cd2a0a..d0e52ccc26431380 100644
> --- a/BaseTools/Source/C/GenFw/GenFw.c
> +++ b/BaseTools/Source/C/GenFw/GenFw.c
> @@ -2932,7 +2932,8 @@ Routine Description:
>   if (mIsConvertXip) {
> DebugEntry->FileOffset = DebugEntry->RVA;
>   }
> -  if (ZeroDebugFlag || DebugEntry->Type != 
> EFI_IMAGE_DEBUG_TYPE_CODEVIEW) {
> +  if ((ZeroDebugFlag || DebugEntry->Type != 
> EFI_IMAGE_DEBUG_TYPE_CODEVIEW) &&
> +  (DebugEntry->Type != EFI_IMAGE_DEBUG_TYPE_EX_DLLCHARACTERISTICS)) {
> memset (FileBuffer + DebugEntry->FileOffset, 0, 
> DebugEntry->SizeOfData);
> memset 

Re: [edk2-devel] [PATCH 3/5] MdePkg/Base.h: Introduce various alignment-related macros

[Marvin Häuser]

On 21. Mar 2023, at 23:29, Kinney, Michael D  wrote:









I am suggesting adding “minimum” because aligning to a boundary could be to the next one or any multiples past that.
 
For example, offset is 1, and alignment request is 4. 
The obvious result is 3, but 7, 11, 15, 19, … 
also satisfy the current description.They do not, the current description explicitly says “*next* alignment boundary”.Best regards,Marvin
 
Mike
 



From: Marvin Häuser 

Sent: Tuesday, March 21, 2023 3:00 PM
To: Kinney, Michael D 
Cc: Gerd Hoffmann ; devel@edk2.groups.io; Ard Biesheuvel ; Wang, Jian J ; Yao, Jiewen ; James Bottomley ; Michael Roth ;
 Wu, Hao A ; Oliver Steffen ; Xu, Min M ; Gao, Liming ; Ni, Ray ; Tom Lendacky ; Aktas, Erdem ; Liu, Zhiguang
 ; Pawel Polawski ; Justen, Jordan L ; Vitaly Cheptsov 
Subject: Re: [PATCH 3/5] MdePkg/Base.h: Introduce various alignment-related macros


 
Hi Mike,






On 21. Mar 2023, at 22:37, Kinney, Michael D <michael.d.kin...@intel.com> wrote:

 

Hi Gerd,

A few comments included below.

Thanks,

Mike




-Original Message-
From: Gerd Hoffmann <kra...@redhat.com>
Sent: Thursday, March 2, 2023 10:51 PM
To: devel@edk2.groups.io
Cc: Ard Biesheuvel <ardb+tianoc...@kernel.org>; Gerd Hoffmann <kra...@redhat.com>; Wang, Jian J <jian.j.w...@intel.com>; Yao,
Jiewen <jiewen@intel.com>; Marvin Häuser <mhaeu...@posteo.de>; James Bottomley <j...@linux.ibm.com>; Michael Roth
<michael.r...@amd.com>; Wu, Hao A <hao.a...@intel.com>; Kinney, Michael D <michael.d.kin...@intel.com>; Oliver Steffen
<ostef...@redhat.com>; Xu, Min M <min.m...@intel.com>; Gao, Liming <gaolim...@byosoft.com.cn>; Ni, Ray <ray...@intel.com>;
 Tom
Lendacky <thomas.lenda...@amd.com>; Aktas, Erdem <erdemak...@google.com>; Liu, Zhiguang <zhiguang@intel.com>; Pawel Polawski
<ppola...@redhat.com>; Justen, Jordan L <jordan.l.jus...@intel.com>; Vitaly Cheptsov <vit9...@protonmail.com>
Subject: [PATCH 3/5] MdePkg/Base.h: Introduce various alignment-related macros

From: Marvin Häuser <mhaeu...@posteo.de>

ALIGNOF: Determining the alignment requirement of data types is
crucial to ensure safe memory accesses when parsing untrusted data.

IS_POW2: Determining whether a value is a power of two is important
to verify whether untrusted values are valid alignment values.

IS_ALIGNED: In combination with ALIGNOF data offsets can be verified.
A more general version of the IS_ALIGNED macro previously defined by several modules.

ADDRESS_IS_ALIGNED: Variant of IS_ALIGNED for pointers and addresses.
Replaces module-specific definitions throughout the codebase.

ALIGN_VALUE_ADDEND: The addend to align up can be used to directly
determine the required offset for data alignment.

Cc: Michael D Kinney <michael.d.kin...@intel.com>
Cc: Liming Gao <gaolim...@byosoft.com.cn>
Cc: Zhiguang Liu <zhiguang@intel.com>
Cc: Vitaly Cheptsov <vit9...@protonmail.com>
Signed-off-by: Marvin Häuser <mhaeu...@posteo.de>
---
MdePkg/Include/Base.h | 95 ++-
1 file changed, 94 insertions(+), 1 deletion(-)

diff --git a/MdePkg/Include/Base.h b/MdePkg/Include/Base.h
index d209e6de280a..2053314b50d1 100644
--- a/MdePkg/Include/Base.h
+++ b/MdePkg/Include/Base.h
@@ -758,6 +758,40 @@ typedef UINTN *BASE_LIST;
#define OFFSET_OF(TYPE, Field)  ((UINTN) &(((TYPE *)0)->Field))
#endif

+/**
+  Returns the alignment requirement of a type.
+
+  @param   TYPE  The name of the type to retrieve the alignment requirement of.
+
+  @return  Alignment requirement, in Bytes, of TYPE.
+**/
+#if defined (__cplusplus)
+//
+// Standard C++ operator.
+//
+#define ALIGNOF(TYPE)  alignof (TYPE)
+#elif defined (__GNUC__) || defined (__clang__) || (defined (_MSC_VER) && _MSC_VER >= 1900)
+//
+// All supported versions of GCC and Clang, as well as MSVC 2015 and later,
+// support the standard operator _Alignof.
+//
+#define ALIGNOF(TYPE)  _Alignof (TYPE)
+#elif defined (_MSC_EXTENSIONS)
+//
+// Earlier versions of MSVC, at least MSVC 2008 and later, support the vendor
+// extension __alignof.
+//
+#define ALIGNOF(TYPE)  __alignof (TYPE)
+#else
+//
+// For compilers that do not support inbuilt alignof operators, use OFFSET_OF.
+// CHAR8 is known to have both a size and an alignment requirement of 1 Byte.
+// As such, A must be located exactly at the offset equal to its alignment
+// requirement.
+//
+#define ALIGNOF(TYPE)  OFFSET_OF (struct { CHAR8 C; TYPE A; }, A)
+#endif
+
/**
  Portable definition for compile time assertions.
  Equivalent to C11 static_assert macro from assert.h.
@@ -793,6 +827,21 @@ STATIC_ASSERT (sizeof (CHAR16)  == 2, "sizeof (CHAR16) does not meet UEFI Specif
STATIC_ASSERT (sizeof (L'A')    == 2, "sizeof (L'A') does not meet UEFI Specification Data Type requirements");
STATIC_ASSERT (sizeof (L"A")    == 4, &quo

Re: [edk2-devel] [PATCH 3/5] MdePkg/Base.h: Introduce various alignment-related macros

[Marvin Häuser]

Hi Mike,

> On 21. Mar 2023, at 22:37, Kinney, Michael D  
> wrote:
> 
> Hi Gerd,
> 
> A few comments included below.
> 
> Thanks,
> 
> Mike
> 
>> -Original Message-
>> From: Gerd Hoffmann mailto:kra...@redhat.com>>
>> Sent: Thursday, March 2, 2023 10:51 PM
>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
>> Cc: Ard Biesheuvel > <mailto:ardb+tianoc...@kernel.org>>; Gerd Hoffmann > <mailto:kra...@redhat.com>>; Wang, Jian J > <mailto:jian.j.w...@intel.com>>; Yao,
>> Jiewen mailto:jiewen@intel.com>>; Marvin Häuser 
>> mailto:mhaeu...@posteo.de>>; James Bottomley 
>> mailto:j...@linux.ibm.com>>; Michael Roth
>> mailto:michael.r...@amd.com>>; Wu, Hao A 
>> mailto:hao.a...@intel.com>>; Kinney, Michael D 
>> mailto:michael.d.kin...@intel.com>>; Oliver 
>> Steffen
>> mailto:ostef...@redhat.com>>; Xu, Min M 
>> mailto:min.m...@intel.com>>; Gao, Liming 
>> mailto:gaolim...@byosoft.com.cn>>; Ni, Ray 
>> mailto:ray...@intel.com>>; Tom
>> Lendacky mailto:thomas.lenda...@amd.com>>; Aktas, 
>> Erdem mailto:erdemak...@google.com>>; Liu, Zhiguang 
>> mailto:zhiguang@intel.com>>; Pawel Polawski
>> mailto:ppola...@redhat.com>>; Justen, Jordan L 
>> mailto:jordan.l.jus...@intel.com>>; Vitaly 
>> Cheptsov mailto:vit9...@protonmail.com>>
>> Subject: [PATCH 3/5] MdePkg/Base.h: Introduce various alignment-related 
>> macros
>> 
>> From: Marvin Häuser 
>> 
>> ALIGNOF: Determining the alignment requirement of data types is
>> crucial to ensure safe memory accesses when parsing untrusted data.
>> 
>> IS_POW2: Determining whether a value is a power of two is important
>> to verify whether untrusted values are valid alignment values.
>> 
>> IS_ALIGNED: In combination with ALIGNOF data offsets can be verified.
>> A more general version of the IS_ALIGNED macro previously defined by several 
>> modules.
>> 
>> ADDRESS_IS_ALIGNED: Variant of IS_ALIGNED for pointers and addresses.
>> Replaces module-specific definitions throughout the codebase.
>> 
>> ALIGN_VALUE_ADDEND: The addend to align up can be used to directly
>> determine the required offset for data alignment.
>> 
>> Cc: Michael D Kinney 
>> Cc: Liming Gao 
>> Cc: Zhiguang Liu 
>> Cc: Vitaly Cheptsov 
>> Signed-off-by: Marvin Häuser 
>> ---
>> MdePkg/Include/Base.h | 95 ++-
>> 1 file changed, 94 insertions(+), 1 deletion(-)
>> 
>> diff --git a/MdePkg/Include/Base.h b/MdePkg/Include/Base.h
>> index d209e6de280a..2053314b50d1 100644
>> --- a/MdePkg/Include/Base.h
>> +++ b/MdePkg/Include/Base.h
>> @@ -758,6 +758,40 @@ typedef UINTN *BASE_LIST;
>> #define OFFSET_OF(TYPE, Field)  ((UINTN) &(((TYPE *)0)->Field))
>> #endif
>> 
>> +/**
>> +  Returns the alignment requirement of a type.
>> +
>> +  @param   TYPE  The name of the type to retrieve the alignment requirement 
>> of.
>> +
>> +  @return  Alignment requirement, in Bytes, of TYPE.
>> +**/
>> +#if defined (__cplusplus)
>> +//
>> +// Standard C++ operator.
>> +//
>> +#define ALIGNOF(TYPE)  alignof (TYPE)
>> +#elif defined (__GNUC__) || defined (__clang__) || (defined (_MSC_VER) && 
>> _MSC_VER >= 1900)
>> +//
>> +// All supported versions of GCC and Clang, as well as MSVC 2015 and later,
>> +// support the standard operator _Alignof.
>> +//
>> +#define ALIGNOF(TYPE)  _Alignof (TYPE)
>> +#elif defined (_MSC_EXTENSIONS)
>> +//
>> +// Earlier versions of MSVC, at least MSVC 2008 and later, support the 
>> vendor
>> +// extension __alignof.
>> +//
>> +#define ALIGNOF(TYPE)  __alignof (TYPE)
>> +#else
>> +//
>> +// For compilers that do not support inbuilt alignof operators, use 
>> OFFSET_OF.
>> +// CHAR8 is known to have both a size and an alignment requirement of 1 
>> Byte.
>> +// As such, A must be located exactly at the offset equal to its alignment
>> +// requirement.
>> +//
>> +#define ALIGNOF(TYPE)  OFFSET_OF (struct { CHAR8 C; TYPE A; }, A)
>> +#endif
>> +
>> /**
>>   Portable definition for compile time assertions.
>>   Equivalent to C11 static_assert macro from assert.h.
>> @@ -793,6 +827,21 @@ STATIC_ASSERT (sizeof (CHAR16)  == 2, "sizeof (CHAR16) 
>> does not meet UEFI Specif
>> STATIC_ASSERT (sizeof (L'A')== 2, "sizeof (L'A') does n

Re: [edk2-devel] [PATCH v2] IntelFsp2Pkg: Fix NASM X64 build warnings.

[Marvin Häuser]

Yes - it does. Most (if not all?) operations on 32-bit registers zero-extend 
the corresponding 64-bit register. This is an AMD64 / Intel 64 design to combat 
partial register stall. Please consult the SDM (or at least try it out).

What I didn’t realize is that “mov eax, eax” apparently defeats register 
renaming optimisations: https://stackoverflow.com/a/45660140

Best regards,
Marvin

> On 19. Mar 2023, at 10:07, S, Ashraf Ali  wrote:
> 
> Hi.,
>  
> Nope, it will not clear the upper 32bit right.
>  
>  
> From: Marvin Häuser  
> Sent: Sunday, March 19, 2023 3:38 AM
> To: S, Ashraf Ali ; devel@edk2.groups.io
> Subject: Re: [edk2-devel] [PATCH v2] IntelFsp2Pkg: Fix NASM X64 build 
> warnings.
>  
> Hi Ashraf,
> 
> ”mov eax, eax” does clear the high 32 Bits of rax.
> 
> Best regards,
> Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101368): https://edk2.groups.io/g/devel/message/101368
Mute This Topic: https://groups.io/mt/97678369/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2] IntelFsp2Pkg: Fix NASM X64 build warnings.

[Marvin Häuser]

Hi Ashraf,

”mov eax, eax” does clear the high 32 Bits of rax.

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101366): https://edk2.groups.io/g/devel/message/101366
Mute This Topic: https://groups.io/mt/97678369/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2] IntelFsp2Pkg: Fix NASM X64 build warnings.

[Marvin Häuser]

Hi Chasel,

If you want to improve the ASM further, maybe opt for “test reg, reg” over “cmp 
reg, 0”?

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101365): https://edk2.groups.io/g/devel/message/101365
Mute This Topic: https://groups.io/mt/97678369/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/3] BaseTools/Conf/tools_def: Fix linking using CLANGDWARF_IA32

[Marvin Häuser]

Which error precisely?

X64 PIE *must not* be disabled, as it’s used to enforce RIP-relative addressing 
where possible and has been for many years. Meanwhile the issues with IA32 is 
PIE was disabled by default for most toolchains, but some enable it nowadays 
(and thus it must be disabled explicitly).

Best regards,
Marvin

> On 17. Mar 2023, at 18:30, Rebecca Cran  wrote:
> 
> I've also seen that error building X64 firmware.
> 
> Could you update those flags too, please?
> 
> 
> -- 
> 
> Rebecca Cran
> 
> 
>> On 3/17/23 8:06 AM, Patrick Rudolph wrote:
>> The clang toolchain might default to fPIE/fPIC, which prevents
>> lld from linking the objects into a binary.
>> 
>> Specify -fno-pie -fno-pic as done on GCC to fix linking.
>> 
>> Test:
>> Building the Universal Payload using the command
>> 'python UefiPayloadPkg/UniversalPayloadBuild.py -a IA32' actually
>> works.
>> 
>> Signed-off-by: Patrick Rudolph 
>> Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=4356
>> ---
>>  BaseTools/Conf/tools_def.template | 14 +++---
>>  1 file changed, 7 insertions(+), 7 deletions(-)
>> 
>> diff --git a/BaseTools/Conf/tools_def.template 
>> b/BaseTools/Conf/tools_def.template
>> index 471eb67c0c..9b59bd75c3 100755
>> --- a/BaseTools/Conf/tools_def.template
>> +++ b/BaseTools/Conf/tools_def.template
>> @@ -2888,7 +2888,7 @@ DEFINE CLANGDWARF_X64_DLINK2_FLAGS= 
>> -Wl,--defsym=PECOFF_HEADER_SIZE=0x22
>>  *_CLANGDWARF_IA32_RC_PATH  = DEF(CLANGDWARF_IA32_PREFIX)llvm-rc
>>*_CLANGDWARF_IA32_ASLCC_FLAGS  = DEF(GCC_ASLCC_FLAGS) -m32 
>> -fno-lto DEF(CLANG38_IA32_TARGET)
>> -*_CLANGDWARF_IA32_ASLDLINK_FLAGS   = 
>> DEF(CLANGDWARF_IA32_X64_ASLDLINK_FLAGS) -Wl,-m,elf_i386 -fuse-ld=lld
>> +*_CLANGDWARF_IA32_ASLDLINK_FLAGS   = 
>> DEF(CLANGDWARF_IA32_X64_ASLDLINK_FLAGS) -Wl,-m,elf_i386 -fuse-ld=lld -no-pie
>>  *_CLANGDWARF_IA32_ASM_FLAGS= DEF(GCC5_ASM_FLAGS) -m32 
>> -march=i386 DEF(CLANG38_IA32_TARGET)
>>  *_CLANGDWARF_IA32_RC_FLAGS = DEF(GCC_IA32_RC_FLAGS)
>>  *_CLANGDWARF_IA32_OBJCOPY_FLAGS=
>> @@ -2897,17 +2897,17 @@ DEFINE CLANGDWARF_X64_DLINK2_FLAGS= 
>> -Wl,--defsym=PECOFF_HEADER_SIZE=0x22
>>  *_CLANGDWARF_IA32_ASLPP_FLAGS  = DEF(GCC_ASLPP_FLAGS) 
>> DEF(CLANG38_IA32_TARGET)
>>  *_CLANGDWARF_IA32_VFRPP_FLAGS  = DEF(GCC_VFRPP_FLAGS) 
>> DEF(CLANG38_IA32_TARGET)
>>  -DEBUG_CLANGDWARF_IA32_CC_FLAGS = DEF(CLANG38_ALL_CC_FLAGS) -m32 
>> -Oz -flto -march=i586 DEF(CLANG38_IA32_TARGET) -g -malign-double
>> +DEBUG_CLANGDWARF_IA32_CC_FLAGS = DEF(CLANG38_ALL_CC_FLAGS) -fno-pic 
>> -fno-pie -m32 -Oz -flto -march=i586 DEF(CLANG38_IA32_TARGET) -g 
>> -malign-double
>>  DEBUG_CLANGDWARF_IA32_DLINK_FLAGS  = 
>> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -flto -Wl,-O3 -Wl,-melf_i386 
>> -Wl,--oformat,elf32-i386
>> -DEBUG_CLANGDWARF_IA32_DLINK2_FLAGS = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O3 -fuse-ld=lld
>> +DEBUG_CLANGDWARF_IA32_DLINK2_FLAGS = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O3 -fuse-ld=lld -no-pie
>>  -RELEASE_CLANGDWARF_IA32_CC_FLAGS   = DEF(CLANG38_ALL_CC_FLAGS) -m32 
>> -Oz -flto -march=i586 DEF(CLANG38_IA32_TARGET) -malign-double
>> +RELEASE_CLANGDWARF_IA32_CC_FLAGS   = DEF(CLANG38_ALL_CC_FLAGS) -fno-pic 
>> -fno-pie -m32 -Oz -flto -march=i586 DEF(CLANG38_IA32_TARGET) -malign-double
>>  RELEASE_CLANGDWARF_IA32_DLINK_FLAGS= 
>> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -flto -Wl,-O3 -Wl,-melf_i386 
>> -Wl,--oformat,elf32-i386
>> -RELEASE_CLANGDWARF_IA32_DLINK2_FLAGS   = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O3 -fuse-ld=lld
>> +RELEASE_CLANGDWARF_IA32_DLINK2_FLAGS   = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O3 -fuse-ld=lld -no-pie
>>  -NOOPT_CLANGDWARF_IA32_CC_FLAGS = DEF(CLANG38_ALL_CC_FLAGS) -m32 
>> -O0 -march=i586 DEF(CLANG38_IA32_TARGET) -g -malign-double
>> +NOOPT_CLANGDWARF_IA32_CC_FLAGS = DEF(CLANG38_ALL_CC_FLAGS) -fno-pic 
>> -fno-pie -m32 -O0 -march=i586 DEF(CLANG38_IA32_TARGET) -g -malign-double
>>  NOOPT_CLANGDWARF_IA32_DLINK_FLAGS  = 
>> DEF(CLANGDWARF_IA32_X64_DLINK_FLAGS) -Wl,-O0 -Wl,-melf_i386 
>> -Wl,--oformat,elf32-i386
>> -NOOPT_CLANGDWARF_IA32_DLINK2_FLAGS = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O0 -fuse-ld=lld
>> +NOOPT_CLANGDWARF_IA32_DLINK2_FLAGS = DEF(CLANGDWARF_IA32_DLINK2_FLAGS) 
>> -O0 -fuse-ld=lld -no-pie
>>##
>>  # CLANGDWARF X64 definitions



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101358): https://edk2.groups.io/g/devel/message/101358
Mute This Topic: https://groups.io/mt/97673648/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [edk2-platforms] Enable GitHub PR, protected branches, and 'push' label

[Marvin Häuser]

> On 17. Mar 2023, at 14:44, Gerd Hoffmann  wrote:
> 
> On Fri, Mar 17, 2023 at 12:32:15PM +0000, Marvin Häuser wrote:
>> Hi Rebecca and Gerd,
>> 
>> Replying to 2 mails at once...
>> 
>>>> On 17. Mar 2023, at 11:36, Rebecca Cran  wrote:
>>> 
>>> I like that proposed workflow.
>>> 
>>> I've also been wondering if we could consider choosing a different
>>> product for patch reviews that supports our desired workflow better,
>>> such as Gitlab or Phorge (the new Phabricator project).
>> 
>> I'd be very cautious with suggesting / approving more tooling. It gets
>> more confusing (what is hosted where), it gets more complicated to
>> maintain (who hosts what and is "guaranteed" to be available to fix
>> things), and so on.
> 
> Agree.  Also from the web-based review tools I've worked with so far
> (not much, only github and gitlab) github is the better one.
> 
>>>> (1) In my experience reviewing patches, especially more complex ones,
>>>> works better in email than in github PR workflows.
>> 
>> I have no experience with things like large-scale patch set review in,
>> say, projects like the Linux kernel. However, in about 7 years of
>> watching edk2-devel and opportunistically participating in patch
>> review myself, I never once encountered something about mail patch
>> review that made me think "oh, that's neat". Quite the opposite - I
>> cannot easily cross-reference when commenting, I cannot easily see
>> more context to the changed lines, and I cannot easily see the end
>> result after all patches in a series have been applied. These are all
>> things that GitHub allows me to do. I keep hearing mail patches "work
>> better", but I never found convincing reasons for these claims. Mind
>> sharing? :)
> 
> (1) Navigation works better for me.  On the email side I have the freedom
> to pick whatever client I like and can configure it the way I like.

Well, customization is not the kind of point one can argue against. :) OK, 
thanks.

> 
> (2) I can easily automate things.  For example it's just two key strokes
> in the mail client to run a script which creates a new branch and
> applies the whole patch series.
> 
> The latter is what I usually do when I want compile and test the series,
> or when I need something plain email doesn't give me (like getting more
> patch context, which indeed is a nice github feature).

Right, but I’d say especially with tools like VS Code, just checking out the PR 
branch (or even opening remotely without downloading it fully) is equally easy. 
This sounds more like a mitigation for the shortcomings of mail patches.

> 
>>>> (2) github doesn't preserve stuff like a mail archive does.  When a
>>>> patch series goes through multiple revision github only preserves
>>>> the latest revision which was actually merged.
> 
>> (I’m not sure whether the old stuff isn’t eventually wiped, though,
>> maybe worth carefully inspecting the documentation for options).
> 
> Yes, this.  For active PRs this usually isn't much of a problem.  But
> try come back after a few months, or even a few years (see Rebecca
> trying to lookup context for a 2016 commit in the archives).

I also didn’t find anything in the GutHub docs regarding guarantees for 
archival. Disgusting.

> 
>>>> * developer opens a draft PR to run CI for the patch series.
>>>> * when the series passes CI and is ready un-draft the PR.
>>>> * github action sends the patch series to the edk2-devel list
>>>>   for review (maybe only after CI passed ...).
>>>> * patch review happens on the list.
>>>> * in case the developer pushes updates to the branch in response to
>>>>   review comments the github action posts v2/v3 of the series too.
>>>> * once review is done merge the PR.
>> 
>> That would at least be a lot better than what we have now.
> 
> While discussing tooling:  Can we move from bugzilla to github issues
> for bug tracking?  That will give us some nice automation and
> integration benefits.  As far I know the blocker for doing that was
> github issues not having a permission system, which is bad for reporting
> security bugs.  But with security bug reporting and processing using
> github security advisories now this point should be moot, no?

Hmm, is there any big problem that requires a solution? While I always like 
streamlining and centralisation, and I would vote for GitHub Issues if it was 
about a new bugtracker, I never heard many complaints about bugzilla. Everyone 
is used to it and there’s a ton 

Re: [edk2-devel] [edk2-platforms] Enable GitHub PR, protected branches, and 'push' label

[Marvin Häuser]

Hi Rebecca and Gerd,

Replying to 2 mails at once...

> On 17. Mar 2023, at 11:36, Rebecca Cran  wrote:
> 
> I like that proposed workflow.
> 
> I've also been wondering if we could consider choosing a different product 
> for patch reviews that supports our desired workflow better, such as Gitlab 
> or Phorge (the new Phabricator project).

I'd be very cautious with suggesting / approving more tooling. It gets more 
confusing (what is hosted where), it gets more complicated to maintain (who 
hosts what and is "guaranteed" to be available to fix things), and so on.

> 
> If anyone would be willing to donate money for colocation, I'd be happy to 
> move one or more of my servers into a datacenter and use them for hosting 
> TianoCore services.
> 
> 
> -- 
> 
> Rebecca Cran
> 
> 
> On 3/17/23 3:33 AM, Gerd Hoffmann wrote:
>> On Thu, Mar 16, 2023 at 01:59:49PM -0600, Rebecca Cran wrote:
>>> Is this still a requirement since Laszlo's departure from the project?
>>> 
>>> I seem to recall it was him who made it a sticking point of moving to a
>>> GitHub PR workflow originally with the requirement to have emails of
>>> everything.
>> I think it is very useful to have everything on the mailing list for
>> a number of reasons:
>> 
>>  (1) In my experience reviewing patches, especially more complex ones,
>>  works better in email than in github PR workflows.

I have no experience with things like large-scale patch set review in, say, 
projects like the Linux kernel. However, in about 7 years of watching 
edk2-devel and opportunistically participating in patch review myself, I never 
once encountered something about mail patch review that made me think "oh, 
that's neat". Quite the opposite - I cannot easily cross-reference when 
commenting, I cannot easily see more context to the changed lines, and I cannot 
easily see the end result after all patches in a series have been applied. 
These are all things that GitHub allows me to do. I keep hearing mail patches 
"work better", but I never found convincing reasons for these claims. Mind 
sharing? :)

>>  (2) github doesn't preserve stuff like a mail archive does.  When a
>>  patch series goes through multiple revision github only preserves
>>  the latest revision which was actually merged.

Is that so? Taking this AUDK PR as an example: 
https://github.com/acidanthera/audk/pull/23
Note the review comments that say "Outdated" and thus refer to previous 
revisions of the PR. I can expand the comments to read their content, see their 
immediate context (of the previous change that did not end up getting merged) 
and I can click the file name to get an "all files changed" view of the 
snapshot that was reviewed. What is missing?
(I’m not sure whether the old stuff isn’t eventually wiped, though, maybe worth 
carefully inspecting the documentation for options).

>>  (3) Search engines seem to be better in indexing mail list archives
>>  than github pull requests.

That, unfortunately, undoubtably is true.

>> 
>> Nevertheless I see some room for improvement in our current workflow.
>> Developers often open a PR anyway for to run the CI.  So maybe we could
>> automate sending the emails and also avoid running CI twice by avoiding
>> both developer and maintainer opening a PR, with a workflow like this:
>> 
>>  * developer opens a draft PR to run CI for the patch series.
>>  * when the series passes CI and is ready un-draft the PR.
>>  * github action sends the patch series to the edk2-devel list
>>for review (maybe only after CI passed ...).
>>  * patch review happens on the list.
>>  * in case the developer pushes updates to the branch in response to
>>review comments the github action posts v2/v3 of the series too.
>>  * once review is done merge the PR.

That would at least be a lot better than what we have now.

Best regards,
Marvin

>> 
>> take care,
>>   Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101335): https://edk2.groups.io/g/devel/message/101335
Mute This Topic: https://groups.io/mt/97636248/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [edk2-platforms] Enable GitHub PR, protected branches, and 'push' label

[Marvin Häuser]

Well, in this form, it complicates our workflow and adds no value. NACK from 
Pedro and me till there at least is CI.

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101283): https://edk2.groups.io/g/devel/message/101283
Mute This Topic: https://groups.io/mt/97636248/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/3] BaseTools/Conf/tools_def: Fix linking using CLANGDWARF_IA32_X64

[Marvin Häuser]

> On 15. Mar 2023, at 23:51, Ard Biesheuvel  wrote:
> 
> On Wed, 15 Mar 2023 at 23:16, Marvin Häuser  wrote:
>> 
>> Hi,
>> 
>> Why does the title mention X64? From what I can see, PIE is unaffected for 
>> X64 (and we really want it to be).
>> 
> 
> Why?

Why what? By “PIE is unaffected for X64” I meant by the patch (which only 
changes IA32 macros, no?). I can also see how the last part is a bit ambiguous, 
but I meant it literally - we really want X64 to be [unaffected] (which it is, 
by the patch, but the title implies otherwise, doesn’t it?).

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101253): https://edk2.groups.io/g/devel/message/101253
Mute This Topic: https://groups.io/mt/97421035/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [edk2-platforms] Enable GitHub PR, protected branches, and 'push' label

[Marvin Häuser]

Hi Mike,

Could this be extended to allow for a full PR workflow, if the package 
maintainers would prefer so? We would like to utilise this for Ext4Pkg. It 
could be considered a trial. :)

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101250): https://edk2.groups.io/g/devel/message/101250
Mute This Topic: https://groups.io/mt/97636248/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/3] BaseTools/Conf/tools_def: Fix linking using CLANGDWARF_IA32_X64

[Marvin Häuser]

Hi,

Why does the title mention X64? From what I can see, PIE is unaffected for X64 
(and we really want it to be).

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#101249): https://edk2.groups.io/g/devel/message/101249
Mute This Topic: https://groups.io/mt/97421035/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] SecurityPkg/DxeImageVerificationLib: Check result of GetEfiGlobalVariable2

[Marvin Häuser]

> On 28. Feb 2023, at 07:12, Gerd Hoffmann  wrote:
> 
>   Hi,
> 
>>> (a) the SecureBoot variable is not present (EFI_NOT_FOUND) according to
>>>the return value, or
>> 
>> @Maintainers Would there be any objection to drop this and skip the SB 
>> checks only when explicitly disabled?
>> Please explicitly respond even if not, so we don't end up with everyone 
>> silently agreeing, but forgetting about the patch after. Thanks! :)
> 
> I hold back v2, waiting for an answer here.
> 
>>> -  if (*SecureBoot == SECURE_BOOT_MODE_DISABLE) {
>>> -FreePool (SecureBoot);
>>> +  if ((VarStatus == EFI_SUCCESS) && (SecureBoot == 
>>> SECURE_BOOT_MODE_DISABLE)) {
>> 
>> I would check the attributes here as well. They should be BS | RT, but
>> explicitly not NV. This would force the SB checks in case a malicious
>> actor somehow managed to store a persistent disable-value variable (be
>> that a bug, physical access, or other means).
> 
> Like this (incremental fixup)?

Sorry, I formulated it a bit vague - what I meant is that the attributes should 
be exactly BS | RT (i.e., equal to), but I see how adding it must not be NV 
sounds like it should be just those three Bits checked. Otherwise, yes, thanks 
a lot!

It’s a read-only status reporting variable, so even with future changes, 
setting any of the other attributes wouldn’t make much sense. BS | RT is what 
the spec currently dictates (there is a table here: 
https://uefi.org/specs/UEFI/2.10/03_Boot_Manager.html).

> 
> Do we have macros for variable attribute checking?
> Havn't seen anything while skimming variable-related headers ...

Don’t think so. Not sure there is much attribute-checking done to begin with 
outside VarCheckLib. The only stack I’ve seen doing this extensively is 
recent-years Mac EFI.

Best regards,
Marvin

> 
> take care,
>  Gerd
> 
> diff --git 
> a/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c 
> b/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> index f29a27e5a053..79c784f77ac8 100644
> --- a/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> +++ b/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> @@ -1688,6 +1688,7 @@ DxeImageVerificationHandler (
>   EFI_STATUSHashStatus;
>   EFI_STATUSDbStatus;
>   EFI_STATUSVarStatus;
> +  UINT32VarAttr;
>   BOOLEAN   IsFound;
> 
>   SignatureList = NULL;
> @@ -1745,7 +1746,7 @@ DxeImageVerificationHandler (
>   }
> 
>   SecureBootSize = sizeof (SecureBoot);
> -  VarStatus  = gRT->GetVariable (EFI_SECURE_BOOT_MODE_NAME, 
> , NULL, , );
> +  VarStatus  = gRT->GetVariable (EFI_SECURE_BOOT_MODE_NAME, 
> , , , );
>   //
>   // Skip verification if SecureBoot variable doesn't exist.
>   //
> @@ -1756,7 +1757,12 @@ DxeImageVerificationHandler (
>   //
>   // Skip verification if SecureBoot is disabled but not AuditMode
>   //
> -  if ((VarStatus == EFI_SUCCESS) && (SecureBoot == 
> SECURE_BOOT_MODE_DISABLE)) {
> +  if ((VarStatus == EFI_SUCCESS) &&
> +  !(VarAttr & EFI_VARIABLE_NON_VOLATILE) &&
> +  (VarAttr & EFI_VARIABLE_BOOTSERVICE_ACCESS) &&
> +  (VarAttr & EFI_VARIABLE_RUNTIME_ACCESS) &&
> +  (SecureBoot == SECURE_BOOT_MODE_DISABLE))
> +  {
> return EFI_SUCCESS;
>   }
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100543): https://edk2.groups.io/g/devel/message/100543
Mute This Topic: https://groups.io/mt/97265237/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] SecurityPkg/DxeImageVerificationLib: Check result of GetEfiGlobalVariable2

[Marvin Häuser]

*Thanks*, Gerd!

Two comments inline.

> On 27. Feb 2023, at 13:55, Gerd Hoffmann  wrote:
> 
> Call gRT->GetVariable() directly to read the SecureBoot variable.  It is
> one byte in size so we can easily place it on the stack instead of
> having GetEfiGlobalVariable2() allocate it for us, which avoids a few
> possible error cases.
> 
> Skip secure boot checks if (and only if):
> 
> (a) the SecureBoot variable is not present (EFI_NOT_FOUND) according to
> the return value, or

@Maintainers Would there be any objection to drop this and skip the SB checks 
only when explicitly disabled?
Please explicitly respond even if not, so we don't end up with everyone 
silently agreeing, but forgetting about the patch after. Thanks! :)

> (b) the SecureBoot variable was read successfully and is set to
> SECURE_BOOT_MODE_DISABLE.
> 
> Previously the code skipped the secure boot checks on *any*
> gRT->GetVariable() error (GetEfiGlobalVariable2 sets the variable
> value to NULL in that case) and also on memory allocation failures.
> 
> Fixes: CVE-2019-14560
> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=2167
> Signed-off-by: Gerd Hoffmann 
> ---
> .../DxeImageVerificationLib.c  | 14 +++---
> 1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git 
> a/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c 
> b/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> index 66e2f5eaa3c0..f29a27e5a053 100644
> --- a/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> +++ b/SecurityPkg/Library/DxeImageVerificationLib/DxeImageVerificationLib.c
> @@ -1671,7 +1671,8 @@ DxeImageVerificationHandler (
>   EFI_IMAGE_EXECUTION_ACTIONAction;
>   WIN_CERTIFICATE   *WinCertificate;
>   UINT32Policy;
> -  UINT8 *SecureBoot;
> +  UINT8 SecureBoot;
> +  UINTN SecureBootSize;
>   PE_COFF_LOADER_IMAGE_CONTEXT  ImageContext;
>   UINT32NumberOfRvaAndSizes;
>   WIN_CERTIFICATE_EFI_PKCS  *PkcsCertData;
> @@ -1686,6 +1687,7 @@ DxeImageVerificationHandler (
>   RETURN_STATUS PeCoffStatus;
>   EFI_STATUSHashStatus;
>   EFI_STATUSDbStatus;
> +  EFI_STATUSVarStatus;
>   BOOLEAN   IsFound;
> 
>   SignatureList = NULL;
> @@ -1742,24 +1744,22 @@ DxeImageVerificationHandler (
> CpuDeadLoop ();
>   }
> 
> -  GetEfiGlobalVariable2 (EFI_SECURE_BOOT_MODE_NAME, (VOID **), 
> NULL);
> +  SecureBootSize = sizeof (SecureBoot);
> +  VarStatus  = gRT->GetVariable (EFI_SECURE_BOOT_MODE_NAME, 
> , NULL, , );
>   //
>   // Skip verification if SecureBoot variable doesn't exist.
>   //
> -  if (SecureBoot == NULL) {
> +  if (VarStatus == EFI_NOT_FOUND) {
> return EFI_SUCCESS;
>   }
> 
>   //
>   // Skip verification if SecureBoot is disabled but not AuditMode
>   //
> -  if (*SecureBoot == SECURE_BOOT_MODE_DISABLE) {
> -FreePool (SecureBoot);
> +  if ((VarStatus == EFI_SUCCESS) && (SecureBoot == 
> SECURE_BOOT_MODE_DISABLE)) {

I would check the attributes here as well. They should be BS | RT, but 
explicitly not NV. This would force the SB checks in case a malicious actor 
somehow managed to store a persistent disable-value variable (be that a bug, 
physical access, or other means).

Best regards,
Marvin

> return EFI_SUCCESS;
>   }
> 
> -  FreePool (SecureBoot);
> -
>   //
>   // Read the Dos header.
>   //
> -- 
> 2.39.2
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100512): https://edk2.groups.io/g/devel/message/100512
Mute This Topic: https://groups.io/mt/97265237/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] Ext4Pkg: Move unicode collation initialization to Start()

[Marvin Häuser]

> On 21. Feb 2023, at 20:25, Pedro Falcato  wrote:
> 
> Marvin,
> 
> Do you want me to spin up a quick v2 without the
> Ext4IsCollationInitialized? Just doing the check internally in
> Ext4InitialiseUnicodeCollation.

It'd be my preferred solution, but it's just a detail.

Reviewed-by: Marvin Häuser mailto:mhaeu...@posteo.de>> in 
any case

> 
> -- 
> Pedro



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100423): https://edk2.groups.io/g/devel/message/100423
Mute This Topic: https://groups.io/mt/97037199/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] Ext4Pkg: Move unicode collation initialization to Start()

[Marvin Häuser]

> On 20. Feb 2023, at 08:52, Ard Biesheuvel  wrote:
> 
> On Fri, 17 Feb 2023 at 21:14, Pedro Falcato  wrote:
>> 
>> There have been reports[1] of failures to boot due to unicode collation
>> protocols not being available at Ext4Dxe load time. Therefore, attempt
>> to initialize unicode collation at Start() time, like done previously in
>> FatPkg/EnhancedFatDxe. By doing so, we move collation initialization
>> to BDS, where the module responsible for protocol installation should
>> have already been loaded and ran.
>> 
>> [1]: https://edk2.groups.io/g/devel/message/100312
>> 
>> Cc: Ard Biesheuvel 
>> Cc: Marvin Häuser 
>> Fixes: d9ceedca6c8f ("Ext4Pkg: Add Ext4Dxe driver.")
>> Signed-off-by: Pedro Falcato 
>> ---
>> RFC-ish patch that implements one of the possibilities discussed in [1].
>> Please test and give feedback.
>> 
>> Odd that this issue was never caught in any other platform. Do the RPi 
>> platforms have
>> something funky going on?
>> 
> 
> It could be something as simple as the order the drivers appear in the
> .FDF file.
> 
>> Features/Ext4Pkg/Ext4Dxe/Collation.c | 18 +-
>> Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.c   | 51 +++-
>> Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.h   | 11 ++
>> 3 files changed, 56 insertions(+), 24 deletions(-)
>> 
>> diff --git a/Features/Ext4Pkg/Ext4Dxe/Collation.c 
>> b/Features/Ext4Pkg/Ext4Dxe/Collation.c
>> index 91d172b1cb89..65c34c33ea93 100644
>> --- a/Features/Ext4Pkg/Ext4Dxe/Collation.c
>> +++ b/Features/Ext4Pkg/Ext4Dxe/Collation.c
>> @@ -1,7 +1,7 @@
>> /** @file
>>   Unicode collation routines
>> 
>> -  Copyright (c) 2021 Pedro Falcato All rights reserved.
>> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>>   Copyright (c) 2005 - 2017, Intel Corporation. All rights reserved.
>> 
>>   SPDX-License-Identifier: BSD-2-Clause-Patent
>> @@ -9,6 +9,7 @@
>> 
>> #include 
>> 
>> +#include 
>> #include 
>> #include 
>> #include 
>> @@ -169,5 +170,20 @@ Ext4StrCmpInsensitive (
>>   IN CHAR16  *Str2
>>   )
>> {
>> +  ASSERT (gUnicodeCollationInterface != NULL);
>>   return gUnicodeCollationInterface->StriColl (gUnicodeCollationInterface, 
>> Str1, Str2);
>> }
>> +
>> +/**
>> +   Check if unicode collation is initialized
>> +
>> +   @retval TRUE if Ext4InitialiseUnicodeCollation() was already called 
>> successfully
>> +   @retval FALSE if Ext4InitialiseUnicodeCollation() was not yet called 
>> successfully
>> +**/
>> +BOOLEAN
>> +Ext4IsCollationInitialized (
>> +  VOID
>> +  )
>> +{
>> +  return gUnicodeCollationInterface != NULL;
>> +}
>> diff --git a/Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.c 
>> b/Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.c
>> index 2a4f5a7bd0ef..de0d633febfb 100644
>> --- a/Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.c
>> +++ b/Features/Ext4Pkg/Ext4Dxe/Ext4Dxe.c
>> @@ -1,7 +1,7 @@
>> /** @file
>>   Driver entry point
>> 
>> -  Copyright (c) 2021 Pedro Falcato All rights reserved.
>> +  Copyright (c) 2021 - 2023 Pedro Falcato All rights reserved.
>>   SPDX-License-Identifier: BSD-2-Clause-Patent
>> **/
>> 
>> @@ -513,26 +513,18 @@ Ext4EntryPoint (
>>   IN EFI_SYSTEM_TABLE  *SystemTable
>>   )
>> {
>> -  EFI_STATUS  Status;
>> -
>> -  Status = EfiLibInstallAllDriverProtocols2 (
>> - ImageHandle,
>> - SystemTable,
>> - ,
>> - ImageHandle,
>> - ,
>> - ,
>> - NULL,
>> - NULL,
>> - NULL,
>> - NULL
>> - );
>> -
>> -  if (EFI_ERROR (Status)) {
>> -return Status;
>> -  }
>> -
>> -  return Ext4InitialiseUnicodeCollation (ImageHandle);
>> +  return EfiLibInstallAllDriverProtocols2 (
>> +   ImageHandle,
>> +   SystemTable,
>> +   ,
>> +   ImageHandle,
>> +   ,
>> +   ,
>> +   NULL,
>> +   NULL,
>> +   NULL,
>> +   NULL
>> +   );
>> }
>> 
>> /**
>> @@ -761,6 +753,19 @@ Ext4Bind (
>>   BlockIo = NULL;
>>   DiskIo  = NULL;
>> 
>> +  // Note: We initialize collation here since this is called in BDS, when 
>> we are likely
>> +  // to have the Unicode Collation protocols available.
>> +  if (!Ext4IsCollati

Re: [edk2-devel] [PATCH 1/1] Ext4Pkg: Move unicode collation initialization to Start()

[Marvin Häuser]

> On 18. Feb 2023, at 16:14, Pedro Falcato  wrote:
> 
> Proper collation (and proper unicode handling in general) is AIUI
> pretty hard and involves a lot of tables, etc.
> So it makes total sense to me why one wants to "dynamically link" this
> in using protocols instead of statically linking it everywhere.
> 
> Should be noted that EnglishDxe makes a mockery out of unicode
> collation and does plain ASCII operations only :))

Yes, and so does FatPei [1]. Also, EnglishDxe has a clear (and unvalidated!) 
assumption that FAT file names are strictly 7-bit ASCII [2]. Without doing any 
proper research, Wikipedia seems to suggest that even EASCII could be 
problematic for FAT file names. Under the assumption that FAT will only ever 
give us 7-bit ASCII file names, it's both unrealistic to assume someone will 
implement proper Unicode collation, but also to fear any observable deviations 
from FatDxe even if this was the case and Ext4Dxe used ASCII-only collation 
(because the extended Unicode collation support would be no-op w.r.t. FAT). The 
ASCII-only support is pretty tiny, it could live in BaseLib without a problem 
(and not be duplicated into FatPkg...).

In my opinion, merge this to fix the bug and at some later point change this to 
static linking (maybe alongside FatDxe, but definitely deduplicate FatPei).

[1] 
https://github.com/tianocore/edk2/tree/02fcfdce1e5ce86f1951191883e7e30de5aa08be/FatPei/FatLiteLib.c#L341-L365
 


[2] 
https://github.com/tianocore/edk2/tree/02fcfdce1e5ce86f1951191883e7e30de5aa08be/MdeModulePkg/Universal/Disk/UnicodeCollation/EnglishDxe/UnicodeCollationEng.c#L386-L406
 

> I did think about that, but then decided against it since I felt that
> initializing multiple times would make little logical sense.

I get it, but moving it inside would be more consistent with similar code and 
remove the risk for forgetting the check if it was ever used elsewhere (I know, 
it probably won't be, but this is a design question).

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100348): https://edk2.groups.io/g/devel/message/100348
Mute This Topic: https://groups.io/mt/97037199/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 1/1] Ext4Pkg: Move unicode collation initialization to Start()

[Marvin Häuser]

> On 17. Feb 2023, at 21:14, Pedro Falcato  wrote:
> 
> @@ -169,5 +170,20 @@ Ext4StrCmpInsensitive (
>   IN CHAR16  *Str2
>   )
> {
> +  ASSERT (gUnicodeCollationInterface != NULL);
>   return gUnicodeCollationInterface->StriColl (gUnicodeCollationInterface, 
> Str1, Str2);
> }

Off-topic (should merge either way), but is there a reason such a function is 
not in BaseLib?
@Ard Do you happen to know?

> /**
> @@ -761,6 +753,19 @@ Ext4Bind (
>   BlockIo = NULL;
>   DiskIo  = NULL;
> 
> +  // Note: We initialize collation here since this is called in BDS, when we 
> are likely
> +  // to have the Unicode Collation protocols available.
> +  if (!Ext4IsCollationInitialized ()) {

Why do we need to expose this? I.e., why can't this be an internal decision of 
Ext4InitialiseUnicodeCollation()?

> +Status = Ext4InitialiseUnicodeCollation (BindingProtocol->ImageHandle);



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100346): https://edk2.groups.io/g/devel/message/100346
Mute This Topic: https://groups.io/mt/97037199/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH edk2-platforms 1/4] Ext4Pkg: Use depex for unicode collation protocols

[Marvin Häuser]

> On 17. Feb 2023, at 16:17, Ard Biesheuvel  wrote:
> 
> So the FAT driver will happily load, but then fail in an obscure
> manner when being started on a controller handle, in a way that is
> indistinguishable (afict) from a partition that has not FAT file
> system in the first place.
> 
> So it doesn't look fine to me at all, tbh. If the collation protocols
> are required, we should check for them in supported(), and make a lot
> of noise in a DEBUG build if the failure condition is an unexpected
> one, i.e., some additional protocol we dependent on (even though we
> pretend to be a UEFI driver)  does not exist.

Right, I meant the approach to rely on the UEFI Driver Binding handling to 
defer locating UC, not the exact implementation. Sounds good.

@Pedro What do you think?

> 
> Alternatively, we might defer installation of the driver binding
> protocol implementation to a callback on the collation protocol
> installation, but that seems like more work for the same result.

Agreed.

> I will +1 any solution that removes the DXE dispatcher entirely as a
> side effect, but unfortunately, that is not going to happen :-)

We will probably attempt something like this downstream at some point (we do 
not care about UEFI PI), so even more reason I dislike dependencies on its 
workings. In a lot of cases (like the architectural protocols), the dependency 
situation is much more simple than what Depex supports. I.e., the protocols 
often are unconditionally installed in the entry point. Then we can just 
re-order the modules at FD build-time rather than relying on runtime code in 
the dispatcher. Complex dependencies, if they cannot be avoided, may be 
rewritten with events, not sure yet.

> We could depex on the UEFI driver prerequisites as well as the
> collation protocol ones, arguably retaining the UEFI_DRIVER status.

That's what I meant, yes, but I prefer the above solution myself.

Best regards,
Marvin



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100324): https://edk2.groups.io/g/devel/message/100324
Mute This Topic: https://groups.io/mt/97025926/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH edk2-platforms 1/4] Ext4Pkg: Use depex for unicode collation protocols

[Marvin Häuser]

> On 17. Feb 2023, at 15:29, Ard Biesheuvel  wrote:
> 
> On Fri, 17 Feb 2023 at 15:05, Marvin Häuser  wrote:
>> 
>> Hi Ard,
>> 
>> Thank you! Is "1/4" a mistake or did I miss the other 3? :)
> 
> Oops.
> 
> It was part of some RPi4 patches but I decided to send it out by itself.
> 
> 
>> Comments inline.
>> 
>> On 17. Feb 2023, at 12:12, Ard Biesheuvel  wrote:
>> 
>> The Unicode collation protocols, however, are different: loading the
>> driver will fail if neither of those are present. So they are not
>> TO_START protocols, and they need to be mentioned in the DEPEX so that
>> the DXE core will not dispatch the driver before the producers of the
>> prerequisite protocols have been dispatched. Also, mark them as
>> SOMETIMES_CONSUMES, as only one of the two is required.
>> 
>> 
>> Right. FatPkg solves this by probing for the protocol in Start() [1], which 
>> should guarantee that all entry points have been executed first, right? I'd 
>> prefer a universal and consistent solution to the issue and this looks fine, 
>> honestly.
>> 
>> [1]
>> https://github.com/tianocore/edk2/blob/02fcfdce1e5ce86f1951191883e7e30de5aa08be/FatPkg/EnhancedFatDxe/Fat.c#L381
>> https://github.com/tianocore/edk2/blob/02fcfdce1e5ce86f1951191883e7e30de5aa08be/FatPkg/EnhancedFatDxe/Fat.inf#L19
>> 
> 
> I'd prefer using existing features for this, rather than open coding it.

It‘s not like we replicate a feature, we just move a function call to suit the 
control flow better. I‘d like a consistent solution and the FatPkg one looks 
fine to me.

> 
> 
>> -  MODULE_TYPE= UEFI_DRIVER
>> +  MODULE_TYPE= DXE_DRIVER
>> 
>> 
>> Is it not unidiomatic to use the UEFI Driver Binding model (UEFI) in a DXE 
>> driver (UEFI PI)?
>> 
> 
> Perhaps. But having a hard dependency on protocols beyond the default
> set defined for UEFI drivers is arguably even worse.

This is still very much a UEFI driver in a logical sense, this is just abusing 
the types to utilise DXE concepts. I‘m not opposed to such things in general, 
but here it looks unnecessary. It doesn’t help I’m not a big fan of the DXE 
dispatcher to begin with. :)

I agree the dependency is awkward, but I have to check the reason and 
alternatives later. In the end, it‘s Pedro‘s call anyway.

> 
> 
>> +[Depex]
>> +  gEfiUnicodeCollationProtocolGuid OR gEfiUnicodeCollation2ProtocolGuid
>> 
>> 
>> Hmm, this will have the side effect that Ext4Dxe may load before (some of) 
>> the architectural protocols, right (modulo implicit dependencies via the UC 
>> protocols)? This would need some careful analysis, or we need to add all of 
>> the architectural protocols to preserve the old behaviour.
>> 
> 
> The ext4 driver does nothing except install protocols in its
> entrypoint, and everything else that happens is under the control of
> the UEFI driver model, afaict.

I guess. There also is a chance that libraries like DebugLib enable advanced 
features only as core services become available. But probably not a big deal.

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100322): https://edk2.groups.io/g/devel/message/100322
Mute This Topic: https://groups.io/mt/97025926/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH edk2-platforms 1/4] Ext4Pkg: Use depex for unicode collation protocols

[Marvin Häuser]

Hi Ard,

Thank you! Is "1/4" a mistake or did I miss the other 3? :)
Comments inline.

> On 17. Feb 2023, at 12:12, Ard Biesheuvel  wrote:
> 
> The Unicode collation protocols, however, are different: loading the
> driver will fail if neither of those are present. So they are not
> TO_START protocols, and they need to be mentioned in the DEPEX so that
> the DXE core will not dispatch the driver before the producers of the
> prerequisite protocols have been dispatched. Also, mark them as
> SOMETIMES_CONSUMES, as only one of the two is required.

Right. FatPkg solves this by probing for the protocol in Start() [1], which 
should guarantee that all entry points have been executed first, right? I'd 
prefer a universal and consistent solution to the issue and this looks fine, 
honestly.

[1]
https://github.com/tianocore/edk2/blob/02fcfdce1e5ce86f1951191883e7e30de5aa08be/FatPkg/EnhancedFatDxe/Fat.c#L381
https://github.com/tianocore/edk2/blob/02fcfdce1e5ce86f1951191883e7e30de5aa08be/FatPkg/EnhancedFatDxe/Fat.inf#L19

> -  MODULE_TYPE= UEFI_DRIVER
> +  MODULE_TYPE= DXE_DRIVER

Is it not unidiomatic to use the UEFI Driver Binding model (UEFI) in a DXE 
driver (UEFI PI)?

> +[Depex]
> +  gEfiUnicodeCollationProtocolGuid OR gEfiUnicodeCollation2ProtocolGuid

Hmm, this will have the side effect that Ext4Dxe may load before (some of) the 
architectural protocols, right (modulo implicit dependencies via the UC 
protocols)? This would need some careful analysis, or we need to add all of the 
architectural protocols to preserve the old behaviour.

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100320): https://edk2.groups.io/g/devel/message/100320
Mute This Topic: https://groups.io/mt/97025926/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [edk2-platforms][PATCH v4 1/1] Ext4Pkg: Fixes double-free in Ext4ReadSymlink

[Marvin Häuser]

Reviewed-by: Marvin Häuser 

> On 17. Feb 2023, at 09:56, Savva Mitrofanov  wrote:
> 
> The SymlinkTmp was deallocated unconditionally, so we shouldn't free it
> again on EFI_ERROR
> 
> Cc: Marvin Häuser 
> Cc: Pedro Falcato 
> Cc: Vitaly Cheptsov 
> Fixes: e81432fbacb7 ("Ext4Pkg: Add symbolic links support")
> Signed-off-by: Savva Mitrofanov 
> ---
> Features/Ext4Pkg/Ext4Dxe/Symlink.c | 1 -
> 1 file changed, 1 deletion(-)
> 
> diff --git a/Features/Ext4Pkg/Ext4Dxe/Symlink.c 
> b/Features/Ext4Pkg/Ext4Dxe/Symlink.c
> index 1189a99ded2b..d80cb9fff92e 100644
> --- a/Features/Ext4Pkg/Ext4Dxe/Symlink.c
> +++ b/Features/Ext4Pkg/Ext4Dxe/Symlink.c
> @@ -243,7 +243,6 @@ Ext4ReadSymlink (
>   Status
>   ));
> FreePool (Symlink16Tmp);
> -FreePool (SymlinkTmp);
> return Status;
>   }
> 
> -- 
> 2.39.1
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100310): https://edk2.groups.io/g/devel/message/100310
Mute This Topic: https://groups.io/mt/97024664/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC 07/13] MdeModulePkg/DxeCore: Permit preliminary CPU arch fallback

[Marvin Häuser]

Sounds good to me, thanks!

Best regards,
Marvin

> On 13. Feb 2023, at 23:07, Ard Biesheuvel  wrote:
> 
> On Mon, 13 Feb 2023 at 22:32, Marvin Häuser  wrote:
>> 
>> Without wanting to blow up your RFC with another one - I discussed this with 
>> various people, including Bret when he was still at Project Mu, and there 
>> was a consensus among them that integrating the whole CPU arch code right 
>> into DxeCore would be a good idea. This would especially remove the hack 
>> that queues permission applications till CpuDxe loads for good, rather than 
>> requiring pro-active consumption of a library that proves this "fallback". 
>> For most of the architectural protocols, especially SecurityStubDxe, I never 
>> got the gist why you would want them to be separate from DxeCore. Obviously 
>> there should be a level of customizability for IBVs and OEMs, though that 
>> can be done statically-linked as well.
>> 
>> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3223
>> 
>> What's your take on this?
>> 
> 
> My take is that page table manipulation should not be part of the CPU
> arch protocol. The DXE core loads images and dispatches them, which
> also involves cache maintenance, for instance, as code pages need to
> be invalidated from the I-cache before you can execute a newly loaded
> image. I think it makes perfect sense for the DXE core to be in charge
> of updating the page table descriptors as well.
> 
> In my approach, the library version is only used before the CPU arch
> protocol appears, so it addresses some of the concerns regarding
> multiple owners. But I'd prefer to see this removed from the CPU arch
> protocol entirely, or at least for the CPU arch protocol interface to
> be deprecated, and redefined in terms of a new DXE services, for
> instance, that is implemented by the DXE core itself. That way, all
> existing users would still see the same protocols, but the way they
> depend on each other would be inverted.



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100141): https://edk2.groups.io/g/devel/message/100141
Mute This Topic: https://groups.io/mt/96937485/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC 13/13] ArmVirtPkg/ArmVirtQemu: Enable hardware enforced W^X memory permissions

[Marvin Häuser]

> On 13. Feb 2023, at 22:59, Ard Biesheuvel  wrote:
> 
> No, the only reason for adding -fpie here is to ensure that statically
> initialized CONST pointers are emitted into .data.rel.ro and not into
> .rodata, as this is under the control of the compiler. Although,
> thinking about this, I wonder if we need to pass this to the linker
> for codegen under LTO as well. But the PIE link itself should be
> unnecessary here.

Oh, what fun. For some reason I thought it would be unsafe to specify -fpie but 
not -pie, but considering PIE relocs are ignored either way, this actually 
makes perfect sense. Sorry! About that last part, the docs say: "It is 
recommended that you compile all the files participating in the same link with 
the same options and also specify those options at link time." [1], so good 
catch!

[1] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options

But what about -fno-plt?

> 
> It will if you pass -pie to the linker, which is why I would prefer to
> avoid that. The main issue IIRC is that the emit-relocs section does
> not cover the entries in the GOT table that also require relocation,
> and are only covered by the PIE .rela section. For AArch64, I added
> relaxation logic to GenFw to actually patch the instructions instead,
> which is always possible given the absence of dynamic linking.
> (d2687f23c909475d80cef32cdf9a5d121f0a9ae6,
> 7b8f69d7e10628d473dd225224d8c2122d25a38d)

Yes, seen, very nice. I do wonder though why GOT entries are generated in the 
first place when symbols are all local and data is within the PC-addressable 
range. Just today, for a X64 build, I actually saw Clang relax a GOT reference 
to __stack_chk_guard itself.

> 
> This means that we don't have to care about compiler generated symbol
> references, and so the relocs emitted by emit-relocs are sufficient,
> and the additional ones emitted into .rela are unused anyway. The only
> remaining absolute references are the ones resulting from statically
> initialized globals, and those will either be in .data or in
> .data.rel.ro (if -fpie is being used)

Right. thank you.

Best regards,
Marvin

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100140): https://edk2.groups.io/g/devel/message/100140
Mute This Topic: https://groups.io/mt/96937498/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC 07/13] MdeModulePkg/DxeCore: Permit preliminary CPU arch fallback

[Marvin Häuser]

Without wanting to blow up your RFC with another one - I discussed this with 
various people, including Bret when he was still at Project Mu, and there was a 
consensus among them that integrating the whole CPU arch code right into 
DxeCore would be a good idea. This would especially remove the hack that queues 
permission applications till CpuDxe loads for good, rather than requiring 
pro-active consumption of a library that proves this "fallback". For most of 
the architectural protocols, especially SecurityStubDxe, I never got the gist 
why you would want them to be separate from DxeCore. Obviously there should be 
a level of customizability for IBVs and OEMs, though that can be done 
statically-linked as well.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3223

What's your take on this?

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100134): https://edk2.groups.io/g/devel/message/100134
Mute This Topic: https://groups.io/mt/96937485/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [RFC 13/13] ArmVirtPkg/ArmVirtQemu: Enable hardware enforced W^X memory permissions

[Marvin Häuser]

Hey Ard,

*Praise* to you for this series. Comments inline.

On Mon, Feb 13, 2023 at 07:19 AM, Ard Biesheuvel wrote:

> 
> Enable the WXN system control bit straight out of reset when running in
> EL1 with the initial ID map from flash. This setting will be inherited
> by the page table code after it sets up the permanent boot time page
> tables, resulting in all memory mappings that are not explicitly mapped
> as read-only to be non-executable.
> 
> Note that this requires runtime drivers to be built with position
> independent codegen, to ensure that all absolute symbol references are
> moved into a separate section in the binary. Otherwise, unmapping the
> pages that are subject to relocation fixups at runtime (during the
> invocation of SetVirtualAddressMap()) could result in code mappings
> losing their executable permissions.

I never actually thought about this. SetVirtualAddressMap() will have to 
relocate its own parent binary, causing issues for software W^X when .text 
relocs are present (like with MSVC builds). :(

> 
> 
> Signed-off-by: Ard Biesheuvel 
> ---
> ArmVirtPkg/ArmVirt.dsc.inc | 1 +
> ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 2 +-
> 2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/ArmVirtPkg/ArmVirt.dsc.inc b/ArmVirtPkg/ArmVirt.dsc.inc
> index 5b18184be263..928dd6330edb 100644
> --- a/ArmVirtPkg/ArmVirt.dsc.inc
> +++ b/ArmVirtPkg/ArmVirt.dsc.inc
> @@ -31,6 +31,7 @@
> [BuildOptions.common.EDKII.DXE_CORE,BuildOptions.common.E=
> DKII.DXE_DRIVER,BuildOp
> =0D
> [BuildOptions.common.EDKII.DXE_RUNTIME_DRIVER]=0D
> GCC:*_*_ARM_DLINK_FLAGS =3D -z common-page-size=3D0x1000=0D
> + GCC:*_*_AARCH64_CC_FLAGS =3D -fpie=0D

Doesn't this mean -pie must be passed to the linker? I saw in the previous 
patch that .plt was added to the linker script, was there a particular reason 
-fno-plt wasn't used here? I just read it may have some unexpected 
side-effects, but I thought it would be safe for our statically-linked UEFI 
environment.

On another (related) matter, I've been spending my last two days looking into 
the whole ELF-to-PE process, because GenFw has been becoming unbearable to us 
downstream. I went through a bunch of old commits which deal with PIE and saw 
it was usually disabled but for X64. The funny thing with X64 (even currently) 
is, that -fpie is combined with -q (a.k.a. --emit-relocs), yielding both object 
file relocs (.rela.sectname) and PIE-related relative relocs (.rela) in the 
same binary (as documented in GenFw, they may overlap!). It's my understanding 
that GenFw currently processes exclusively the -q relocs and not the -fpie 
relocs (which should be safe as done for X64, I have no experience with ARM 
whatsoever). However, when PIE is involved anyway, it makes most sense to me to 
use its related relocs for the translation over a dance with the object file 
relocs. This change will cause the same behaviour for AARCH64 RT drivers now, 
right?

In an ideal world, I suppose all architectures but IA32 (due to lacking 
efficient pcrel addressing) should be using PIE, as most (often all with X64) 
GOT references can be relaxed, as we strictly deal with local symbols. Though I 
have to wonder how unideal the world really is. :)

Best regards,
Marvin

> 
> GCC:*_*_AARCH64_DLINK_FLAGS =3D -z common-page-size=3D0x1=0D
> =0D
> [LibraryClasses.common]=0D
> diff --git
> a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelpe=
> r.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> index 5ac7c732f6ec..51c089a45ffc 100644
> --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> @@ -38,7 +38,7 @@
> .set SCTLR_EL1_ITD, 0x1 << 7=0D
> .set SCTLR_EL1_RES1, (0x1 << 11) | (0x1 << 20) | (0x1 << 22) | (0=
> x1 << 28) | (0x1 << 29)=0D
> .set sctlrval, SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA | SCTLR_EL1_IT=
> D | SCTLR_EL1_SED=0D
> - .set sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES=
> 1=0D
> + .set sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES=
> 1 | SCTLR_EL1_WXN=0D
> =0D
> =0D
> ASM_FUNC(ArmPlatformPeiBootAction)=0D
> --=20
> 2.39.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#100133): https://edk2.groups.io/g/devel/message/100133
Mute This Topic: https://groups.io/mt/96937498/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 3/5] UefiCpuPkg/PiSmmCpuDxeSmm: Consume SMM Base Hob for SmBase info

[Marvin Häuser]

Hi Jiaxin,

1) mSmmInitialized *must* be volatile. Your current code may cause anything, 
from skipping waiting entirely (the loop is optimized away as the compiler 
considers it free from side effects) to stalling (the memory access is 
optimized away as the compiler considers it locally-immutable).

2) ASSERTs on memory allocation failures are one of the most terrible edk2 
"paradigms".

3) Comparisons against boolean constants are not allowed by the code style spec.

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#99971): https://edk2.groups.io/g/devel/message/99971
Mute This Topic: https://groups.io/mt/96871374/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v4 2/5] UefiCpuPkg/SmmBaseHob.h: Add SMM Base HOB Data

[Marvin Häuser]

Hi Ray and Jiaxin,

Quick reminder that [1] invokes undefined behaviour and [0] is not legal C, but 
a compiler extension. The canonical way is a flexible array member using [].

Off-topic and nitpicking, but I started wondering why so many structures are 
packed. For UEFI, packing would only help saving RAM when there‘s actual 
padding (unlike this case, where there isn‘t). All architectures require 
natural data alignment („unless specifies otherwise“), so there should be no 
compatibility concerns for structures specific to UEFI and UEFI PI ever. When 
considering ABIs outside UEFI, even then natural alignment is the strictest 
realistically possible outside super niche platforms, as even executable 
formats don‘t pack their structures (this structure also is inherently 
naturally aligned).

Best regards,
Marvin


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#99968): https://edk2.groups.io/g/devel/message/99968
Mute This Topic: https://groups.io/mt/96871372/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 3/3] MdeModulePkg/DxeCore: Unconditionally set memory protections

[Marvin Häuser]

> On 8. Feb 2023, at 19:49, Ard Biesheuvel  wrote:
> 
> This is all copy-pasted from MdeModulePkg/Core/Dxe/Misc/MemoryProtection.c

:(

> 
> The ordering here is a bit tricky. As soon as the CPU arch protocol is
> exposed, every allocation will be remapped individually, resulting in
> page table splits and therefore recursion.

So the issue is the order of event handlers or allocations within the event 
dispatcher? :( Oh lord...

Can we maybe clarify the comment with something like "While 
DxeCore/InitializeDxeNxMemoryProtectionPolicy() would in theory perform this 
task, allocations between the protocol installation and the invocation of its 
event handler may trigger the issue."?

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#99812): https://edk2.groups.io/g/devel/message/99812
Mute This Topic: https://groups.io/mt/96835917/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 2/3] ArmPkg/CpuDxe: Perform preliminary NX remap of free memory

[Marvin Häuser]

Thanks!! :) Comments inline.

> On 8. Feb 2023, at 18:58, Ard Biesheuvel  wrote:
> 
> The DXE core implementation of PcdDxeNxMemoryProtectionPolicy already
> contains an assertion that EfiConventionalMemory and EfiBootServicesData
> are subjected to the same policy when it comes to the use of NX
> permissions. The reason for this is that we may otherwise end up with
> unbounded recursion in the page table code, given that allocating a page
> table would then involve a permission attribute change, and this could
> result in the need for a block entry to be split, which would trigger
> the allocation of a page table recursively.
> 
> For the same reason, a shortcut exists in ApplyMemoryProtectionPolicy()
> where, instead of setting the memory attributes unconditionally, we
> compare the NX policies and avoid touching the page tables if they are
> the same for the old and the new memory types. Without this shortcut, we
> may end up in a situation where, as the CPU arch protocol DXE driver is
> ramping up, the same unbounded recursion is triggered, due to the fact
> that the NX policy for EfiConventionalMemory has not been applied yet.
> 
> To break this cycle, let's remap all EfiConventionalMemory regions
> according to the NX policy for EfiBootServicesData before exposing the
> CPU arch protocol to the DXE core and other drivers. This ensures that
> creating EfiBootServicesData allocations does not result in memory
> attribute changes, and therefore no recursion.
> 
> Signed-off-by: Ard Biesheuvel 
> ---
> ArmPkg/Drivers/CpuDxe/CpuDxe.c   | 77 
> ArmPkg/Drivers/CpuDxe/CpuDxe.inf |  2 +
> 2 files changed, 79 insertions(+)
> 
> diff --git a/ArmPkg/Drivers/CpuDxe/CpuDxe.c b/ArmPkg/Drivers/CpuDxe/CpuDxe.c
> index d04958e79e52..83fd6fd4e476 100644
> --- a/ArmPkg/Drivers/CpuDxe/CpuDxe.c
> +++ b/ArmPkg/Drivers/CpuDxe/CpuDxe.c
> @@ -11,6 +11,8 @@
> 
> #include 
> 
> +#include 
> +
> BOOLEAN  mIsFlushingGCD;
> 
> /**
> @@ -227,6 +229,69 @@ InitializeDma (
>   CpuArchProtocol->DmaBufferAlignment = ArmCacheWritebackGranule ();
> }
> 
> +STATIC
> +VOID
> +RemapUnusedMemoryNx (
> +  VOID
> +  )
> +{
> +  UINT64 TestBit;
> +  UINTN  MemoryMapSize;
> +  UINTN  MapKey;
> +  UINTN  DescriptorSize;
> +  UINT32 DescriptorVersion;
> +  EFI_MEMORY_DESCRIPTOR  *MemoryMap;
> +  EFI_MEMORY_DESCRIPTOR  *MemoryMapEntry;
> +  EFI_MEMORY_DESCRIPTOR  *MemoryMapEnd;
> +  EFI_STATUS Status;
> +
> +  TestBit = LShiftU64 (1, EfiBootServicesData);
> +  if ((PcdGet64 (PcdDxeNxMemoryProtectionPolicy) & TestBit) == 0) {
> +return;
> +  }
> +
> +  MemoryMapSize = 0;
> +  MemoryMap = NULL;
> +
> +  Status = gBS->GetMemoryMap (
> +  ,
> +  MemoryMap,
> +  ,
> +  ,
> +  
> +  );
> +  ASSERT (Status == EFI_BUFFER_TOO_SMALL);
> +  do {
> +MemoryMap = (EFI_MEMORY_DESCRIPTOR *)AllocatePool (MemoryMapSize);

nitpick: *If* there is a V2, maybe drop the cast?

> +ASSERT (MemoryMap != NULL);

I know ASSERTing for NULL is a common pattern for some reason, but I'd rather 
not have more code with this added.

> +Status = gBS->GetMemoryMap (
> +,
> +MemoryMap,
> +,
> +,
> +
> +);

Another nitpick, isn't it common practice to call the Core* functions directly 
within *Core? I know there is code that doesn't, but the former should be more 
efficient.

> +if (EFI_ERROR (Status)) {
> +  FreePool (MemoryMap);
> +}
> +  } while (Status == EFI_BUFFER_TOO_SMALL);

Is this guaranteed to terminate? I mean, I get the idea - try again with the 
larger size and when allocating the bigger buffer, its potential new entry will 
already be accounted for. However, I saw downstream code that tried something 
like this (they actually added a constant overhead ahead of time) bounded by 
something like 5 iterations. Might just have been defensive programming - 
probably was -, but it's not trivial to verify, I think, is it?

Regarding the added constant, the spec actually says the following, which 
obviously is just to shortcut a second round of GetMemoryMap(), but still:
"The actual size of the buffer allocated for the consequent call to 
GetMemoryMap() should be bigger then the value returned in MemoryMapSize"

It appears the spec did not define a canonical way to call GetMemoryMap(), did 
it? :(

> +
> +  ASSERT_EFI_ERROR (Status);
> +
> +  MemoryMapEntry = MemoryMap;
> +  MemoryMapEnd   = (EFI_MEMORY_DESCRIPTOR *)((UINT8 *)MemoryMap + 
> MemoryMapSize);
> +  while ((UINTN)MemoryMapEntry < (UINTN)MemoryMapEnd) {
> +if (MemoryMapEntry->Type == EfiConventionalMemory) {
> +  ArmSetMemoryRegionNoExec (
> +MemoryMapEntry->PhysicalStart,
> +EFI_PAGES_TO_SIZE 

Re: [edk2-devel] [PATCH 4/4] ArmPkg/CpuDxe: Implement EFI memory attributes protocol

[Marvin Häuser]

> On 7. Feb 2023, at 19:19, Taylor Beebe  wrote:
> 
> If I understand Marvin correctly, he means that there either needs to be a 
> requirement that if you change the attributes of an allocated buffer you must 
> change them back before freeing, or the memory management logic should handle 
> the possibility that the memory attributes may have changed since allocation.

Yes. More explicitly, I fear the former is what is happening and the latter is 
what the protocol and DXE service descriptions suggest.

Best regards,
Marvin

> In my opinion introducing the Memory Attribute Protocol requires more 
> attribute accounting when allocating and freeing. I believe the two changes 
> linked below ensure that attributes are properly set.
> 
> 1. 
> https://github.com/microsoft/mu_basecore/blob/788c17f750323efc206cdb042a2c14a587dcf27a/MdeModulePkg/Core/Dxe/Mem/Page.c#L1570
> 2. 
> https://github.com/microsoft/mu_basecore/blob/788c17f750323efc206cdb042a2c14a587dcf27a/MdeModulePkg/Core/Dxe/Misc/MemoryProtection.c#L1562



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#99762): https://edk2.groups.io/g/devel/message/99762
Mute This Topic: https://groups.io/mt/96664071/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH 4/4] ArmPkg/CpuDxe: Implement EFI memory attributes protocol

[Marvin Häuser]

> On 7. Feb 2023, at 18:56, Ard Biesheuvel  wrote:
> 
> On Tue, 7 Feb 2023 at 11:13, Marvin Häuser  wrote:
>> 
>> 
>> On 7. Feb 2023, at 11:01, Ard Biesheuvel  wrote:
>> 
>> Actually, it seems UnprotectUefiImage () is corrent under the
>> assumption that all code regions have EFI_MEMORY_XP cleared by
>> default.
>> 
>> However, if you redefine the policy to set EFI_MEMORY_XP on code
>> regions by default, and only permit execution after remapping the code
>> read-only explicitly, and only then clearing EFI_MEMORY_XP, that
>> routine should revert the region to EFI_MEMORY_XP. But given the
>> existing ASSERT()s on having EFI_MEMORY_XP cleared for all code
>> regions, the code as it is currently is not incorrect.
>> 
>> 
>> Right. My main issue is, it’s nowhere documented that manually changed 
>> permissions must be restored to their default before freeing. Within 
>> DxeCore, this is easily done using the PCDs, but outside (say you allocate a 
>> trampoline buffer and then free it), you would need to manually query the 
>> permissions, store them, and restore later.
>> 
> 
> Agreed. However, I'd still prefer to only call
> SetMemorySpaceAttributes() if needed

Hmm, couldn’t there be some optimisation within the function itself? To my 
understanding, the memory / GCD maps should have the permission information 
without having to look them up with a page table walk, no? Again, just talking 
high-level here, ignoring any low-level details.

> , and setting the same attributes
> on the entire image allocation at least permits us to double check
> whether the new attributes are already set on a region, and avoid the
> call if that is the case.
> 
> Perhaps we should just set EFI_MEMORY_XP on all images regardless of
> the policy - the memory should no longer be executable anyway, and
> what we currently do is remap the entire region RWX after it has
> executed, which is kind of nasty.

I’d rather have FreePages() take care of that honestly. Or do you mean as a 
workaround for tightened policies like Mu or AUDK?

> 
>> I did *not* look into the implementation code in detail, but does the new 
>> memory permission protocol impose the same constraint implementation-wise 
>> and if so, is this documented anywhere?
>> 
> 
> Not sure I follow you: which constraint is that?

Having to reset the permissions to the type’s defaults prior to freeing.

> 
>> PS: Fetched the wrong link in my last mail: 
>> https://lkml.org/lkml/2022/12/15/352
>> 
> 
> Yeah saw that. I'm hoping to get that in for v6.4 as we missed v6.3 by
> now. (I did take his patch that adds the definition of the EFI memory
> attribute protocol only, as I need it for EFI zboot)

:)

Best regards,
Marvin 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#99758): https://edk2.groups.io/g/devel/message/99758
Mute This Topic: https://groups.io/mt/96664071/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-