Re: DPP: Linker issue with functions implemented in C header files
On Tuesday, 18 February 2020 at 09:20:08 UTC, Andre Pany wrote:
Hi Petar,
Hi Andre, I'm happy to help :)
thank you very much for the explanation and the code sample.
Filling the az_span anonymous member was the tricky part,
I thought it would be not possible to do so, but you showed me
the trick.
I wouldn't call it a trick, I was using standard struct literal
initialization (the very syntax that DIP1031 proposes to
deprecate).
For example:
struct Inner { int x, y; }
struct Outer { Inner inner; }
// You can initialize Outer in various ways:
// 1)
auto o1 = Outer(Inner(1, 2));
// 2)
Outer o2 = { inner: Inner(1, 2) };
// 3)
Outer o3 = { Inner(1, 2) };
// 4)
Outer o4 = { inner: { x: 1, y: 2} };
// 5)
Outer o5 = { { x: 1, y: 2} };
// 6)
Outer o6;
o6.inner.x = 1;
o6.inner.y = 1;
For POD (plain old data) struct like that, all six variants are
equivalent (of course there more possible variations).
Since there's no `private` protection modifier in C, the only
thing C library authors can do is make it inconvenient to access
struct fields (by prefixing them with underscores), but they
can't really prevent it.
For example, without this syntax, in pure C you can initialize a
span like this:
char my_string[] = "Hey";
az_span span;
span._internal.ptr = my_string;
span._internal.length = sizeof(my_string) - 1;
span._internal.capacity = sizeof(my_string) - 1;
And with almost the same syntax you can do this in D:
string my_string = "Hey";
az_span span;
span._internal.ptr = cast(ubyte*)my_string.ptr; // note: I think
this should be safe, because of [1]
span._internal.length = my_string.length;
span._internal.capacity = my_string.length;
It's just that that author wanted to prevent accidental bugs by
pushing you to use the inline helper functions or macros (which
are technically not needed).
[1]:
https://github.com/Azure/azure-sdk-for-c/blob/25f8a0228e5f250c02e389f19d88c064c93959c1/sdk/core/core/inc/az_span.h#L22
I will do it like you have proposed but had also already created
a ticket for the Azure SDK developer:
https://github.com/Azure/azure-sdk-for-c/issues/359
There should be a more convenient way to fill a az_span
structure.
To be honest, I don't think the authors will agree to change
this, as putting inline functions in headers files is a pretty
common practice in both C and C++.
There are two benefits to that:
1) Potentially better performance, because the code is easier to
inline
2) It's possible to provide header-only libraries (not the case
here), that don't require build steps.
For reference, here is my dockerfile which does the DPP call
and linking:
Cool, I'll check it later!
``` dockerfile
FROM dlang2/ldc-ubuntu:1.20.0 as ldc
RUN apt-get install -y git libssl-dev uuid-dev
libcurl4-openssl-dev curl
RUN curl -OL
https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \
&& mkdir /opt/cmake \
&& sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake
--skip-license \
&& ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake
RUN git clone https://github.com/Azure/azure-sdk-for-c.git \
&& cd azure-sdk-for-c \
&& git submodule update --init --recursive
RUN cd azure-sdk-for-c \
&& mkdir build \
&& cd build \
&& cmake ../ \
&& make
RUN apt-get install -y clang-9 libclang-9-dev
RUN ln -s /usr/bin/clang-9 /usr/bin/clang
COPY az_storage_blobs.dpp /tmp/
RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help
RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp --
/tmp/az_storage_blobs.dpp \
--include-path /azure-sdk-for-c/sdk/core/core/inc \
--include-path /azure-sdk-for-c/sdk/core/core/internal \
--include-path /azure-sdk-for-c/sdk/storage/blobs/inc \
--include-path
/azure-sdk-for-c/sdk/transport_policies/curl/inc \
--preprocess-only
ADD blobs_client_example.d /tmp/blobs_client_example.d
RUN ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \
/azure-sdk-for-c/build/sdk/core/core/libaz_core.a \
/azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \
/azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a
\
-of=/tmp/app
```
Kind regards
André
Cheers,
Petar
Re: DPP: Linker issue with functions implemented in C header files
On Tuesday, 18 February 2020 at 08:32:47 UTC, Petar Kirov
[ZombineDev] wrote:
On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
Hi,
I try to get wrap the "Azure SDK for C" using DPP and have
following issue.
Functions, which are actually implemented in C header files
will cause
linker errors:
https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
Example:
AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr,
int32_t length, int32_t capacity) {
return (az_span){ ._internal = { .ptr = ptr, .length =
length, .capacity = capacity, }, };
}
Error message:
/tmp/app.o:az_storage_blobs.d:function
_D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
I do not know C well, is this the expected behavior and should
I ask the Azure SDK developers to not implement functions
within C header files?
Kind regards
André
I think the problem is that you haven't actually linked in the
Azure SDK C library.
Dpp translates the header declarations from C to D, but the
actual definitions (function bodies) are not part of the
process. The executable code for the function definitions
should be inside either a static or dynamic library provided by
the SDK.
From the the project's readme file, it looks like they're using
CMake as the build system generator (afterwards both make and
ninja should be valid choices for building):
mkdir build
cd build
cmake ../
make
In cases like this, it's best to checkout the CMakeLists.txt
files of the individual sub project, like this one:
https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt
As you can see, there are several outputs of the build process,
among which:
- add_library(az_core ...)
This defines a library named az_core which can produce either a
static (.a on Linux, .lib on Windows) or dynamic library file
(.so on Linux, .dll on Windows). (If no configuration is
specified, I think it's static by default).
So the final file name would be libaz_core.{a,so} on Linux.
For the .c files to be built, a list of include directories
must be specified, where the various .h would located
(containing function and type declarations). This done like so:
target_include_directories (az_core PUBLIC ...)
The 'PUBLIC' argument to the target_include_directories
specifies that if you want to use the library, you need to use
the same include directories, as those needed for building it.
- add_executable(az_core_test ..)
This defines an executable build output, which looks is only
used for testing, so it's not interesting to us, except that it
can serve as an example app using the az_core library.
---
So in summary, if you want to use the az_core library, you need
to:
1. Build it
2. Run Dpp like so:
d++ \
--include-path target_include_directories>
You will need to repeat the same steps for any other part of
the Azure C SDK.
TL;DR
After I went through all those steps I got a similar linker
error for az_http_response_init. After looking where is the
actual function definition, it turned out that it's not defined
in a .c file, but it is an inline function part of a header
file.
Searching for az_span_init revealed the same (I could have
saved myself some time by reading your message more carefully
:D).
So, to answer your original question, the problem is that dpp
translates only declarations, not function definitions (such as
inline functions like that).
For now, your best course of action is to translate all inline
function definition by hand. Since in C inline functions are
mostly short and simple functions (a better alternative to
macros), hopefully that won't be too much work.
Also, looking at macros like AZ_SPAN_FROM_STR, there's really
very little chance that they could be correctly translated
automatically. As the things they do are likely not valid even
in @system D code (without additional casts), so it's better to
write your own D functions by hand anyway.
Here's what I tried:
test.dpp:
#include
#include
import std.stdio;
void main()
{
char[] resp =
"HTTP/1.2 404 We removed the\tpage!\r\n" ~
"\r\n" ~
"But there is somebody. :-)".dup;
az_span response_span =
{{
ptr: cast(ubyte*)resp.ptr,
length: cast(int)resp.length,
capacity: cast(int)resp.length
}};
az_http_response response;
az_result result = az_http_response_init(
, response_span);
writeln(result);
}
d++ --compiler ldmd2 --include-path ./inc test.dpp
./build/libaz_core.a
Hi Petar,
thank you very much for the explanation and the code sample.
Filling the az_span anonymous member was the tricky part,
I thought it would be not possible to do so, but you showed me
the trick.
I will do it like you have proposed but had also already created
a ticket for the Azure SDK developer:
Re: DPP: Linker issue with functions implemented in C header files
On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
Hi,
I try to get wrap the "Azure SDK for C" using DPP and have
following issue.
Functions, which are actually implemented in C header files
will cause
linker errors:
https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
Example:
AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr,
int32_t length, int32_t capacity) {
return (az_span){ ._internal = { .ptr = ptr, .length =
length, .capacity = capacity, }, };
}
Error message:
/tmp/app.o:az_storage_blobs.d:function
_D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
I do not know C well, is this the expected behavior and should
I ask the Azure SDK developers to not implement functions
within C header files?
Kind regards
André
I think the problem is that you haven't actually linked in the
Azure SDK C library.
Dpp translates the header declarations from C to D, but the
actual definitions (function bodies) are not part of the process.
The executable code for the function definitions should be inside
either a static or dynamic library provided by the SDK.
From the the project's readme file, it looks like they're using
CMake as the build system generator (afterwards both make and
ninja should be valid choices for building):
mkdir build
cd build
cmake ../
make
In cases like this, it's best to checkout the CMakeLists.txt
files of the individual sub project, like this one:
https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt
As you can see, there are several outputs of the build process,
among which:
- add_library(az_core ...)
This defines a library named az_core which can produce either a
static (.a on Linux, .lib on Windows) or dynamic library file
(.so on Linux, .dll on Windows). (If no configuration is
specified, I think it's static by default).
So the final file name would be libaz_core.{a,so} on Linux.
For the .c files to be built, a list of include directories must
be specified, where the various .h would located (containing
function and type declarations). This done like so:
target_include_directories (az_core PUBLIC ...)
The 'PUBLIC' argument to the target_include_directories specifies
that if you want to use the library, you need to use the same
include directories, as those needed for building it.
- add_executable(az_core_test ..)
This defines an executable build output, which looks is only used
for testing, so it's not interesting to us, except that it can
serve as an example app using the az_core library.
---
So in summary, if you want to use the az_core library, you need
to:
1. Build it
2. Run Dpp like so:
d++ \
--include-path target_include_directories>
You will need to repeat the same steps for any other part of the
Azure C SDK.
TL;DR
After I went through all those steps I got a similar linker error
for az_http_response_init. After looking where is the actual
function definition, it turned out that it's not defined in a .c
file, but it is an inline function part of a header file.
Searching for az_span_init revealed the same (I could have saved
myself some time by reading your message more carefully :D).
So, to answer your original question, the problem is that dpp
translates only declarations, not function definitions (such as
inline functions like that).
For now, your best course of action is to translate all inline
function definition by hand. Since in C inline functions are
mostly short and simple functions (a better alternative to
macros), hopefully that won't be too much work.
Also, looking at macros like AZ_SPAN_FROM_STR, there's really
very little chance that they could be correctly translated
automatically. As the things they do are likely not valid even in
@system D code (without additional casts), so it's better to
write your own D functions by hand anyway.
Here's what I tried:
test.dpp:
#include
#include
import std.stdio;
void main()
{
char[] resp =
"HTTP/1.2 404 We removed the\tpage!\r\n" ~
"\r\n" ~
"But there is somebody. :-)".dup;
az_span response_span =
{{
ptr: cast(ubyte*)resp.ptr,
length: cast(int)resp.length,
capacity: cast(int)resp.length
}};
az_http_response response;
az_result result = az_http_response_init(
, response_span);
writeln(result);
}
d++ --compiler ldmd2 --include-path ./inc test.dpp
./build/libaz_core.a
DPP: Linker issue with functions implemented in C header files
Hi,
I try to get wrap the "Azure SDK for C" using DPP and have
following issue.
Functions, which are actually implemented in C header files will
cause
linker errors:
https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
Example:
AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr,
int32_t length, int32_t capacity) {
return (az_span){ ._internal = { .ptr = ptr, .length = length,
.capacity = capacity, }, };
}
Error message:
/tmp/app.o:az_storage_blobs.d:function
_D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
I do not know C well, is this the expected behavior and should I
ask the Azure SDK developers to not implement functions within C
header files?
Kind regards
André
