[Mono-docs-list] Managed/Unmanaged Code Interop Documentation

[Jonathan Pryor]

Miguel had asked me to write this up in July after a lengthy list
discussion, so here's a summary of managed/unmanaged code interop.

I'd be interested in hearing of any suggestions for improvement, and of
how to add this file to monodoc.

Thanks,
 - Jon

Title: Managed Code Interop

Everything You (Never?) Wanted to Know About Marshaling (And Were Afraid To Ask)

The Common Language Infrastructure (CLI) is designed to make it "easy" to interoperate with existing code. In principal, all you need to do is create a DllImport function declaration for the legacy code to invoke, and the runtime will handle the rest. For example:

[System.Runtime.InteropServices.DllImport ("libc.so")] private static extern int getpid ();

The above C# function declaration would invoke the POSIX getpid(2) system call on platforms that have the libc.so library (other platforms would generate a MissingMethodException). Simple. Straightforward. What could be easier?

The problem is that most existing code is far more complex. Strings will need to be passed, structures may need to be passed, memory management practices will become involved... Existing code is a complex beast, and the interop layer needs to support this complexity.

Marshaling

How does code interop work? Given a managed call site (the function call), and an unmanaged callee site (the function that's being called), each parameter in the call site is "marshaled" (converted) into an unmanaged equivalent. The marshaled data is in turn placed on the runtime stack (along with other data), and the unmanaged function is invoked.

The complexity is due to the marshaling. For simple types, such as integers and floating-point numbers, marshaling is a bitwise-copy ("blitting"), just as would be the case for unmanaged code. String types introduce additional complexity, as you need to specify the form of string conversion. The runtime stores strings as UTF-16-encoded strings, and these will likely need to be marshaled to a more appropriate form (ANSI strings, UTF-8 encoded strings, etc.). Strings get special support.

Marshaling behavior is controlled through the DllImport and MarshalAs attributes.

Strings

Strings are special. String marshaling behavior is also highly platform dependent.

String marshaling for a function call can be specified in the function declaration with the DllImport attribute, by setting the CharSet field. The default value for this field is CharSet.Auto, which implies "magic."

Some background. The Microsoft Win32 API supports two forms of strings, ANSI (localized) strings and Unicode strings. It supports these string formats by appending an "A" for Ansi string APIs and a "W" ("wide") for Unicode string APIs.

Consider this Win32 API description:

[System.Runtime.InteropServices.DllImport ("gdi32.dll", CharSet=CharSet.Auto, CallingConvention=CallingConvention.StdCall)] private static extern bool TextOut ( System.IntPtr hdc, int nXStart, int nYStart, string lpString, int cbString);

When TextOut is called, the "magic" properties of String marshaling become apparent. The runtime will look for these functions, in this order, inside the gdi32.dll library:

1. TextOutW for Unicode string marshaling
2. TextOutA for Ansi string marshaling
3. TextOut with the platform-default marshaling

The runtime will use the first function that it finds in that list. Unicode marshaling is preferred, as (ideally) the System.String can be passed as-is to the function, as long as the function doesn't modify the string parameter. Ansi marshalling will require translating the Unicode string into an 8-bit string (or DBCS, depending on the country) in the users locale. Most (all?) of the time, this WILL NOT be UTF-8, so you CAN NOT assume that CharSet.Ansi will generate UTF-8-encoded strings.

Perhaps in the future the CharSet enumeration will contain more choices, such as UnicodeLE (little-endian), UnicodeBE (big-endian), Utf7, Utf8, and other common choices. Additionally, making such a change would also likely require changing the UnmanagedType enumeration. However, these would need to go through ECMA, so it won't happen next week. (Unless some time has passed since this was originally written, in which case it may very well be next week. But don't count on it.)

More Control

Using the DllImport attribute works if you want to control all the strings in a function, but what if you need more control? You would need more control if a string is a member of a structure, or if the function uses multiple different types of strings as parameters. In these circumstances, the MarshalAsAttribute can be used, setting the Value property (which is set in the constructor) to a value from the UnmanagedType enumeration. For example:

[DllImport ("does-not-exist")] private static extern void Foo ( [MarshalAs(UnmanagedType.LPStr)] string ansiString, [MarshalAs(UnmanagedType.LPWStr)] string unicodeString, [MarshalAs(UnmanagedType.LPTStr)] string platformString);

As you can guess by reading the example, UnmanagedType.LPStr will marshal the input string into an Ansi string, UnmanagedType.LPWStr will marshal the input string into a Unicode string (effectively doing nothing), and UnmanagedType.LPTStr will convert the string to the platform's default string encoding. For all flavors of Windows NT (Windows NT 3.51 and 4.0, Windows 2000, Windows XP, Windows Server 2003) the platform default encoding is Unicode, while for all Windows 9x flavors (Windows 95, 98, ME) the platform default encoding is Ansi.

There are other UnmangedType string marshaling options, but they're primarily of interest in COM Interop (BStr, AnsiBStr, TBStr).

TODO: Does anybody know the default encoding for Unix platforms? I would guess Ansi as well...

If UnmanagedType doesn't provide enough flexibility for your string marshaling needs (for example, you're wrapping GTK+ and you need to marshal strings in UTF-8 format), look at the Custom Marshaling section.

Passing Caller-Modifiable Strings

A common C idiom is for the caller to provide the callee a buffer to fill. For exmple, consider strncpy(3):

char* strncpy (char *dest, const char *src, size_t n);

We can't use System.String for both parameters, as strings are immutable. This is OK for "src", but "dest" will be modified, and the caller should be able to see the modification.

The solution is to use a System.Text.StringBuilder, which gets special marshaling support from the runtime. This would allow strncpy(3) to be wrapped and used as:

[DllImport ("libc.so")] private static extern void strncpy (StringBuilder dest, string src, uint n); private static void UseStrncpy () { StringBuilder sb = new StringBuilder (256); strncpy (sb, "this is the source string", sb.Capacity); Console.WriteLine (sb.ToString()); }

Some things to note is that the return value of strncpy(3) was changed to "void", as there is no way to specify that the return value will be the same pointer address as the input "dest" string buffer, and thus it doesn't need to be marshalled. If "string" were used instead, an extra copy of the return value would be made. The StringBuilder is allocated with the correct amount of storage as a constructor parameter, and this amount of storage is passed to strncpy(3) to prevent buffer overflow.

TODO

What's the memory management policy for using "string" as a return value? Does the runtime expect to free it?

See Also

• MSDN: Default marshaling for Strings

Class and Structure Marshaling

Conceptually, classes and structures are marshalled to native code by:

1. The runtime allocates a chunk of unmanaged memory.
2. The managed class data is copied into the unmanaged memory.
3. The unmanaged function is invoked, passing it the unmanaged memory information instead of the managed memory information. This must be done so that if a GC occurs, the unmanaged function doesn't need to worry about it. (And yes, you need to worry about GCs, as the unmanaged function could call back into the runtime, generating a GC.)
4. The unmanaged memory is copied into managed memory.

The principal difference between class and structure marshaling is which of these conceptual steps actually occurs. :-)

Class Marshaling

Remember that classes are heap-allocated and garbage-collected in the CLI. As such, you cannot pass classes by value to unmanaged functions, only by reference:

struct UnmanagedStruct { int a, b, c; }; void WRONG (struct UnamangedStruct pass_by_value) { } void RIGHT (struct UnmanagedStruct *pass_by_reference) { }

This means that you cannot use classes to invoke unmanaged functions that expect a stack-allocated variable (such as the WRONG function, above).

There are two other issues with classes. First of all, classes by default use Auto layout. This means that the ordering of class data members is unknown, and won't be determined at runtime. The runtime can rearrange the order of members in any way it chooses, to optimize for access time or data layout space. As such, you MUST use the StructLayoutAttribute and specify a LayoutKind value of Sequential or Explicit.

Secondly, classes (again, by default) only have in-bound marshaling. That is, Step 4 (copying the unmanaged memory representation back into managed memory) is ommitted. If you need the unmanaged memory to be copied back into managed memory, you must addorn the DllImport function declaration argument with an Out attribute. You will also need to use the In attribute if you want copy-in and copy-out behavior. To summarize:

• Using [In] is equivalent to not specifying any parameter attributes, and will skip Step 4 (copying unmanaged memory into managed memory).
• Using [Out] will skip Step 2 (copying managed memory into unmanaged memory).
• Use [In, Out] to both copy managed memory to unmanaged memory before the unmanaged function call, and then copy unmanaged memory back to managed memory after the function call.

Structure Marshaling

There are two primary differences between classes and structures. First, structures do not need to be allocated on the heap; they can be stack allocated. Secondly, they use Sequential LayoutKind by default, so structure declarations do not need any additional attributes to use them with unmanaged code (assuming that the default sequential layout rules are correct for the unmanaged structure).

These differences permit structures to be passed by-value to unmanaged functions, unlike classes. Additionally, since structures are typically located on the stack (unless they're boxed or part of a class instance), if you pass a structure to an unmanaged function by-reference, the structure will be passed directly to the unmanaged function, without an intermediate unmanaged memory copy. This means that you may not need to specify the Out attribute to see changes made by unmanaged code.

Summary

It's always easier to show the code, so... Given the following unmanaged code definition:

struct UnmanagedStruct { int n; }; void PassByValue (struct UnmanagedStruct s); void PassByReference (struct UnmanagedStruct *s);

The class wrapper would be:

[StructLayout (LayoutKind.Sequential)] class ClassWrapper { public int n; /* cannot wrap function PassByValue */ /* PassByReference */ [DllImport ("mylib")] public static extern void PassByReference ([In, Out] ClassWrapper s); }

While the structure wrapper would be:

struct StructWrapper { public int n; /* PassByValue */ [DllImport ("mylib")] public static extern void PassByValue (StructWrapper s); /* PassByReference */ [DllImport ("mylib")] public static extern void PassByReference (ref StructWrapper s); }

TODO

How are return values handled? What are the differences between classes and structures?

Marshaling Class and Structure Members

Aside from the major differences between classes and structures outlined above, the members of classes and structures are marshaled identically.

The general rule of advice is this: never pass classes or structures containing members of reference type (classes) to unmanaged code. This is because unmanaged code can't do anything safely with the unmanaged reference (pointer), and the CLI runtime doesn't do a "deep marshal" (marshal members of marshaled classes, and their members, ad infinitum).

The immediate net effect of this is that you can't have string and array members of marshaled classes.

It's not quite as bad as this makes out. You can't pass strings and arrays BY DEFAULT. If you help the runtime marshaler by addorning the members with the MarshalAs attribute, the marshaler can figure out what to do and you can live life peacefully with the ever-present strings and arrays. Alternatively, string marshaling can also be set by setting the CharSet member on the StructLayout attribute for the class or structure.

Unions

A C union (in which multiple members share the same offset into a structure) can be simulated by using the FieldOffset attribute and specifying the same offset for the union members.

See Also

• Marshaling Data with Platform Invoke
• Arrays Sample

Arrays Embedded Within Structures

This might be of use. From David Jesk (http://www.chat.net/~jeske/):

This time I have some PInvoke information to share, so that when someone else runs into this issue they can see what I've done.

In my ClearSilver (www.clearsilver.net, an HTML template system) C# wrapper, I wanted to access this C-struct:

typedef struct _neo_err { int error; int err_stack; int flags; char desc[256]; const char *file; const char *func; int lineno; /* internal use only */ struct _neo_err *next; } NEOERR;

My philosophy of using unsafe struct pointers, and just accessing the struct out in unmanaged memory is great, and it's exactly what I want to do. However, handling "char dest[256]" is not straightforward.

In C# arrays are reference types. Using one makes the struct a managed type, and I can't put the array size in. The following is conceptually what I want to do, however, it's obviously invalid:

[StructLayout(LayoutKind.Sequential)] unsafe struct NEOERR { public int error; public int err_stack; public int flags; public byte[256] desc; // this is invalid, can't contain size public const byte *file; public const byte *func; public int lineno; /* internal use only */ private NEOERR *next; };

This dosn't work either:

[MarshalAs (UnmanagedType.LPStr, SizeConst=256)] public string desc;

Because in this case, I don't want to marshal the data. I just want to talk to it in place. The solution I could come up with is this:

[StructLayout(LayoutKind.Explicit)] unsafe struct NEOERR { [FieldOffset(0)] public int error; [FieldOffset(4)] public int err_stack; [FieldOffset(8)] public int flags; // public byte[256] dest; // not representable [FieldOffset(12)] public byte dest_first_char; // use this as an address?? [FieldOffset(268)] public byte *file; // const [FieldOffset(272)] public byte *func; // const [FieldOffset(276)] public int lineno; };

UGH! First, this is obviously annoying. Second, the only way I can figure to get access to "char dest[256]" is to use "char* dest = &nerr->dest_first_char;" and then just use dest as a pointer to the string. I've dug through the documentation, and I can't find any better solution.

Obviously it would be ideal if there were a way to represent a value-type array. I wonder how Managed C++ handles "char foo[256];" in a struct.

Custom Marshaling

TODO: custom marshaling...

See Also

• ICustomMarshal Interface

Avoiding Marshaling

Marshaling is no panacea, as marshaling implies copying data. Marshaling may be problematic because the data translation is a complex, time-consuming process. Alternativly, it may be problematic because it isn't possible to copy the data, as the data isn't known or is likely to change.

An example of the latter would be the GTK+ libraries. GTK+ is an object-oriented toolkit written in C. As with all object-oriented libraries, there can be an unknown number of derived classes, each of which having a different class size. Furthermore, class instances are typically accessed through pointers. As such, marshaling the entire class between managed and unmanaged memory is not an option, as a copy isn't desired, access to the same instance is.

Another example is when using "opaque" data types; that is, types through which interaction is solely through pointers, and nothing about the internals of the type is public. This describes a large portion of the Win32 API, where HANDLE is used to represent most objects.

There are two ways to handle this in C#: the "type-safe" way, which involves using pointers and the "unsafe" C# language features, and the CLS-compliant way, which uses System.IntPtr to stand in for a void pointer.

In effect, the separation between managed and unmanaged memory is made explicit. Managed memory remains typesafe, while unmanaged memory is not (since System.IntPtr is used to point into unmanaged memory, and there is no way to ensure the actual type of what the System.IntPtr refers to).

Be warned that this may not be safe, if the "unmanaged" memory is itself garbage collected. This may be the case if the unmanaged memory is handled by a different runtime system (Python, Ruby, Lisp, etc.) or a garbage collector is being used (Boehm). If the unmanaged memory is garbage collected, then the System.IntPtr won't be updated when unmanaged memory undergoes a garbage collection, resulting in memory corruption.

For example, given the unmanaged API:

typedef void* HANDLE; bool CreateItem (HANDLE *item); void DestroyItem (HANDLE item); int GetInfo (HANDLE item);

The "type-safe" C# wrapper (using "unsafe" code) is:

struct Item { [DllImport ("library")] public static unsafe extern bool CreateItem (out Item* item); [DllImport ("library")] public static unsafe extern void DestroyItem (Item* item); [DllImport ("library")] public static unsafe extern int GetInfo (Item* item); } class ExampleUsage { public static unsafe void Main () { Item* item = null; Item.CreateItem (out item); int n = Item.GetInfo (item); System.Console.WriteLine ("item count: {0}", n.ToString()); Item.DestroyItem (item); } }

This is "type-safe" in that you can't pass arbitrary memory locations to the static Item functions, you must pass a pointer to an Item structure. This isn't a strict amount of type safety, but it is likely to minimize accidental memory corruption. It's biggest problem is that it uses "unsafe" code, and thus may not be usable from other .NET languages, such as Visual Basic .NET and _javascript_.

The CLS compliant version uses System.IntPtr to refer to unmanaged memory. This is similar to what the Marshal class does to interoperate with unmanaged memory.

class Item { [DllImport ("library")] public static extern bool CreateItem (out System.IntPtr item); [DllImport ("library")] public static extern void DestroyItem (System.IntPtr item); [DllImport ("library")] public static extern int GetInfo (System.IntPtr item); } class ExampleUsage { public static unsafe void Main () { System.IntPtr item = null; Item.CreateItem (out item); int n = Item.GetInfo (item); System.Console.WriteLine ("item count: {0}", n.ToString()); Item.DestroyItem (item); } }

This is "unsafe" in that it is easier to accidently mis-use pointers. For example, if you're using two different libraries and wrapping them using System.IntPtr, it is possible to pass an object allocated from one library to a function exported by the other library, and the CLI Runtime will not catch this error, while the "unsafe" C# code would catch this error.

However, this isn't normally considered a problem, however, as most managed code shouldn't interact with P/Invoke code, but should instead interact with managed wrappers for the unmanaged code, which can provide a more natural interface to managed clients.

Meaning of "Unsafe"

I think the real problem is that "unsafe" is an overloaded term. It can refer to the use of the "unsafe" C# keyword, and it can be used as "anything that isn't safe", which, as you note, doesn't require the "unsafe" keyword.

So, "unsafe" can mean (a) C# keyword; (b) violates .NET type system (similar to (a)); (c) may be insecure (reading files from a web client); (d) capable of causing a segfault. There are likely other meanings people can dream up as well. Note that (d) doesn't imply (b), as far as .NET is concerned. The runtime could itself have a bug that generates a segfault, but this doesn't violate the type system.

IntPtr doesn't require a violation of the type system, as you can't get the address of a .NET object (unless you "pin" it, which would require the appropriate Security rights), and is thus principally useful for interacting with unmanaged code, which exists outside of the .NET type system.

Surely, this is pure semantics, but I can see the designers perspective.

Security

Well, to speak on .NET's behalf, .NET has a highly flexible security system. You can't invoke DllImported functions unless your app has the appropriate security rights -- generally, that the app is running on the local machine. If you're running it from a network share, or from a web site (similar to Java Applets), then your app will get a SecurityException.

You can get lots of security exceptions for various things, actually. Opening files can generate a security exception, for example.

System.Security.Permissions.SecurityPermission is needed with SecurityPermissionFlag.UnmanagedCode specified in order to perform a P/Invoke.

Programs can't specify this permission, though, they can only request it (or demand it, and if they can't get it, a SecurityException is thrown).

Administrators are the people who specify what permissions an application actually receives.

That's about the limits of my knowledge -- Security isn't my forte. You might find the following topics interesting.

• Requesting Permissions
• Security Syntax
• Code Access Security
• Inheritance Demands

See Also

• Unsafe Code

Thanks

A large portion of this document was generated as a result of a mono-list discussion between Jonathan Pryor and David Jeske. See: http://lists.ximian.com/archives/public/mono-list/2003-July/014886.html.