Re: [Tinycc-devel] Question about in-memory compilation, target
That is nice, I wish I had thought of it. Any thoughts about my suggestion from a few days back that tcc_relocate() could be dropped from libtcc.h, and instead be automatically called on the first call to tcc_get_symbol()? On Tue, Feb 13, 2024 at 4:15 PM grischka wrote: > On 11.02.2024 11:08, draco via Tinycc-devel wrote: > > You're right, I didn't verify closely, but the ABI is still intact. > Sorry... > > To be nice I've made tcc_relocate() abort with a notice > when it's called with the former two-step method ;) > > -- gr > > > > > Am 10.02.24 um 22:12 schrieb Eric Raible: > >> > This means, that not only the public API changes, but also the > >> > libtcc.dll/so ABI, making all programs using libtcc crash without > warning. > >> > > >> > Is this intended? > >> > > >> > Michael > >> I just tried it. It looks like only programs that _don't_ use > >> TCC_RELOCATE_AUTO would be affected (at least on my x86_64 debian box). > >> Looks like the breakage would only be in a program uses manual memory > management, > >> that uses the system tcc (instead of a private version), and where the > system tcc is > >> updated w/out rebuilding the application. Anyone out there who would > be in that situation? > >> - Eric > > > > > > ___ > > Tinycc-devel mailing list > > Tinycc-devel@nongnu.org > > https://lists.nongnu.org/mailman/listinfo/tinycc-devel > > > > ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation, target
On 11.02.2024 11:08, draco via Tinycc-devel wrote: You're right, I didn't verify closely, but the ABI is still intact. Sorry... To be nice I've made tcc_relocate() abort with a notice when it's called with the former two-step method ;) -- gr Am 10.02.24 um 22:12 schrieb Eric Raible: > This means, that not only the public API changes, but also the > libtcc.dll/so ABI, making all programs using libtcc crash without warning. > > Is this intended? > > Michael I just tried it. It looks like only programs that _don't_ use TCC_RELOCATE_AUTO would be affected (at least on my x86_64 debian box). Looks like the breakage would only be in a program uses manual memory management, that uses the system tcc (instead of a private version), and where the system tcc is updated w/out rebuilding the application. Anyone out there who would be in that situation? - Eric ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation, target
You're right, I didn't verify closely, but the ABI is still intact. Sorry... Am 10.02.24 um 22:12 schrieb Eric Raible: > This means, that not only the public API changes, but also the > libtcc.dll/so ABI, making all programs using libtcc crash without warning. > > Is this intended? > > Michael I just tried it. It looks like only programs that _don't_ use TCC_RELOCATE_AUTO would be affected (at least on my x86_64 debian box). Looks like the breakage would only be in a program uses manual memory management, that uses the system tcc (instead of a private version), and where the system tcc is updated w/out rebuilding the application. Anyone out there who would be in that situation? - Eric ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation, target
I've just taken a closer look at libtcc.h. Now, I realize that this is radical, but for the sake of discussion... if we're willing to break compatibility it seems to me that just one new API: /* add option as on a comment line (multiple supported) */ LIBTCCAPI int tcc_argv_add(TCCState *s, const char *str); could replace all of: LIBTCCAPI int tcc_set_options(TCCState *s, const char *str); LIBTCCAPI void tcc_set_lib_path(TCCState *s, const char *path); LIBTCCAPI int tcc_add_include_path(TCCState *s, const char *pathname); LIBTCCAPI int tcc_add_sysinclude_path(TCCState *s, const char *pathname); LIBTCCAPI void tcc_define_symbol(TCCState *s, const char *sym, const char *value); LIBTCCAPI void tcc_undefine_symbol(TCCState *s, const char *sym); LIBTCCAPI int tcc_add_file(TCCState *s, const char *filename); Also, now that tcc_relocate() has no options, tcc_get_symbol() could just remember if it's been called and call tcc_relocate(s) on its first invocation. So this would also disappear: /* do all relocations (needed before using tcc_get_symbol()) */ LIBTCCAPI int tcc_relocate(TCCState *s1); Food for thought, perhaps. - Eric ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation, target
> This means, that not only the public API changes, but also the > libtcc.dll/so ABI, making all programs using libtcc crash without warning. > > Is this intended? > > Michael I just tried it. It looks like only programs that _don't_ use TCC_RELOCATE_AUTO would be affected (at least on my x86_64 debian box). Looks like the breakage would only be in a program uses manual memory management, that uses the system tcc (instead of a private version), and where the system tcc is updated w/out rebuilding the application. Anyone out there who would be in that situation? - Eric ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation, target
This means, that not only the public API changes, but also the libtcc.dll/so ABI, making all programs using libtcc crash without warning. Is this intended? Michael Ok, I like it to remove stuff: https://repo.or.cz/tinycc.git/blob/b671fc0594625eb5ac147ec83be6d0c1fc1a6ad5:/libtcc.h ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
On 09.02.2024 01:44, Eric Raible wrote: > Then, instead of adding a new API to support the > "run without state" > option better (as you suggest), we could just as well remove that > option entirely, and have a simpler and more "lovely" API then ... > > What do you think? > I think removing that option entirely would be fine. Others might disagree. But if we _keep_ that option, then we should have tcc_unprotect(). With respect to "tweak the state a bit", that sounds risky if we're really trying to stabilize for a release. Ok, I like it to remove stuff: https://repo.or.cz/tinycc.git/blob/b671fc0594625eb5ac147ec83be6d0c1fc1a6ad5:/libtcc.h On an unrelated note, there was a commit a while back that proposed putting more into TCCState, such that no global state at all exists. I am in favor of that proposal, which kind of implies an even bigger TCCState. Yeah, answer is always the same: - eyes would hurt from seeing "s1->" all over the place - finger would hurt for those who'd like to contribute - not to mention what if you'd try to rebase some patch that you already have See also https://repo.or.cz/tinycc.git/commitdiff/af686a796bda94dc92fc3ad140ef438dafa08950 --- grischka ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
Sean Conner wrote in <20240208102201.gb22...@brevard.conman.org>: |It was thus said that the Great grischka via Tinycc-devel once stated: |> On 07.02.2024 09:38, Eric Raible wrote: ... | Lua contains a way to set a custom realloc() function (via |lua_newstate()), thus allowing someone who is embedding Lua into an |application to use a custom allocator, or even just restrict the amount of |memory Lua can use. There is also a function in Lua to obtain the \ |allocator |function Lua is currently using (via lua_getallocf()). I think the possibility to replace an allocator is a good thing. I was frustrated that one of the first things of the libressl fork was to remove that possibility (i had found lots of problems in the MUA i took maintainership of by hooking that thing), and such. I loved the old libraries, graphics and such, which per-se supported hooking their allocators. Then again, today, so many pieces of cake install on-fork, etc, handlers, use so-called sophisticated linker techniques, that whatever you do you have a bunch of things using either the standard allocator, or their very own, non-hookable thing, that you are total at odds, anyhow. As a nice counter-example, just this/last week FreeBSD introduced the split of POSIX+ systemcalls into a libsys library, and out of the standard C library. But this is a tiny drop, and non-portable it is of course, too. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
It was thus said that the Great grischka via Tinycc-devel once stated: > On 07.02.2024 09:38, Eric Raible wrote: > > The alternative is having to know about messy system-dependent details, > > which seems very much against the spirit of the (lovely) libtcc API. > > Well, if it's "lovely" then maybe because it's still small and > fairly easily to read. > > In any case, there is always a conflict between simplicity > and any new addition, even if they may come very convenient > in some case. > > Maybe you (or other people) could tell a bit more why now that > tcc_set_realloc() > can be useful enough that tcc should have it, more exactly? > > And if so, why then do we need > tcc_get_realloc() > too? Is that just an exercise about "setters" and "getters" or > is it really useful? Lua contains a way to set a custom realloc() function (via lua_newstate()), thus allowing someone who is embedding Lua into an application to use a custom allocator, or even just restrict the amount of memory Lua can use. There is also a function in Lua to obtain the allocator function Lua is currently using (via lua_getallocf()). I checked, and back in 2012 when I created a Lua module around TCC [1] there was no way to set the TCC allocator to use the allocator from Lua. If someone went to the trouble of setting up a custom allocator for Lua, then any Lua module that might allocate memory should probably use it. I'll admit, my use case of TCC is a bit unusual, and my request to make tcc_relocate_ex() public after you made it private for some reason was met with over a month long argument about it. -spc [1] https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
On 07.02.2024 09:38, Eric Raible wrote: > The alternative is having to know about messy system-dependent details, > which seems very much against the spirit of the (lovely) libtcc API. Well, if it's "lovely" then maybe because it's still small and fairly easily to read. In any case, there is always a conflict between simplicity and any new addition, even if they may come very convenient in some case. Maybe you (or other people) could tell a bit more why now that tcc_set_realloc() can be useful enough that tcc should have it, more exactly? And if so, why then do we need tcc_get_realloc() too? Is that just an exercise about "setters" and "getters" or is it really useful? While at it, why do we need these ones tcc_get_error_func() tcc_get_error_opaque() seen that people should be able to remember what they have set, in their own code too? As to "tcc_relocate()" and "tcc_unprotect(mem)", that would probably good to have. On the other hand it's already built-in into tcc_delete(). Maybe it is not worth to keep up that option to compile into memory provided by the user at all. If we'd tweak the state data a bit to keep only the minimum required instead, for example with "tcc -run tcc.c -v" code size: - 421047 bytes state data sizes: - 1088 bytes : struct TCCState itself (on i386) - 5649 bytes : public symbols (= 160 public symbols ~= 35 bytes each) - 546 bytes : file names, include paths, defines, etc. - 7283 bytes total Which means that in this case the state data is less than 2% of the total memory usage. Then, instead of adding a new API to support the "run without state" option better (as you suggest), we could just as well remove that option entirely, and have a simpler and more "lovely" API then ... What do you think? -- grischka When using void *mem = malloc(tcc_relocate(s, NULL)); tcc_relocate(s, mem); wouldn't it make sense for tcc to export a function to safely restore the pages to R/W in a platform independent way? I haven't been able to untangle tcc_relocate() -> tcc_relocate_ex() -> set_pages_executable(), but it seems to me that since tcc sizes and then populates the 'mem' from above, it could certainly add metadata to allow a helper could undo the page protection and any other platform-specific actions that are required before free()-ing the memory. Perhaps something like tcc_unprotect(mem), which would only be allowed when one is done with the memory passed to tcc_relocate(). The alternative is having to know about messy system-dependent details, which seems very much against the spirit of the (lovely) libtcc API. Comments? ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
Nice!
Thank you... that's exactly what I needed.
Brad
On 2024-01-19 08:09, grischka via Tinycc-devel wrote:
On 15.01.2024 00:51, Brad Robinson via Tinycc-devel wrote:
Questions:
1. Is it possible to keep, use and then release the compiled code
after the initial compiler instance has been deleted
FYI below some version of libtcc_test.c to run without state.
Note the LoadDll is needed only when the exe itself was not
linked with msvcrt, i.e. to prevent it from being unloaded
in tccelf_delete.
-- gr
void *mem; int size; HANDLE dll; DWORD xxx;
if ((size = tcc_relocate(s, NULL)) < 0)
return 1;
tcc_relocate(s, mem = malloc(size));
if (!(func = tcc_get_symbol(s, "foo")))
return 1;
dll = LoadLibrary("msvcrt.dll");
tcc_delete(s);
func(32);
FreeLibrary(dll);
VirtualProtect(mem, size, PAGE_READWRITE, );
#if _WIN64
RtlDeleteFunctionTable(*(RUNTIME_FUNCTION**)mem);
#endif
free(mem);
return 0;
___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
On 15.01.2024 00:51, Brad Robinson via Tinycc-devel wrote:
Questions:
1. Is it possible to keep, use and then release the compiled code after the
initial compiler instance has been deleted
FYI below some version of libtcc_test.c to run without state.
Note the LoadDll is needed only when the exe itself was not
linked with msvcrt, i.e. to prevent it from being unloaded
in tccelf_delete.
-- gr
void *mem; int size; HANDLE dll; DWORD xxx;
if ((size = tcc_relocate(s, NULL)) < 0)
return 1;
tcc_relocate(s, mem = malloc(size));
if (!(func = tcc_get_symbol(s, "foo")))
return 1;
dll = LoadLibrary("msvcrt.dll");
tcc_delete(s);
func(32);
FreeLibrary(dll);
VirtualProtect(mem, size, PAGE_READWRITE, );
#if _WIN64
RtlDeleteFunctionTable(*(RUNTIME_FUNCTION**)mem);
#endif
free(mem);
return 0;
___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
Hi Michael, Thanks, yep that's what I found too except I think that code doesn't run if you relocate to a user supplied block of memory (instead of the tcc managed one). My problem was that tcc_run_free() wasn't run on my allocated code memory when I freed it - leaving pages that have been returned to the heap marked as read-only/executable. Happy enough to just keep the compiler instance around for now, but a little concerned what else it might be holding onto in memory (ie: other remnants from the compile process). Brad On 2024-01-17 06:14, draco via Tinycc-devel wrote: Is it possible to keep, use and then release the compiled code after the initial compiler instance has been deleted (ie: after tcc_delete). Actually I strugled with the same problem and found a partial solution: If you look up the code in libtcc.c, in function tcc_delete is the line tcc_run_free(s1); You have to outcomment this and it will not free the runtime memory; there you can also see, what else get's freed. But you can't free this later on, since s1 gets discarded, and if you dont save it somewhere its lost. Michael -- __ Aktuelle Lichtmaschinenkunst http://mir52.wordpress.com Glory be to thee Hong Kong ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Re: [Tinycc-devel] Question about in-memory compilation target
Is it possible to keep, use and then release the compiled code after the initial compiler instance has been deleted (ie: after tcc_delete). Actually I strugled with the same problem and found a partial solution: If you look up the code in libtcc.c, in function tcc_delete is the line tcc_run_free(s1); You have to outcomment this and it will not free the runtime memory; there you can also see, what else get's freed. But you can't free this later on, since s1 gets discarded, and if you dont save it somewhere its lost. Michael -- __ Aktuelle Lichtmaschinenkunst http://mir52.wordpress.com Glory be to thee Hong Kong ___ Tinycc-devel mailing list Tinycc-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/tinycc-devel
