I am replying to this post: http://lists.motion-twin.com/pipermail/neko/2010-January/002694.html
Thanks for making the time and appreciation, my sentiment is reciprocal, and hoping I can get even more feedback and pointers. I was re-reading my post right before I read yours, and I realized that I had ignored that dynamic objects are usually (always?) internally implemented as hash tables. However, first let us note that NekoVM has an array data type, and thus arrays could be densely packed to be binary compatible with C. Additionally, it is not technically impossible to store a dynamic object internally as a packed struct, then seamlessly convert it to a hash table on first dynamic demand, and vice versa. And as I work through scenarios in my mind, it seems to me so far that such conversions would rarely occur for the static (final) object case and thus not induce significant overhead. And the gain would be the ability to interopt with maximum performance on native data structures, e.g. image data, etc.. Apparently the dynamic language need not even be aware such internal conversions are occurring. I note my prior stated goal of employing the 80/20 rule, in the sense that we sacrifice some insignificant performance reduction for the dynamic cases, in exchange for maximizing performance (where it matters most) and interoperability with the native C code. If I were trying to think of a feature that could make a VM stand out from the crowd and drive widespread interest, perhaps this idea is potentially a good candidate? This idea synergizes with NekoVM's stated goal of maximizing interoperability at the data type layer. Another important layer of interoperability should occur perhaps at the implementing languages layer above the VM layer, wherein languages should understand the object interfaces (e.g. struct in C, class/interface in OOP, etc) expressed in (header or include files of) different languages. Or perhaps it is best to agree on a universal syntax for object interfaces. Perhaps this universal format could also include a final keyword hint. I do not see how languages can interopt at the data type layer, if they do not also interopt at the object interface? That is one thing that is escaping me about the current design goal of NekoVM? Thanks for mentioning the general problem with values that are pointers (or references) to a dynamic data type, since they will be pointing to the metadata, not directly to the value for the data type as is the case with C pointers. One generalized solution (internal to the VM implementation) is that pointers could point to the values, and then the metadata is accessed by looking up the pointer in a hash table. That would of course make the dynamic languages much slower for pointers (references) to data, but is that a common use case that requires speed? Also, I am thinking that binary interoperability for this case would be needed much less frequently than for packed structs that do not contain pointers. So perhaps the conversion be done on demand internally. Agreed that the C code should be able to access the dynamic object hash tables (or request it be flattened to a packed struct), and a universal format for the dynamic object hash tables exists implicity in the VM, and this could be (already is?) accessed with standard functions in the "C FFI". And agreed that the goal is not to remove the dynamic capability. Rather the goal is to tradeoff an insignificant and seamless (unaware to the dynamic language) performance reduction for the dynamic cases, in exchange for maximizing performance and interoperability where is matters most for the C binary case. Again I re-iterate the goal is 80/20 principle applied to performance and interoperability. It seems to me that it would be amazingly cool to be manipulating and interopting with native binary data structures from dynamic languages, and vice versa from C, without sacrificing any dynamic capability. Seems I am here at the NekoVM mailing list, because as far as I know, it is the VM that is closest in design goals to this idea I have. I am also thinking that perhaps others have already explored this idea? -- Neko : One VM to run them all (http://nekovm.org)
