On Mon, 28 Jan 2013, Ron Pinkas wrote: Hi,
Sorry for late respone. > >Please only remember that I was not alone in Harbour modifications in > >last years. Viktor made really great job in general code and used types > >cleanup. Mindaugas added important extensions to compiler and HVM/RTL > >code, Pritapl is extensively working on set of addons like HBIDE, HBQT, > >HBXBPP. Also many other people worked on Harbour. > Thanks for your modesty, but I am convinced that none of this would > ever take place, unless you first revived Harbour, single handedly, > as you did, and even after, those IMO, are minor, and superficial > contributions, compared to your work. :-) Thanks for nice words but it's not really true. I think that people using HBMK2 found Viktor modifications as the most important ones because they allow him to easy move to Harbour and without any doubts such basic functionality is the most important for them so for sure it was extremely important for the whole project. Anyhow I'm most interested in the future of both projects so lets leave this discussion. I feel that I can hurt someone continuing it, i.e it was not my intention to discard Andi work. I only wanted to point that some not working ports may create false imaginations about real functionality of original code and competitions of its authors. I hope that Andi seriously rethink his decision because I'm finding him as the very important person in the future modifications. > >In Harobur there is text filewhere I tried to discribe main differences > >between both projects: > >http://http://harbour-project.svn.sourceforge.net/viewvc/harbour-project/trunk/harbour/doc/xhb-diff.txt?view=log > >This may show you what should be done. I'll try to update this file > >and add some information about other differences. > >As you can seen not too much. I think that in less then month all > >modifications can be finished. > Actually this is not shocking for me, and I was debating it my self. > I am not sure what would be easier, as it mostly depends on the > DRIVE of the specific individual[s] interested in caring this task. > Sadly, I am not in a position to volunteer for this Job, for many > personal reasons, so I can not suggest what would be easier. I trust > that the person willing to take on this Job, will have his > preference. The most important differences between both project are well known. Anyhow it's also something that it was not very popular in news list but it's very important for general language description in the future: small incompatibilities to Clipper and anomalies in implementation (also in Clipper) which where cleaned or at least documented in Harbour. We do not have to follow exactly some decisions anyhow it's much easier to operate on code where some problems are resolved even to add again some extensions because programmer can see some interactions which were unknown for him in initial implementation. It also allows to eliminate duplicated code or code which is to danger and may break HVM code. Good example is OOP implementation. Many of __cls*() and __obj*() functions are repeated or very similar and I can onlu guess that no one exactly knows what each of them and what are the differences without careful xHarbour core code analyzing. Here the situation in Harbour is also bad. I left many functions just for backward compatibility but without any doubts some of them should be removed and rest replaced by new small set of functions with precisely defined actions. We should also agree some extensions so they can be implemented in all places. If we do not understand something in current implementation then we should remove it - I'm serious there are things which never worked or stop to work correctly long time ago and now it's not possible to guess why we have them and what they should exactly do. In last week I tried to document xHarbour HVM OOP functions and compare the to Harbour. Also with other functionality and it's really gray area. I.e. why xHarbour has PUBLISHED scope. Looks that it's the same as EXPORT + persistent flag or at least it duplicates such functionality but with separate set of functions (also duplicated). For serialization code new duplicated functions were added which tries to merge both functionality. To make it more complicated we have yet another serialization method by HBPRESISTENT class which is merged with other serializations with many side effects, etc. > >The windows only extensions and windows API wrappers have WIN_ and > >WAPI_ prefixes in Harbour and are part of HBWIN library. I think it's > >good solution because it well separates non portable MS-Windows only > >code what helps users to create portable programs and isolate local > >to system extensions. > Agree, except, I would in general prefer resolution by means of > namespace support. IMO, NameSpaces are critical feature of any > modern compiler, and long ago we should have implemented and > standardized namespace usage to have code like this: > USING NAMESPACE Clipper > To force usage of STRICT Clipper compatible RTL Functions, vs. > USING NAMESPACE Harbour > To prefer EXTENDED Clipper RTL. or of course manual overrriding: > Clipper.SubStr(...) vs. Harbour.SubStr(), etc. > As well as: > USING NAMESPACE Windows > I hope you agree that it is more flexible, and elegant, then using prefixes. I agree that namespace is very important. Anyhow it is much more wider problem which has many interactions with compiler and with HVM at runtime when dynamic libraries are loaded and unloaded during application life. In case of [x]Harbour it also interacts with macrocompiler so we need dynamic code execution context. Such context should resolve also other things like limited functionality (sandboxes) for foreign code. Finally it should also include support for named parameters with default values what seems to be very important in current days, i.e. I have dynamic JSON RPC (1.0 and 2.0) server, client and peer connection library for Harbour so I can exchange data with most of other languages but without named parameters in core Harbour code it has reduce functionality to position parameters or dedicated user functions which can decode default parameter values from partial parameters in hash array. A lot of things have to be changed to fully implement everything and I'm not sure it can be done in reasonable time by volunteers. > >The namespace support is sth what you will have to make yourselves. > >I'm not familiar with this code. > It has been many years since I wrote it, and I know it is not 100% > elegant, due to reliance on PP code, and having to use multiple > compilation when cyclic reliance between 2 sources exists, but I > believe it to be very complete and stable, and it offers great > flexibility and full namespace support, to both C code, and PRG > code, as well as Macro executed code. I would strongly appreciate > your review \xharbour\doc\namespaces.txt for the full set of > functionality it offers, in terms of a Namespace MODEL, then we > could discuss the code, which I am really much less interested in. OK, I'll try to look at it in spare time. > >In Harbour compiler can be used at runtime. It means that > >program which works like xBaseScript can be implemented in just few > >lines and .prg files are executed with full speed just like > >after -gc2 compilation. > Interesting - what about the copyright exception which was NOT given > to the Compiler sources? Still exists so such programs have to be released on GPL license. In fact in Harbour the folowing files do not contain Harobur excption: cmdcheck.c genc.c genhrb.c harbour.y hbfunchk.c hbgenerr.c hbident.c hbmain.c A lof of code inside is written by me but not all so I still have on my TODO list to rewrite them. It can be done quite easy starting from macrocompiler grammar rules and creating new flow control for compiler mode. It should resolve many bad things which are forced by current grammar rules defined in harbour.y Anyhow it's not critical for me and I do not find anything wrong that HBRUN in Harbour is on GPL license. > >I even create hbscript which can be used as active script in > >MS-Windows. I haven't committed it yet. It covers similar functionality > >from xHarbour.com - I tested it with xHarbour examples and they > >worked well. > > Very good :-) I'll try to commit it soon so you can see it in action. It was written by me and Mindaugas. It's pure C code though it defines C++ objects for COM/OLE interface. If you agree then I would like to attach one of your html example file from xBaseScript just for comparison. I'm not MS-Windows programmer so this code to be alive needs support from Windows users. > >The real problem can appear at runtime and is caused by binary > >compatibility with older code. > >HB_SERIALIZE() gives incompatible results. It means that > >HB_DESERIALIZE() from Harbour cannot decode data encoded by > >current xHarbour HB_SERIALIZE(). > >HB_SERIALIZE() is used by SQLRD so this problem is not such trivial > >because it may block migrating to new xHarbour version. > >Probably this can be resolved adding support for signatures used > >by xHarbour in serialization code. I haven't though about it > >before. I'll look at it closer and if it's possible I'll commit > >such patch to Harbour. > Saw you already committed. :-) Yes and I added also support for decoding data with cross refrences, objects and HBPERSITENT objects. I also recognizes correctly serialized codeblocks skipping their body (decoded as NIL). It means that now everything except codeblocks serialized directly or indirectly by xHarbour version of HB_SERIALIZE() is correctly decoded by Harbour. When I was checking xHarbour serializtion code I've found some problems which definitely have to be fixed. I'll commit critical fixes soon. > >hb_valToExp() gives different results then ValToPrg() but in fact > >Harbour creates real expressions which can be macrocompiled so it's > >rather like a fix for current xHarbour behavior. > Mmm, are you not aware of xHarbour's ValToPrgExp()? I have to missed it or I exploited some problems with it in the past. When I touched HB_SERIALIZE() I also checked other serialization code with some OOP functionality in Harbour and xHarbour. As result I created new general functions __obj{Set,Get,Restore}IVars() which can be used as general serialization function regardless of object definition. They can serialize whole object instance area also overloaded by descendant classes which needs super casting or have nonvirtual hidden messages. These functions can be used to replace all other __obj*() and __cls*() functions which tries to operate on object instance variables. I also updated hb_valToExp() to use them so now the serialization is more similar to xHarbour though it does not use PRIVATE variables so deserialization is reentrant safe. In the future I will want to document how programmer can control which object items should be serialized automatically. I plan to add support for VAR ... NOSAVE like in xBase++ to explicitly disable serialization for given variables. For backward compatibility if objects has variables with PERSISTENT attribute then only this variables will be saved. It should cover HBPERSITENT class functionality. Pointer items and codeblock will be eliminated. Exported persitent variables will give functional replacement for published scope. Looking at different serialization code implemented in xHarbour I tried to document some most important things I found. It should help in creating one centralized and well documented serialization code: 1. HB_SERIALIZE() - it does not serialize timestamp values "T" - it wrongly serialize LONGLONG values and then cannot desrialize data containing LONGLONG values - it does not serialize pointer items "P" - for objects it uses: HBPersistent:SaveToText() or __ClsGetPropertiesAndValues( oObj ) - stores CODEBLOCKs as "B" + HB_SERIALIZE( HB_SaveBlock( <bCode> ) ) 2. ValToPrg() / ValToPrgExp() - it does not correctly serialize timestamp values "T" - potential problems with strings containig chr(0) or "\" characters StringToLiteral() does not work correctly. - it does not serialize memo strings "M" - it converts pointer items to numbers "P"->"N" - no support for cross and cyclic references in hashes - deserialization by macrocompiler is not reentrant safe so if execution of macrocompiled data activates some code which may internally use deserialization of other ValToPrg() values then it breaks upper level data because it operates on common private variable. It is not unusual situation, i.e. it cannot be used with code where objects retrieve their initial state from serialized data inside constructors. - for objects it uses: __objGetValueDiff( oObj ) effectively it works like __ClsGetPropertiesAndValues( oObj ) in HBPersistent():SaveToText() but saves also EXPORTED variables and does not saves private and hidden vars declared with PERSISTENT attribute. Unfortunately it wrongly saves CLASSVARs with some unexpected values extracted from object instance area - this is serious bug because it badly interacts with deserialization code. 3. HBPersistent():SaveToText() - it does not correctly serialize hashes (generates broken output) because it wrongly uses ValToPrg() instead of ValToPrgExp() for types which are not directly supported. - no support for cross and cyclic references - to serialize objects which are HBPersistent class descendant (so also Self) it uses: __ClsGetPropertiesAndValues( oObj ) but reduces the properties list to the ones which have different values different then in object created by: __ClsInst( Self:ClassH ) - other objects in instance variables are not serialized. 4. Functions to clone complex items at runtime. They are not strictly serialization functions but share some similar problems like cross references and creating next object instances so they have to be synced with serialization code. Now xHarbour has: AClone(), HClone(), __objClone() - HClone() in xHarbour does not clone anything, it's simple HCOPY() so it has to be fixed and bound with AClone() code so they cab use common references list. - AClone() - clones only arrays but doesn't clone hashes and does not make any deeper check for hash contents so items like: a := { { "V" => NIL } } a[1,"V"] := a are not correctly cloned. It means that both functions have to use common internal clone engine like in Harbour. So far in xHarbour only HB_SERIALIZE() tries to follow internal references in arrays, object and hashes. - __objClone() - it does not clone nested objects - only instance area is cloned. In xHarbour there is support for class method declared with CONSTRUCTOR attribute. Such methods are executed with HB_OO_MCLSCTOR_INSTANCE parameter when new object is created and with HB_OO_MCLSCTOR_CLONE parameter inside __objClone() function. It means that theoretically it's possible to implement some custom initialization for cloned objects and deeper cloning for object items. Unfortunately it's not bound with internal list of cross references so user cannot implement custom deep cloning for objects with internal cross references. Also AClone() is not bound with it and ignores object with defined clone/copy/init constructor. This have to be resolved yet also in Harbour. We should agree how to define copy/init constructors or which messages send to new objects when they are cloned or deserialized - now deserialization code completely ignores copy constructors, in opposite Xbase++ sends :notifyLoaded() message to deserialized objects so programmer can overload it and make some necessary initialization. We should also add support for nested cloning so user can access cloned item list and redirect it to common clone engine or at least define that objects support deep cloning so we can use standard array cloning method for such objects. I also checked some helper functions used in serialization/clone operations and here are some notes: - HB_SaveBlock( <bCode> ) does not serialize detached locals so it's not necessary to detect cyclic references between them and other data. Anyhow general codeblock serialization may support it and in such case it's necessary to bind it with common list of references. - StringToLiteral() does not work correctly, use HB_STRTOEXP() instead which gives correct results also for binary strings and it's much faster. BTW I'm the author of this function not David ;-) - __objGetValueDiff( oObj ) It uses __clsGetIVarNamesAndValues( Self, HB_OO_CLSTP_EXPORTED + ; HB_OO_CLSTP_PUBLISHED ) but reduces the properties list to the ones different then in __clsGetIVarNamesAndValues( __ClsInst( Self:ClassH ), ; HB_OO_CLSTP_EXPORTED + ; HB_OO_CLSTP_PUBLISHED ) This reduction (just like in HBPersistent():SaveToText()) does not work correctly for complex items because they are cloned during object initialization and deep comparison is not implemented so it never reduce arrays used in INIT clause in class definition. - __ClsGetPropertiesAndValues( oObj ) It looks for HB_OO_CLSTP_PUBLISHED or PERSITENT messages which have non zero uiData index. It means that it gives wrong results for CLASS VAR declared as PUBLISHED and METHOD/ACCESS/ASSIGN ... INLINE declared as PERSISTENT or PROPERTY because they use uiData internally as index to diffeent structures, this is serious bug. - __clsGetIVarNamesAndValues( oObj, nScope ) It looks for variables with given scope (or all if scope is 0) checking message function hb___msgGet*Data(). It gives wrong results for CLASS VAR this is serious bug, fortunately ignores shared vars due to 0 in uiData PMETHOD member. I also found that either Harbour and xHarbour cannot serialize correctly supercast objects. In fact the results are unpredictable because it creates two objects of given class but the first one has only one element instance are and in this element it has real object. Definitely it has to be fixed in both compilers. I already added it to my TOFIX list. Enough, lot about serialization code but binary compatibility with existing data stored somewhere is usually the most important thing which can block future development and extensions. This is the reason why I never liked to add extensions to RDD if they are not precisely defined and can create anomalies (i.e. SIx3 encryption in memos). RDDs is yet another thing anyhow here the differences are much smaller. In general RDD is subject for separate discussion but here situation is much simpler. xHarbour has few copies of DBFCDX called BMDBFCDX, DBFMDX, REDBFCDX, REDBFFPT, DBFNET, etc. All such RDDs can be safely removed because they do not give any new functionality in comparison to pure DBFCDX in Harbour core code and BM* RDDs in contrib. There are some differences in DBF headers but as I can see most of them you have to remove when people begins to report serious compatibility problems with Clipper and other xbase compatible languages. > >I believe that you can quite fast catch all such differences and > >agree what to do with them. > Yes, tough I suspect there are many more hidden, such as extension > to Clipper RTL functions, that in Harbour were either not > implemented, or implemented as new HB_<SomeFunc>(). Yes but most of them are documented in XHB library so it should be easy to update the code. Enough. Now I really have to return to my own work. best regards, Przemek ------------------------------------------------------------------------------ Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb _______________________________________________ xHarbour-developers mailing list xHarbour-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xharbour-developers
