Hi Jolyon, What you have just stated is interesting to me. In the last couple of years I've been making do by sending a number of different developers to conduct installations for us. Almost always the developer would return with a list of "to do" items even though the product had over 10,000 documented features. The customers were often left with a sense of fear about the product and how it worked, due to the fact that an analyst programmer is always seeking to find fault and inherantly aplify it out into something its not. Thats what an analyst programmer does.
I eventually hired a specialist installer/trainer, who goes onsite and in half the time leaves the customer so in love with the product that they call me with praise. Its a complete turnaround. Note that the developers I've sent have been highly competant Senior developers but they are always looking to solve every small item a customer raises. Sometimes the requests are just not worth the effort, and thats a business decision that Analyst Programmers are rarely capable of making. When I asked about the midas the facts were very straight forward and I don't think contradictory at all. * I have a legacy piece of software which very very rarely corrupts a local data file - fact * I do not know yet where this corruption occurs. DB Server, Drivers, In Memory, Saving of the file, Server DB. * There are a number of steps and improvements I can take for each of these areas. Regardless of the better long term solutions, we will take a defensive programming measure to protect against the loading of corrupt files. Its not true to take the standpoint that if someone asks for the solution then they should be able to provide it e.g If I ask for Word 2010 should I be capable of sitting down and writing the application suite? I appreciate the time taken for peoples opinions, but sometimes you give a guy a blog, it goes to their head :) Matt. > Oh, I should thank you for providing the bones of a blog post. :) > > http://www.deltics.co.nz/blog > > Thank you. > > > > -----Original Message----- > From: delphi-boun...@delphi.org.nz [mailto:delphi-boun...@delphi.org.nz] > On > Behalf Of Matthew Comb > Sent: Monday, 17 January 2011 22:34 > To: NZ Borland Developers Group - Delphi List > Cc: 'NZ Borland Developers Group - Delphi List' > Subject: Re: [DUG] Validating CDS files > > Hi Jolyon, > > I almost wish I hadn't asked this question now :) > > I'm well aware of the options, and setting up a webservice to retrieve a > hashed/checksumed payload set, is our preferred approach yes. This is a > legacy piece of software and doesn't adhere to our architectural patterns > and practices. > > Knowing and acknowledging this is the better solution, I asked only one > very simple question. > > Is there a known way to validate a binary CDS file. > > All other solutions and suggestions are appreciated but really not > necessary. I may have lead the thread offtrack. Sorry for that. > > Cheers, > > Matt. > > > >> I am thoroughly confused now... >> >> If you are talking about a *stream* that you theorise is being corrupted >> in >> mid-flow over the wire, then I don't see how XML really helps ensuring >> the >> integrity of that stream, or at least not how it is any more helpful >> than >> other techniques ... you can't do anything with XML until the document >> is >> complete as an incomplete document is not valid no matter how valid it >> may >> be when eventually complete. >> >> So you can't do any validation until your potentially interrupted stream >> has >> finished streaming and you have a complete file to work with. >> >> That being the case, if XML is a viable way to circumvent your apparent >> problem then why can you not simply eliminate the streaming part >> entirely >> ? >> Dump your output into a file on the server side of your communication, >> then >> transmit the entire file, thereby leveraging the integrity checks of the >> connection to ensure reliable transmission. >> >> Hashing files in this way is a trivially simple and relatively efficient >> matter using MD5, for example. >> >> >> I am not entirely convinced that you are solving the right problem here. >> >> >> The infrequency of the apparent corruption could surely potentially be >> as >> much due to an infrequent access to some data that is already corrupt on >> the >> server as it is to some sporadic wireless network corruption, no ? >> >> >> -----Original Message----- >> From: delphi-boun...@delphi.org.nz [mailto:delphi-boun...@delphi.org.nz] >> On >> Behalf Of Matthew Comb >> Sent: Monday, 17 January 2011 19:20 >> To: NZ Borland Developers Group - Delphi List >> Subject: Re: [DUG] Validating CDS files >> >> Paul, >> >> Thanks for your thoughts, I was tending towards reverse engineering the >> format, I could not see any obvious tokens at footer, and wondered if >> someone had beaten me to it. >> >> I actually prefer the simplicity of your idea of compressing/encrypting >> the xml file. Thats a tidyier solution for now until we can get to the >> hashing >> >> Not sure why we haven't thought of that already.... >> >> Cheers, >> >> Matt. >> >> >>> Matthew wrote: >>> >>>> I wasn't suggesting that wireless was changing byte >>>> structure, but if you are streaming data, and your datastream >>>> gets disconnected, then you could end up with an incomplete transfer. >>>> >>>> I'm not 100% sure that midas catches all scenarios when >>>> working off a remote data instance ? >>>> >>>> Note we use dbx4mysql + midas. >>>> >>>> Note also that I cannot rule out the drivers either and also >>>> could be the data out of the db server, its a tricky one to >>>> track down, as you basically have a black box from db -> >>>> dbxmysql + midas... >>>> >>>> What I do know is that its very rare. e.g. maybe 1 in 10,000 >>>> usages corrupts the file and has occurred in more than 1 >>>> location, so it does not appear to be station specific. >>>> >>>> Logically with these stats, I can only put that down to a >>>> flakey connection, otherwise the error rate would be more often. >>> >>> If you can't go the hash route in the short term, it sounds to me like >>> you'll need to reverse engineer the cds format to determine whether >>> there is a examinable file tail. >>> >>> I don't imagine the format will be _that_ hard to reverse engineer but >>> then I've done a fair bit of binary reverse engineering so maybe I'm >>> being a bit cavalier. If you make a number of simple and small cds >>> files >>> and compare and contrast them, you can probably work your way up in >>> determining the encoding. >>> >>> Assuming it was implemented using some form of streaming class model, >>> there may well be a common interface and then a cds outputter and an >>> xml >>> outputter. If so, there is probably going to be a fairly 1-to-1 mapping >>> between the binary format and the xml format in terms of logical >>> schema. >>> >>> These xml inspited compressed binary schemas are often all pretty >>> similar - they replace the text tags and attributes with binary tokens >>> taken from a dictionary to avoid repetition and reduce the size. >>> >>> As an side, often you're better off just zipping the xml rather than >>> implementing and debugging your own tokenized 'binary xml' engine but >>> that's a whole different argument :-) >>> >>> Anyway, you might get lucky and quickly determine there is a common >>> metadata trailer or end of data stream signature in the binary schema, >>> similar to how zip files replicate their metadata directory and place >>> it >>> at the end of the file. >>> >>> This may seem overkill for your current problem since you're aiming for >>> a short term solution, but having a cds stream viewer which can load >>> into a virtual tree view might be a useful in-house debugging tool down >>> the track. >>> >>> Cheers, >>> Paul. >>> >>> _______________________________________________ >>> NZ Borland Developers Group - Delphi mailing list >>> Post: delphi@delphi.org.nz >>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>> Subject: >>> unsubscribe >>> >> >> >> _______________________________________________ >> NZ Borland Developers Group - Delphi mailing list >> Post: delphi@delphi.org.nz >> Admin: http://delphi.org.nz/mailman/listinfo/delphi >> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: >> unsubscribe >> >> >> _______________________________________________ >> NZ Borland Developers Group - Delphi mailing list >> Post: delphi@delphi.org.nz >> Admin: http://delphi.org.nz/mailman/listinfo/delphi >> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: >> unsubscribe >> > > > _______________________________________________ > NZ Borland Developers Group - Delphi mailing list > Post: delphi@delphi.org.nz > Admin: http://delphi.org.nz/mailman/listinfo/delphi > Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: > unsubscribe > > _______________________________________________ > NZ Borland Developers Group - Delphi mailing list > Post: delphi@delphi.org.nz > Admin: http://delphi.org.nz/mailman/listinfo/delphi > Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: > unsubscribe > _______________________________________________ NZ Borland Developers Group - Delphi mailing list Post: delphi@delphi.org.nz Admin: http://delphi.org.nz/mailman/listinfo/delphi Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: unsubscribe
