Hi team,
I've read over the white paper proposing eSDTs and my initial impressions are that they are well thought out and certainly necessary. I am all in favor of this migration.
Of course, this change poses several risks to Hackystat. I have outlined a few below that I believe we can mitigate (or at least be prepared for).
1) Data reorganization - One of the examples cited in the paper was the promotion of a key-value pair called "Foo" from the "additionalInfo" field to a field of its own. I agree that this functionality is practical and will certainly be used for certain eSDTs. However, I did not understand what will happen to the key-value pair in the "additionalInfo" field. Will we remove it? That would seem like the logical thing to do, but I think we should guard against doing this at first just for the sole purpose of doing rollbacks. Data reorganization can be scary and error prone and I believe we should maintain the original structure of our data for a "trial-period" of several results in production before removing it.
2) Also, in line with Takuya's comments, I think we should have an alert sent to hackstat-dev-l whenever a new field is added to an eSDT, just as a sanity check.
3) Recycling field names: We need to protect against the case in which a new field name matches one that was previously used. Moreover, I think we should disallow this practice completely. You can imagine the chaos that would ensue if old data was populating a new field (with perhaps a different context) just because the field names matched coincidentally.
Otherwise, I think this is a great idea and I'm happy to volunteer for the Jira issue in which we delete the "className" field from the FileMetric SDT and move it to the PropertyList when it gets migrated to an eSDT! That one has been bugging me for a while!
Thanks, Mike
2)
Philip Johnson wrote:
I've just finished a white paper that outlines an approach to addressing some of our longstanding problems related to evolution in way we want to represent sensor data over time:
<http://hackydev.ics.hawaii.edu/hackyDevSite/doc/evolutionarySDT.html>
It's also available as a link from the HackyDevSite home page.
My goal is to implement this mechanism in time for the next stable release. Before I start ripping away at the guts of hackyKernel, I would appreciate your perspectives on the following:
* Do the proposed enhancements appear useful?
* Are there other enhancements we should consider?
* Are there potential problems/dangers with the proposed enhancements?
* Are there important implementation issues I should remember to consider?
* Do the proposed enhancements appear to address the problems you have encountered related to sensor data evolution? If not, please describe your situation so I can think about what else I need to incorporate into the design.
Thanks!
Cheers, Philip
p.s. Before I start on this, I am going to complete work on upgrading Hackystat to JDOM 1.0. This "official" version of JDOM is unfortunately NOT backward compatible with older versions of JDOM, and so we're going to need to update a lot of our systems simultaneously. While this is going to be a pain, the start of a release cycle is a reasonable time to inflict it on ourselves and we need to do it sooner or later.
