Hi John- Your assumptions are spot on. Attached is my patch which enables us to do this currently. I’m not sure how much work it’d be, but being able to specify this on a select would be awesome!
BTW, I saw in an earlier thread that you were working on compression. Is the compression functionality only for HDF5, or will it work against normal filesystems? Thanks again! Sean On Jun 20, 2014, at 12:41 AM, K. John Wu <[email protected]> wrote: > Hi, Sean, > > Having a callback for adjusting the content of a bundle could be a > very useful thing as you have pointed out. Would you mind share your > implementation? I would try it out and see what additional changes > might make sense. > > Here are my notes on this issue right now: > > - you mentioned about where to attach the callback. My thinking is > that it might be good to associate it with a select clause. The > bundle object would retrieve the callback from the select clause. > > - the callback function you proposed seems to transform one "array" > into another "array" of the same type, e.g., time_t array to another > time_t array (with new time zone and so on). It might be good > explicitly limit the transformation to this type. It avoids creating > new colValues in a bundle object, and avoids changing the type > colValues objects. > > - under the above assumption, the interface of the callback function > might need the ibis::bundle object and could directly utilize the > functions columnType and columnArray to complete its operations. > > I could have seriously missed the mark, so it would be good to see > what you have.. > > John > > > > On 5/1/14, 7:07 PM, Sean McNamara wrote: >> Hi John- >> >> I was thinking more about that date formatting use case that I had mentioned >> earlier, where some light transformations need to be performed at run-time >> on a column that is getting grouped on. Instead of putting that logic into >> fastbit I was thinking it would be super handy if there was a way that one >> could register a callback for these situations. >> >> After examining the fastbit code a bit I think it could be accomplished the >> following way: >> >> If a callback is somehow registered, the ibis::bundles::bundles constructors >> would just need to call the callback and pass it the colList before sort() >> gets called. With the colList in hand a users callback could pick a column >> and dynamic_cast to descendents of ibis::colValues, and adjust data as >> needed. >> >> In the instance of date formatting, the callback would really just need to >> adjust the epoch time by some simple timezone offset and DST rules. >> >> I think the key idea here is that the callback just needs to be called >> before the sort and reduce happens. >> >> I have a proof of concept of this working locally, but it is kind of a hack >> as it uses a single global callback variable. I would be happy to >> contribute it, but I figured it would be wise to first run the idea by you. >> I figure you would probably have a more elegant way to do this that does not >> use a global var =) >> >> Thanks so much, >> >> Sean >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
0001-adding-transform-callback.patch
Description: 0001-adding-transform-callback.patch
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
