If I were to provide a patch, it would have to be to both pyfuse3 and s3ql. The optimal solution based on analysis so far seems to be:
- pyfuse returns only integer values for dates, the values are nano-seconds since epoch. - s3ql stores these values in the database as two integer values (_sec and _ns), derived from the pyfuse3 value using integer division and modulus. - s3ql retrieves these values and reconstructs them. - In-database date comparisons need to be modified to compare the two values appropriately. This would fix the problem I think, but produces potential issues for anything that reads the s3ql database directly, and anything that relies on pyfuse _ns methods returning dates as a real-value number of seconds. Over to you...I'll keep quiet for a while... On Tuesday, November 24, 2020 at 1:52:50 PM UTC+11 Grunthos wrote: > OK...I know I'm talking to myself here, but it's worth documenting the > thoughts in case they are useful... > > *If *pyfuse returned a valid representation of nano-seconds since 1970 > (say, for example, a python3 INT value -- which is unbounded), then S3QL > would need to store them differently, since SQLite will do nothing more > than 64 bit numeric values. There are three broad possibilities, I think: > > - store as strings. This would be a resounding 'no' since comparisons > would not work as expected, and it would be deeply inefficient. > - store as blobs: maybe. Research required to (a) ensure sqlite > compares blobs with correct endianness and (b) has committed to continue > to > do so. Both seem likely, but conversion to/from might be painful. > - store as two values (eg. seconds and nanoseconds), retrieve and > reassemble as a pythin INT. > > > On Tuesday, November 24, 2020 at 1:31:21 PM UTC+11 Grunthos wrote: > >> This of course potentially impacts S3QL **if** it stores FS dates as >> numeric values...since SQLite suffers the same limitation. >> >> On Tuesday, November 24, 2020 at 1:27:42 PM UTC+11 Grunthos wrote: >> >>> I've added a bug to pyfuse3; the ability for me to consider a patch >>> depends in very large part on what level of API compatibility you want to >>> break! >>> >>> Since 64 bits is insufficient to represent a nano-second timestamp, >>> something has to become incompatible. I think the choice is down to: >>> >>> - add numpy (or similar) to the pyfuse dependencies >>> - remove the *_ns properties (or at least make them return a >>> struct/object instead) >>> >>> On Monday, November 23, 2020 at 10:42:47 PM UTC+11 Grunthos wrote: >>> >>>> Further, if my analysis is correct, then it suggests anything that uses >>>> the pyfuse composite (double/float) field may have breakage. >>>> >>>> On Monday, November 23, 2020 at 10:30:52 PM UTC+11 Grunthos wrote: >>>> >>>>> *definitely* looks like a rounding problem with using a >>>>> double-precision value to represent seconds + ns since 1970. For a >>>>> standard >>>>> double, only 6 decimal digits work...for ns, you need 9 places. >>>>> >>>>> Try this in python: >>>>> >>>>> >>> 1577880000.9999999 >>>>> >>>>> <- this is the source of the problem. >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/6f7e3584-9afa-48be-81c0-5cd41a006c24n%40googlegroups.com.
