The patch I have submitted to pyfuse3 seems to fix the problem on my test machine...
On Tuesday, November 24, 2020 at 2:22:28 PM UTC+11 Grunthos wrote: > If I were to provide a patch, it would have to be to both pyfuse3 and > s3ql. The optimal solution based on analysis so far seems to be: > > - pyfuse returns only integer values for dates, the values are > nano-seconds since epoch. > - s3ql stores these values in the database as two integer values (_sec > and _ns), derived from the pyfuse3 value using integer division and > modulus. > - s3ql retrieves these values and reconstructs them. > - In-database date comparisons need to be modified to compare the two > values appropriately. > > This would fix the problem I think, but produces potential issues for > anything that reads the s3ql database directly, and anything that relies on > pyfuse _ns methods returning dates as a real-value number of seconds. > > Over to you...I'll keep quiet for a while... > > On Tuesday, November 24, 2020 at 1:52:50 PM UTC+11 Grunthos wrote: > >> OK...I know I'm talking to myself here, but it's worth documenting the >> thoughts in case they are useful... >> >> *If *pyfuse returned a valid representation of nano-seconds since 1970 >> (say, for example, a python3 INT value -- which is unbounded), then S3QL >> would need to store them differently, since SQLite will do nothing more >> than 64 bit numeric values. There are three broad possibilities, I think: >> >> - store as strings. This would be a resounding 'no' since comparisons >> would not work as expected, and it would be deeply inefficient. >> - store as blobs: maybe. Research required to (a) ensure sqlite >> compares blobs with correct endianness and (b) has committed to continue >> to >> do so. Both seem likely, but conversion to/from might be painful. >> - store as two values (eg. seconds and nanoseconds), retrieve and >> reassemble as a pythin INT. >> >> >> On Tuesday, November 24, 2020 at 1:31:21 PM UTC+11 Grunthos wrote: >> >>> This of course potentially impacts S3QL **if** it stores FS dates as >>> numeric values...since SQLite suffers the same limitation. >>> >>> On Tuesday, November 24, 2020 at 1:27:42 PM UTC+11 Grunthos wrote: >>> >>>> I've added a bug to pyfuse3; the ability for me to consider a patch >>>> depends in very large part on what level of API compatibility you want to >>>> break! >>>> >>>> Since 64 bits is insufficient to represent a nano-second timestamp, >>>> something has to become incompatible. I think the choice is down to: >>>> >>>> - add numpy (or similar) to the pyfuse dependencies >>>> - remove the *_ns properties (or at least make them return a >>>> struct/object instead) >>>> >>>> On Monday, November 23, 2020 at 10:42:47 PM UTC+11 Grunthos wrote: >>>> >>>>> Further, if my analysis is correct, then it suggests anything that >>>>> uses the pyfuse composite (double/float) field may have breakage. >>>>> >>>>> On Monday, November 23, 2020 at 10:30:52 PM UTC+11 Grunthos wrote: >>>>> >>>>>> *definitely* looks like a rounding problem with using a >>>>>> double-precision value to represent seconds + ns since 1970. For a >>>>>> standard >>>>>> double, only 6 decimal digits work...for ns, you need 9 places. >>>>>> >>>>>> Try this in python: >>>>>> >>>>>> >>> 1577880000.9999999 >>>>>> >>>>>> <- this is the source of the problem. >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/cb147495-820d-4bcd-a906-579571c428dfn%40googlegroups.com.
