Mathieu, Yes, it does work. (unrelated, see my initial power outage, I need to perform a full reindex, there's only a handful of blobs visible in the UI, but I suspect the reindex will take days)
>From the startup logs here's the reference to bogus longitude: : 2018/04/25 11:56:13 index/corpus: loaded prefix "imagesize": 112525 rows, 2m15.159643544s 2018/04/25 11:56:13 index: bogus longitude in value of row "exifgps|sha1-42c3170acbd238e446bbe291683d356937799a02" = "45.54396388888889|-4.294966352e%2B09" 2018/04/25 11:56:13 index/corpus: loaded prefix "exifgps": 10283 rows, 2m15.381055594s 2018/04/25 11:56:17 index/corpus: loaded prefix "filetimes": 389038 rows, 2m19.187400693s 2018/04/25 11:56:26 index/corpus: loaded prefix "wholetofile": 389038 rows, 2m28.148759532s 2018/04/25 11:56:30 index/corpus: stats: 1639.528 MiB mem: 4458718 blobs (47.791 GiB) (3700134 schema (49571 permanode, 1232926 file (112525 image), ...) 2018/04/25 11:56:30 index/corpus: scanning CPU usage: 1m6.042858s I'll see if I can track down the original image file and test into an empty store. Adrian On Wed, 25 Apr 2018 at 11:18 Mathieu Lonjaret <[email protected]> wrote: > Adrian, we've just merged > https://camlistore-review.googlesource.com/c/camlistore/+/15627 > Does it work for you now? > > Also, if it does, and if you know where that blob comes from (which > file), could please try to upload it to a dummy server with an empty > index? As Brad said, I think we already now reject bogus GPS (and you > probably got that blob in before we did), but it would be nice to know > for sure. > Better yet, if you find that file and you don't mind sharing it with > me, I could try that out for you. > > > > On 24 April 2018 at 17:19, Brad Fitzpatrick <[email protected]> wrote: > > Hah. I think we should try to parse the value (even if it's in 1e+02 > > form), and then ignore it if it's out of range. > > > > > > On Tue, Apr 24, 2018 at 5:18 PM, Adrian Tritschler <[email protected]> > wrote: > >> Digging into it a bit further. > >> > >> Something looks unpleasant in the blob that's reported in the panic, > take a > >> look at the longitude! > >> > >> /opt/camlistore/blobs/sha1$ sudo exiftool -s > >> 42/c3/sha1-42c3170acbd238e446bbe291683d356937799a02.dat|grep GPS > >> GPSVersionID : 2.2.0.0 > >> GPSLatitudeRef : North > >> GPSLongitudeRef : West > >> GPSAltitudeRef : Above Sea Level > >> GPSLatitude : 45 deg 32' 38.27" N > >> GPSLongitude : 4294966352 deg 0' 0.00" W > >> GPSPosition : 45 deg 32' 38.27" N, 4294966352 deg 0' > >> 0.00" W > >> > >> Adrian > >> > >> > >> On Wed, 25 Apr 2018 at 03:13 Brad Fitzpatrick <[email protected]> wrote: > >>> > >>> Adrian, > >>> > >>> Like Mathieu said, the panic is fixed. And don't worry about the > >>> "blobpacked: 0 large blobs found in index, 3608 missing from index" > >>> part. That's just because we added a new thing to the index to make > >>> start-up integrity checks faster. You didn't lose anything. Let it do > >>> its recommended full repair once and you'll have the index populated > >>> for future start-ups. > >>> > >>> > >>> > >>> On Mon, Apr 23, 2018 at 11:03 PM, Adrian Tritschler > >>> <[email protected]> wrote: > >>> > Following a power outage at home, my long-running camlistored was > killed > >>> > and > >>> > has (probably) written all over its indexes. > >>> > > >>> > After I moved $HOME/.config/camlistore to perkeep and changed > reference > >>> > to > >>> > gpg keyring, starting the current perkeepd starts its self checking > and > >>> > then > >>> > panics: > >>> > > >>> > 2018/04/24 15:49:57 Starting perkeepd version 2018-04-23-2286b6442b; > Go > >>> > go1.10 (linux/amd64) > >>> > 2018/04/24 15:49:57 Starting to listen on http://0.0.0.0:80 (for > Let's > >>> > Encrypt challenges) > >>> > 2018/04/24 15:49:57 Starting to listen on https://localhost:443 > >>> > 2018/04/24 15:50:09 blobpacked: checking integrity of packed blobs > >>> > against > >>> > index... > >>> > 2018/04/24 15:56:56 blobpacked: 0 large blobs found in index, 3608 > >>> > missing > >>> > from index > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-0004d482fff38bbdff6c4263b2d70e5602957b15 > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-0017a08d36a4a146dde89cb2130f469837726f9a > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-0019a4797e3b483e34c52a6f9331873097f3b761 > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-0035597bf6ae264bd89a15943e11b8b15d2178d1 > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-004b8239d7e912838274f17388280dd1a5348744 > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-005f3c8765f0891a5ca60cd354c445bdd72f3571 > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-00746496ab7228460ec7d1e9fb244e906ed84d1b > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-007feccb1fc84966e0a162d3e648de5cb584a69d > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-0089cba2bb906a8558082fa94a5f9bce186905bb > >>> > 2018/04/24 15:56:56 sample missing large blob: > >>> > sha1-008b6ecaaddc8e62ee510af43a9d7f31879adb52 > >>> > 2018/04/24 15:56:56 Caught panic installer handlers: Done called more > >>> > than > >>> > Start > >>> > goroutine 1 [running]: > >>> > runtime/debug.Stack(0xc4200c2190, 0x2, 0xc420969a80) > >>> > /usr/local/go/src/runtime/debug/stack.go:24 +0xa7 > >>> > runtime/debug.PrintStack() > >>> > /usr/local/go/src/runtime/debug/stack.go:16 +0x22 > >>> > > perkeep.org/pkg/serverinit.(*Config).InstallHandlers.func1(0xc4207a9c80) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:548 > >>> > +0x9e > >>> > panic(0xf101e0, 0x1796ef0) > >>> > /usr/local/go/src/runtime/panic.go:505 +0x229 > >>> > > >>> > > perkeep.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc420de5050 > , > >>> > 0xc420664500, 0x11600fd, 0xe) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:309 > >>> > +0x198 > >>> > panic(0xf101e0, 0x1796ef0) > >>> > /usr/local/go/src/runtime/panic.go:505 +0x229 > >>> > > >>> > > perkeep.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc420de51d0 > , > >>> > 0xc420664500, 0x1155125, 0x4) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:309 > >>> > +0x198 > >>> > panic(0xf101e0, 0x1796ef0) > >>> > /usr/local/go/src/runtime/panic.go:505 +0x229 > >>> > perkeep.org/vendor/go4.org/syncutil.(*Gate).Done(0xc4202d8480) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/vendor/go4.org/syncutil/gate.go:39 > >>> > +0x66 > >>> > perkeep.org/vendor/go4.org/syncutil.(*Gate).Done-fm() > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/sorted/sqlkv/sqlkv.go:212 > >>> > +0x2a > >>> > perkeep.org/pkg/sorted/sqlkv.(*iter).Close(0xc4209da000, 0x11549a3, > 0x2) > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/sorted/sqlkv/sqlkv.go:279 > >>> > +0xad > >>> > > >>> > > perkeep.org/pkg/blobserver/blobpacked.(*storage).checkLargeIntegrity(0xc42067aa80 > , > >>> > 0x1, 0x1799680, 0xc420af9150) > >>> > > >>> > > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/blobserver/blobpacked/blobpacked.go:428 > >>> > +0x533 > >>> > perkeep.org/pkg/blobserver/blobpacked.newFromConfig(0x17ab240, > >>> > 0xc420664500, > >>> > 0xc420de49c0, 0x0, 0x0, 0x0, 0x0) > >>> > > >>> > > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/blobserver/blobpacked/blobpacked.go:353 > >>> > +0x96e > >>> > perkeep.org/pkg/blobserver.CreateStorage(0x1165f6b, 0xa, 0x17ab240, > >>> > 0xc420664500, 0xc420de49c0, 0x4, 0x1, 0x2, 0x1a) > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/blobserver/registry.go:111 > >>> > +0xc3 > >>> > > perkeep.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc420664500, > >>> > 0x1155125, 0x4) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:322 > >>> > +0x25c > >>> > perkeep.org/pkg/serverinit.(*handlerLoader).GetStorage(0xc420664500, > >>> > 0x1155125, 0x4, 0x2, 0x2, 0x0, 0x40) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:252 > >>> > +0x43 > >>> > perkeep.org/pkg/blobserver/replica.newFromConfig(0x17ab240, > >>> > 0xc420664500, > >>> > 0xc420de4b10, 0x7, 0xc4204e0870, 0xc422d71901, 0x0) > >>> > > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/blobserver/replica/replica.go:115 > >>> > +0x26c > >>> > perkeep.org/pkg/blobserver.CreateStorage(0x116209f, 0x7, 0x17ab240, > >>> > 0xc420664500, 0xc420de4b10, 0xe, 0x0, 0x1, 0x30) > >>> > /home/ajft/src/go/src/ > perkeep.org/pkg/blobserver/registry.go:111 > >>> > +0xc3 > >>> > > perkeep.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc420664500, > >>> > 0x11600fd, 0xe) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:322 > >>> > +0x25c > >>> > perkeep.org/pkg/serverinit.(*handlerLoader).setupAll(0xc420664500) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:232 > >>> > +0x92 > >>> > perkeep.org/pkg/serverinit.(*Config).InstallHandlers(0xc420977f80, > >>> > 0x179a5e0, 0xc422d4cfa0, 0xc420de2d20, 0x1f, 0x0, 0x0, 0x0, 0x0, 0x0, > >>> > ...) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/pkg/serverinit/serverinit.go:617 > >>> > +0xa0e > >>> > main.Main(0x0, 0x0) > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/server/perkeepd/camlistored.go:765 > >>> > +0x4b5 > >>> > main.main() > >>> > > >>> > /home/ajft/src/go/src/perkeep.org/server/perkeepd/camlistored.go:708 > >>> > +0x29 > >>> > Error parsing config: Caught panic: Done called more than Start > >>> > > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "Perkeep" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to [email protected]. > >>> > For more options, visit https://groups.google.com/d/optout. > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Perkeep" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to [email protected]. > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Perkeep" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected]. > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups "Perkeep" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
