Re: Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Ted Dunning
On Wed, Aug 14, 2019 at 5:35 AM Koen De Groote wrote: > ... > I've read that the log files are preallocated. I see them as being 65MB a > piece. > Yes. Preallocation of logs is an important performance trick. The point is that if a file doesn't change length when you write to it, then the file

Re: Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Koen De Groote
Al right, I'll try and go forward with that info. Thanks. On Wed, Aug 14, 2019 at 2:55 PM Norbert Kalmar wrote: > Read doesn't matter, there will be no txn logs for read. Only modifications > are logged (state changes to the datatree). > > As for why all txn logs are 64MB: > "preAllocSize >

Re: Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Norbert Kalmar
Read doesn't matter, there will be no txn logs for read. Only modifications are logged (state changes to the datatree). As for why all txn logs are 64MB: "preAllocSize (Java system property: zookeeper.preAllocSize) To avoid seeks ZooKeeper allocates space in the transaction log file in blocks of

Re: Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Koen De Groote
>Without a snapshot, we cannot delete the log files, as we would have no >means of recovery. txn logs applied to the snapshot gives us back the >state. Without snapshot, all txn logs needs to be "replayed" in a recovery. >And you need all the log files created since your last snapshot (in this

Re: Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Norbert Kalmar
Hi, Without a snapshot, we cannot delete the log files, as we would have no means of recovery. txn logs applied to the snapshot gives us back the state. Without snapshot, all txn logs needs to be "replayed" in a recovery. And you need all the log files created since your last snapshot (in this

Code to clean up transaction logs needs snapshots before it works?

2019-08-14 Thread Koen De Groote
Greetings all. I was debugging something an ran into this bit of code: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/PurgeTxnLog.java#L81 If I understand it correctly, it seems that this means log.x files will only get deleted if