zeroshade commented on code in PR #1594:
URL: https://github.com/apache/arrow-go/pull/1594#discussion_r4603962531


##########
parquet/file/page_reader.go:
##########
@@ -905,6 +910,11 @@ func (p *serializedPageReader) SeekToPageWithRow(rowIdx 
int64) error {
 
                p.rowsSeen = 0
                p.pageOrd = 0
+               if p.dataOffset > p.baseOffset {
+                       p.cryptoCtx.StartDecryptWithDictionaryPage = false
+               } else if p.dictOffset > 0 {
+                       p.cryptoCtx.StartDecryptWithDictionaryPage = true
+               }

Review Comment:
   When `dictOffset == 0` but the chunk starts with a dictionary page 
(`StartDecryptWithDictionaryPage` was initially true), neither branch runs. The 
flag stays `false` from the first read, so the reseek decrypts the dictionary 
page header with the data page header AAD. To reproduce, drop the 
`p.baseOffset, p.dataOffset, p.dictOffset = 1, 1, 1` line from 
`TestReseekOnEncryptedChunkWithSharedPageOffset`:
   
   ```
   decrypting page header: decrypting encrypted payload: cipher: message 
authentication failed
   ```
   
   The fix is to restore the flag's initial value whenever the seek restarts at 
the chunk start:
   
   ```suggestion
                p.cryptoCtx.StartDecryptWithDictionaryPage = p.dataOffset <= 
p.baseOffset && p.chunkStartsWithDict
   ```
   
   Here `chunkStartsWithDict` is a new field on `serializedPageReader` that 
holds `ctx.StartDecryptWithDictionaryPage`. Set it next to `cryptoCtx = *ctx` 
in `init`, `NewPageReader` and `Reset`. This alone isn't enough, though (see 
`dictionaryAtDataOffset`).



##########
parquet/file/page_reader.go:
##########
@@ -740,16 +740,23 @@ func extractStats(dataHeader dataheader) (pageStats 
metadata.EncodedStatistics)
 
 // dictionaryAtDataOffset reports whether the page at data_page_offset may be a
 // dictionary page, which is so when the chunk records no dictionary offset or
-// the same offset as the data offset. Encrypted chunks are excluded because 
the
-// AAD of a page header depends on the page type, which is unknown before the
-// header is read.
+// the same offset as the data offset. Encrypted chunks that record no offset
+// are excluded because the AAD of a page header depends on the page type, 
which
+// is unknown before the header is read.
 func (p *serializedPageReader) dictionaryAtDataOffset() bool {
-       return p.dataOffset <= p.baseOffset && p.cryptoCtx.MetaDecryptor == nil
+       return p.dataOffset <= p.baseOffset && (p.dictOffset > 0 || 
p.cryptoCtx.MetaDecryptor == nil)

Review Comment:
   If only the flag is fixed, the `dictOffset == 0` reseek then fails with 
`parquet: column chunk cannot have more than one dictionary`. That's because 
`skipDict` is false here, so the dictionary page goes to the column reader a 
second time. If the chunk's metadata says it starts with a dictionary page, the 
dictionary AAD is known in advance, so checking for the dictionary page is safe 
even when the chunk is encrypted:
   
   ```suggestion
        return p.dataOffset <= p.baseOffset && (p.chunkStartsWithDict || 
p.cryptoCtx.MetaDecryptor == nil)
   ```
   
   With this change and the one in `SeekToPageWithRow`, all existing tests and 
this PR's tests pass, and so does the `dictOffset == 0` reseek. Chunks whose 
flag starts as false are still never checked, so 
`TestEncryptedChunkWithoutDictionaryIsNotProbed` still passes. The doc comment 
above would then say "encrypted chunks whose metadata declares no dictionary 
page" instead of "that record no offset".



##########
parquet/file/page_reader.go:
##########
@@ -740,16 +740,23 @@ func extractStats(dataHeader dataheader) (pageStats 
metadata.EncodedStatistics)
 
 // dictionaryAtDataOffset reports whether the page at data_page_offset may be a
 // dictionary page, which is so when the chunk records no dictionary offset or
-// the same offset as the data offset. Encrypted chunks are excluded because 
the
-// AAD of a page header depends on the page type, which is unknown before the
-// header is read.
+// the same offset as the data offset. Encrypted chunks that record no offset
+// are excluded because the AAD of a page header depends on the page type, 
which
+// is unknown before the header is read.
 func (p *serializedPageReader) dictionaryAtDataOffset() bool {
-       return p.dataOffset <= p.baseOffset && p.cryptoCtx.MetaDecryptor == nil
+       return p.dataOffset <= p.baseOffset && (p.dictOffset > 0 || 
p.cryptoCtx.MetaDecryptor == nil)
 }
 
 func (p *serializedPageReader) GetDictionaryPage() (*DictionaryPage, error) {
        probe := p.dictOffset == 0 && p.dictionaryAtDataOffset()
        if p.dictOffset > 0 || probe {
+               // This reads the dictionary page header wherever the stream 
is, so it needs
+               // the dictionary AAD. The flag must still describe the next 
page of p.r
+               // afterwards.

Review Comment:
   Nit: this doesn't read "wherever the stream is". It reads from offset 0 of 
`p.r.Outer()`, whatever position `p.r` is at. Maybe: "This reads the dictionary 
page from the start of the chunk, independent of p.r, so it needs the 
dictionary AAD. Restore the flag so it still describes the next page of p.r."



##########
parquet/file/page_reader_internal_test.go:
##########
@@ -247,6 +247,28 @@ func 
TestSeekToRowZeroOnEncryptedChunkStartingWithDictionaryPage(t *testing.T) {
        require.Equal(t, []int32{10, 20, 20, 10, 10, 10, 20, 10}, vals[:n])
 }
 
+// With data_page_offset equal to dictionary_page_offset a seek goes back to 
the
+// dictionary page, whose header takes the dictionary AAD again.
+func TestReseekOnEncryptedChunkWithSharedPageOffset(t *testing.T) {
+       // RLE_DICTIONARY indices 0,1,1,0,0,0,1,0 at bit width 1
+       indices := []byte{1, 3, 0x46}
+       p := encryptedChunk(t, true, 8, dictPageSpec(10, 20), dataPageSpec(8, 
format.Encoding_RLE_DICTIONARY, indices))

Review Comment:
   Please also cover the `dictOffset == 0` case, which is what `encryptedChunk` 
gives you by default. For example, run this test for both `dictOffset` values, 
or add a variant without the offsets assignment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to