zeroshade commented on code in PR #1594:
URL: https://github.com/apache/arrow-go/pull/1594#discussion_r4603962531
##########
parquet/file/page_reader.go:
##########
@@ -905,6 +910,11 @@ func (p *serializedPageReader) SeekToPageWithRow(rowIdx
int64) error {
p.rowsSeen = 0
p.pageOrd = 0
+ if p.dataOffset > p.baseOffset {
+ p.cryptoCtx.StartDecryptWithDictionaryPage = false
+ } else if p.dictOffset > 0 {
+ p.cryptoCtx.StartDecryptWithDictionaryPage = true
+ }
Review Comment:
When `dictOffset == 0` but the chunk starts with a dictionary page
(`StartDecryptWithDictionaryPage` was initially true), neither branch runs. The
flag stays `false` from the first read, so the reseek decrypts the dictionary
page header with the data page header AAD. To reproduce, drop the
`p.baseOffset, p.dataOffset, p.dictOffset = 1, 1, 1` line from
`TestReseekOnEncryptedChunkWithSharedPageOffset`:
```
decrypting page header: decrypting encrypted payload: cipher: message
authentication failed
```
The fix is to restore the flag's initial value whenever the seek restarts at
the chunk start:
```suggestion
p.cryptoCtx.StartDecryptWithDictionaryPage = p.dataOffset <=
p.baseOffset && p.chunkStartsWithDict
```
Here `chunkStartsWithDict` is a new field on `serializedPageReader` that
holds `ctx.StartDecryptWithDictionaryPage`. Set it next to `cryptoCtx = *ctx`
in `init`, `NewPageReader` and `Reset`. This alone isn't enough, though (see
`dictionaryAtDataOffset`).
##########
parquet/file/page_reader.go:
##########
@@ -740,16 +740,23 @@ func extractStats(dataHeader dataheader) (pageStats
metadata.EncodedStatistics)
// dictionaryAtDataOffset reports whether the page at data_page_offset may be a
// dictionary page, which is so when the chunk records no dictionary offset or
-// the same offset as the data offset. Encrypted chunks are excluded because
the
-// AAD of a page header depends on the page type, which is unknown before the
-// header is read.
+// the same offset as the data offset. Encrypted chunks that record no offset
+// are excluded because the AAD of a page header depends on the page type,
which
+// is unknown before the header is read.
func (p *serializedPageReader) dictionaryAtDataOffset() bool {
- return p.dataOffset <= p.baseOffset && p.cryptoCtx.MetaDecryptor == nil
+ return p.dataOffset <= p.baseOffset && (p.dictOffset > 0 ||
p.cryptoCtx.MetaDecryptor == nil)
Review Comment:
If only the flag is fixed, the `dictOffset == 0` reseek then fails with
`parquet: column chunk cannot have more than one dictionary`. That's because
`skipDict` is false here, so the dictionary page goes to the column reader a
second time. If the chunk's metadata says it starts with a dictionary page, the
dictionary AAD is known in advance, so checking for the dictionary page is safe
even when the chunk is encrypted:
```suggestion
return p.dataOffset <= p.baseOffset && (p.chunkStartsWithDict ||
p.cryptoCtx.MetaDecryptor == nil)
```
With this change and the one in `SeekToPageWithRow`, all existing tests and
this PR's tests pass, and so does the `dictOffset == 0` reseek. Chunks whose
flag starts as false are still never checked, so
`TestEncryptedChunkWithoutDictionaryIsNotProbed` still passes. The doc comment
above would then say "encrypted chunks whose metadata declares no dictionary
page" instead of "that record no offset".
##########
parquet/file/page_reader.go:
##########
@@ -740,16 +740,23 @@ func extractStats(dataHeader dataheader) (pageStats
metadata.EncodedStatistics)
// dictionaryAtDataOffset reports whether the page at data_page_offset may be a
// dictionary page, which is so when the chunk records no dictionary offset or
-// the same offset as the data offset. Encrypted chunks are excluded because
the
-// AAD of a page header depends on the page type, which is unknown before the
-// header is read.
+// the same offset as the data offset. Encrypted chunks that record no offset
+// are excluded because the AAD of a page header depends on the page type,
which
+// is unknown before the header is read.
func (p *serializedPageReader) dictionaryAtDataOffset() bool {
- return p.dataOffset <= p.baseOffset && p.cryptoCtx.MetaDecryptor == nil
+ return p.dataOffset <= p.baseOffset && (p.dictOffset > 0 ||
p.cryptoCtx.MetaDecryptor == nil)
}
func (p *serializedPageReader) GetDictionaryPage() (*DictionaryPage, error) {
probe := p.dictOffset == 0 && p.dictionaryAtDataOffset()
if p.dictOffset > 0 || probe {
+ // This reads the dictionary page header wherever the stream
is, so it needs
+ // the dictionary AAD. The flag must still describe the next
page of p.r
+ // afterwards.
Review Comment:
Nit: this doesn't read "wherever the stream is". It reads from offset 0 of
`p.r.Outer()`, whatever position `p.r` is at. Maybe: "This reads the dictionary
page from the start of the chunk, independent of p.r, so it needs the
dictionary AAD. Restore the flag so it still describes the next page of p.r."
##########
parquet/file/page_reader_internal_test.go:
##########
@@ -247,6 +247,28 @@ func
TestSeekToRowZeroOnEncryptedChunkStartingWithDictionaryPage(t *testing.T) {
require.Equal(t, []int32{10, 20, 20, 10, 10, 10, 20, 10}, vals[:n])
}
+// With data_page_offset equal to dictionary_page_offset a seek goes back to
the
+// dictionary page, whose header takes the dictionary AAD again.
+func TestReseekOnEncryptedChunkWithSharedPageOffset(t *testing.T) {
+ // RLE_DICTIONARY indices 0,1,1,0,0,0,1,0 at bit width 1
+ indices := []byte{1, 3, 0x46}
+ p := encryptedChunk(t, true, 8, dictPageSpec(10, 20), dataPageSpec(8,
format.Encoding_RLE_DICTIONARY, indices))
Review Comment:
Please also cover the `dictOffset == 0` case, which is what `encryptedChunk`
gives you by default. For example, run this test for both `dictOffset` values,
or add a variant without the offsets assignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]