laskoviymishka commented on code in PR #2644:
URL: https://github.com/apache/iceberg-go/pull/2644#discussion_r4628784461


##########
manifest_deflate.go:
##########
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package iceberg
+
+import (
+       "bytes"
+       "compress/flate"
+       "fmt"
+       "io"
+       "math"
+
+       "github.com/twmb/avro/ocf"
+)
+
+// manifestDeflateCodec reuses the inflater and output buffer across blocks of
+// one manifest. The OCF reader makes its own copy of any bytes kept by a 
decoded
+// entry, so the output buffer can be overwritten on the next block.

Review Comment:
   Could the comment name the decoder as the component that copies? 
`slab.string`/`slab.bytes` copy out of the block unless `AliasInput` is set 
([deser.go](https://github.com/twmb/avro/blob/d4de8e1130760f0d035e690c4505ece4901332bf/deser.go#L132-L156)),
 and `ocf.Reader` drops that option 
([ocf.go](https://github.com/twmb/avro/blob/d4de8e1130760f0d035e690c4505ece4901332bf/ocf/ocf.go#L360-L370)).
 That pair is what the buffer reuse depends on.
   
   ```suggestion
   // one manifest. The avro decoder copies strings and bytes out of the block 
unless
   // avro.AliasInput is set, and ocf.Reader drops that option, so the output 
buffer
   // can be overwritten on the next block.
   ```



##########
manifest_deflate_test.go:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package iceberg
+
+import (
+       "bytes"
+       "io"
+       "strings"
+       "testing"
+
+       "github.com/stretchr/testify/require"
+)
+
+func TestManifestReaderKeepsDeflateEntriesFromEarlierBlocks(t *testing.T) {
+       schema := NewSchema(1, NestedField{ID: 1, Name: "id", Type: 
PrimitiveTypes.Int64, Required: true})
+       snapshotID := int64(1)
+       // Each path exceeds the writer's 64 KiB block threshold.

Review Comment:
   Should this assert that the manifest really has two blocks? The split 
depends on the writer's 64 KiB default 
([`defaultBlockBytes`](https://github.com/twmb/avro/blob/d4de8e1130760f0d035e690c4505ece4901332bf/ocf/ocf.go#L445)).
 If that default or our writer options change, both entries can land in one 
block, and the test would still pass while covering only one block. Counting 
`DecompressBounded` calls would rule that out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to