jbonofre commented on code in PR #18764:
URL: https://github.com/apache/iceberg/pull/18764#discussion_r4625060058
##########
core/src/main/java/org/apache/iceberg/avro/ValueWriters.java:
##########
@@ -115,7 +115,8 @@ public static ValueWriter<BigDecimal> decimal(int
precision, int scale) {
}
public static ValueWriter<Variant> variants() {
- return VariantWriter.INSTANCE;
+ // not a shared instance: the writer reuses a buffer for serialization
+ return new VariantWriter();
Review Comment:
This fixes callers that build one writer tree per file, but not callers that
share a tree across threads. `IcebergEncoder` is public, builds a single
`DataWriter` in its constructor, and keeps its output buffer and encoder in
thread-locals so that one instance can be used from several threads. Concurrent
`encode` calls for a schema with a variant column still share this writer's
buffers.
The other writers in this class solve the same problem with a thread-local
scratch buffer (`UUIDWriter`, `DecimalWriter`). Could `VariantBinaryWriter` do
the same? That covers every caller, and `variants()` can keep returning a
shared instance.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
+ ExecutorService executor = Executors.newFixedThreadPool(WRITERS);
+ try {
+ CyclicBarrier start = new CyclicBarrier(WRITERS);
+ List<Future<byte[]>> results = Lists.newArrayList();
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ int id = writer;
+ results.add(executor.submit(() -> write(id, start)));
+ }
+
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ BinaryDecoder decoder =
DecoderFactory.get().binaryDecoder(results.get(writer).get(), null);
+ ValueReader<Variant> reader = ValueReaders.variants();
+ for (int row = 0; row < VALUES; row += 1) {
+ Variant variant = reader.read(decoder, null);
+
assertThat(variant.value().asPrimitive().get()).isEqualTo(value(writer, row));
+ }
+ }
+ } finally {
+ executor.shutdownNow();
+ }
+ }
+
+ private static byte[] write(int writer, CyclicBarrier start) throws
Exception {
+ // each call builds its own writer, as each Avro file writer does
+ ValueWriter<Variant> variantWriter = ValueWriters.variants();
Review Comment:
If the buffer moves to a thread-local, this test should share one writer
between both threads. That covers the shared-tree case (for example
`IcebergEncoder`) that a writer per thread can't catch.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
+ ExecutorService executor = Executors.newFixedThreadPool(WRITERS);
+ try {
+ CyclicBarrier start = new CyclicBarrier(WRITERS);
+ List<Future<byte[]>> results = Lists.newArrayList();
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ int id = writer;
+ results.add(executor.submit(() -> write(id, start)));
+ }
+
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ BinaryDecoder decoder =
DecoderFactory.get().binaryDecoder(results.get(writer).get(), null);
+ ValueReader<Variant> reader = ValueReaders.variants();
+ for (int row = 0; row < VALUES; row += 1) {
+ Variant variant = reader.read(decoder, null);
+
assertThat(variant.value().asPrimitive().get()).isEqualTo(value(writer, row));
+ }
+ }
+ } finally {
+ executor.shutdownNow();
+ }
+ }
+
+ private static byte[] write(int writer, CyclicBarrier start) throws
Exception {
+ // each call builds its own writer, as each Avro file writer does
+ ValueWriter<Variant> variantWriter = ValueWriters.variants();
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(out,
null);
+ start.await();
Review Comment:
nit: if one task fails before reaching the barrier, the other blocks here
forever, and so does the `get()` above. Using `await(timeout, unit)` and
`get(timeout, unit)` makes the test fail instead of hang.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
Review Comment:
Nit: make the class and method package-private and drop the `test` prefix
from the method name.
##########
core/src/main/java/org/apache/iceberg/avro/ValueWriters.java:
##########
@@ -115,7 +115,8 @@ public static ValueWriter<BigDecimal> decimal(int
precision, int scale) {
}
public static ValueWriter<Variant> variants() {
- return VariantWriter.INSTANCE;
+ // not a shared instance: the writer reuses a buffer for serialization
+ return new VariantWriter();
Review Comment:
`IcebergEncoder` is public, builds one writer in its constructor, and is set
up to be used from several threads.
So concurrent `encode` calls on a schema with a variant column still share
this writer's buffers.
The comment suggest a per-thread buffer in `VariantBinaryWriter`, which is
how `UUIDWriter` and `DecimalWriter` already work.
With that, `variants()` can stay a single shared instance.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
+ ExecutorService executor = Executors.newFixedThreadPool(WRITERS);
+ try {
+ CyclicBarrier start = new CyclicBarrier(WRITERS);
+ List<Future<byte[]>> results = Lists.newArrayList();
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ int id = writer;
+ results.add(executor.submit(() -> write(id, start)));
+ }
+
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ BinaryDecoder decoder =
DecoderFactory.get().binaryDecoder(results.get(writer).get(), null);
+ ValueReader<Variant> reader = ValueReaders.variants();
+ for (int row = 0; row < VALUES; row += 1) {
+ Variant variant = reader.read(decoder, null);
+
assertThat(variant.value().asPrimitive().get()).isEqualTo(value(writer, row));
+ }
+ }
+ } finally {
+ executor.shutdownNow();
+ }
+ }
+
+ private static byte[] write(int writer, CyclicBarrier start) throws
Exception {
+ // each call builds its own writer, as each Avro file writer does
+ ValueWriter<Variant> variantWriter = ValueWriters.variants();
Review Comment:
If the buffer becomes per-thread, the test should share one writer between
both threads. A writer per thread can't catch the shared-writer case.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
+ ExecutorService executor = Executors.newFixedThreadPool(WRITERS);
+ try {
+ CyclicBarrier start = new CyclicBarrier(WRITERS);
+ List<Future<byte[]>> results = Lists.newArrayList();
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ int id = writer;
+ results.add(executor.submit(() -> write(id, start)));
+ }
+
+ for (int writer = 0; writer < WRITERS; writer += 1) {
+ BinaryDecoder decoder =
DecoderFactory.get().binaryDecoder(results.get(writer).get(), null);
+ ValueReader<Variant> reader = ValueReaders.variants();
+ for (int row = 0; row < VALUES; row += 1) {
+ Variant variant = reader.read(decoder, null);
+
assertThat(variant.value().asPrimitive().get()).isEqualTo(value(writer, row));
+ }
+ }
+ } finally {
+ executor.shutdownNow();
+ }
+ }
+
+ private static byte[] write(int writer, CyclicBarrier start) throws
Exception {
+ // each call builds its own writer, as each Avro file writer does
+ ValueWriter<Variant> variantWriter = ValueWriters.variants();
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(out,
null);
+ start.await();
Review Comment:
Nit: maybe add timeouts to `await` and `get` so the test fails instead of
handing.
##########
core/src/test/java/org/apache/iceberg/avro/TestVariantWriterConcurrency.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.avro;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.variants.Variant;
+import org.apache.iceberg.variants.VariantMetadata;
+import org.apache.iceberg.variants.Variants;
+import org.junit.jupiter.api.Test;
+
+public class TestVariantWriterConcurrency {
+ private static final int WRITERS = 2;
+ private static final int VALUES = 20_000;
+ private static final VariantMetadata METADATA = Variants.emptyMetadata();
+
+ @Test
+ public void testConcurrentWritersDoNotShareBuffers() throws Exception {
Review Comment:
nit: the class and method can be package-private, and new test methods
shouldn't use the `test` prefix, e.g. `concurrentWritersDoNotShareBuffers`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]