This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/maven/parquet-protobuf/com.google.protobuf-protobuf-java-3.19.6
in repository https://gitbox.apache.org/repos/asf/parquet-mr.git
omit 62a7415c7 Bump protobuf-java from 3.17.3 to 3.19.6 in /parquet-protobuf
add 704ef93ff PARQUET-2176: Column index/statistics truncation in
ParquetWriter (#989)
add 44dc3a4ae Performance optimization to ByteBitPackingValuesReader (#962)
add 62b774cd0 PARQUET-2142: parquet-cli without hadoop throws
java.lang.NoSuchMethodError on any parquet file access command (#990)
add d75596bea PARQUET-1711: support recursive proto schemas by limiting
recursion depth (#995)
add dd5533053 PARQUET-2196: Support LZ4_RAW codec (#1000)
add d057b39d9 PARQUET-2195: Add scan command to parquet-cli (#998)
add c8c6386f8 PARQUET-2177: Fix parquet-cli not to fail showing
descriptions (#991)
add 8206384a0 PARQUET-2198 : Updating jackson data bind version to fix
CVEs (#1005)
add 433de8df3 nested encr info (#1009)
add eb2122d4f PARQUET-2224: Publish SBOM artifacts (#1017)
add 2fa8f94ba PARQUET-2219: ParquetFileReader skips empty row group (#1018)
add e763aca67 PARQUET-2226: Support merge bloom filters (#1020)
add d6417dfad PARQUET-2227: Refactor several file rewriters to use a new
unified ParquetRewriter implementation (#1014)
add fd1326a8a PARQUET-2173: Fix parquet build against hadoop 3.3.3+ (#985)
add c36be7b99 PARQUET-2229: ParquetRewriter masks and encrypts the same
column (#1021)
add 261f7d267 Fix notIn for columns with null values. (#1028)
add 720aa3d3c PARQUET-2228: ParquetRewriter supports more than one input
file (#1026)
add bf38e0fe5 PARQUET-2241: Fix ByteStreamSplitValuesReader with nulls
(#1025)
add d1f59a033 PARQUET-2247: Fail-fast if CapacityByteArrayOutputStream
write overflow (#1031)
add 65f540779 PARQUET-2243: Support zstd-jni in DirectCodecFactory (#1027)
add c9cfe8214 PARQUET-2246: Add short circuit logic to column index
filter. (#1030)
add 79e29d44b PARQUET-2230: Add a new rewrite command powered by
ParquetRewriter (#1034)
add 4e9e79c89 PARQUET-2251 Avoid generating Bloomfilter when all pages of
a column are encoded by dictionary in parquet v1 (#1033)
add 9fcf4cea0 PARQUET-2103: Fix crypto exception in print toPrettyJSON
(#1019)
add 40bffcfdd Add Gang Wu as committer (#1037)
add 43ce92862 PARQUET-2230: [CLI] Deprecate commands replaced by rewrite
add d730fa746 PARQUET-2159: Vectorized BytePacker decoder using Java
VectorAPI (#1011)
add cc145b3f4 PARQUET-2252: Make row range methods public (#1038)
add 72442b5c5 PARQUET-2164: Check size of buffered data to prevent page
data from overflowing (#1032)
add d38044f53 PARQUET-2202: Review usage and implementation of
Preconditions.checkargument method (#1035)
add 1235003e7 PARQUET-2258: Storing toString fields in FilterPredicate
instances can lead to memory pressure
add 8cbbaefd8 PARQUET-2262: Fix local build failure due to missing
surefire.argLine (#1045)
add a06afc9e0 MINOR: Prepare for 1.14.0 development (#1047)
add e9313638f PARQUET-2260: Bloom filter size shouldn't exceed the
configured maxBytes (#1043)
add a1fdcfaa6 MINOR: Fix description of 'parquet.writer.version'
add 4ace2adb2 Bump protobuf-java from 3.17.3 to 3.19.6 in /parquet-protobuf
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (62a7415c7)
\
N -- N -- N
refs/heads/dependabot/maven/parquet-protobuf/com.google.protobuf-protobuf-java-3.19.6
(4ace2adb2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/{test.yml => vector-plugins.yml} | 10 +-
CHANGES.md | 127 +-
README.md | 15 +
dev/COMMITTERS.md | 1 +
parquet-arrow/pom.xml | 2 +-
parquet-avro/pom.xml | 2 +-
.../parquet/avro/AvroIndexedRecordConverter.java | 2 +-
.../apache/parquet/avro/AvroRecordConverter.java | 6 +-
.../org/apache/parquet/avro/AvroWriteSupport.java | 10 +-
.../parquet/avro/TestByteStreamSplitE2E.java | 114 +
parquet-benchmarks/pom.xml | 2 +-
parquet-cli/README.md | 31 +-
parquet-cli/pom.xml | 2 +-
.../java/org/apache/parquet/cli/BaseCommand.java | 12 +-
.../src/main/java/org/apache/parquet/cli/Help.java | 11 +-
.../src/main/java/org/apache/parquet/cli/Main.java | 4 +
.../src/main/java/org/apache/parquet/cli/Util.java | 6 +-
.../parquet/cli/commands/ColumnMaskingCommand.java | 17 +-
.../parquet/cli/commands/PruneColumnsCommand.java | 15 +-
.../parquet/cli/commands/RewriteCommand.java | 131 +
.../commands/{CatCommand.java => ScanCommand.java} | 53 +-
.../apache/parquet/cli/commands/SchemaCommand.java | 2 +-
.../cli/commands/TransCompressionCommand.java | 5 +-
.../java/org/apache/parquet/cli/json/AvroJson.java | 16 +-
...ionCommandTest.java => RewriteCommandTest.java} | 18 +-
.../{CatCommandTest.java => ScanCommandTest.java} | 8 +-
parquet-column/pom.xml | 2 +-
.../apache/parquet/column/ParquetProperties.java | 6 +-
.../parquet/column/impl/ColumnWriterBase.java | 2 +-
.../bitpacking/ByteBitPackingValuesReader.java | 38 +-
.../values/bloomfilter/BlockSplitBloomFilter.java | 27 +
.../column/values/bloomfilter/BloomFilter.java | 24 +
.../ByteStreamSplitValuesReader.java | 27 +-
.../values/delta/DeltaBinaryPackingConfig.java | 2 +-
.../values/dictionary/PlainValuesDictionary.java | 2 +-
.../parquet/filter2/predicate/Operators.java | 32 +-
.../filter2/predicate/PrimitiveToBoxedClass.java | 2 +-
.../recordlevel/FilteringGroupConverter.java | 2 +-
.../internal/column/columnindex/IndexIterator.java | 6 +-
.../filter2/columnindex/ColumnIndexFilter.java | 14 +-
.../internal/filter2/columnindex/RowRanges.java | 31 +-
.../parquet/schema/LogicalTypeAnnotation.java | 4 +-
.../org/apache/parquet/schema/PrimitiveType.java | 4 +-
.../main/java/org/apache/parquet/schema/Types.java | 26 +-
.../bloomfilter/TestBlockSplitBloomFilter.java | 40 +
parquet-common/pom.xml | 2 +-
.../java/org/apache/parquet/Preconditions.java | 134 +-
.../bytes/CapacityByteArrayOutputStream.java | 23 +-
.../hadoop/metadata/CompressionCodecName.java | 5 +-
.../java/org/apache/parquet/util/DynMethods.java | 6 +-
.../java/org/apache/parquet/TestPreconditions.java | 196 +-
parquet-encoding/pom.xml | 2 +-
.../column/values/bitpacking/BytePacker.java | 33 +
.../parquet/column/values/bitpacking/Packer.java | 28 +
parquet-format-structures/pom.xml | 2 +-
parquet-generator/pom.xml | 2 +-
parquet-hadoop-bundle/pom.xml | 2 +-
parquet-hadoop/README.md | 14 +-
parquet-hadoop/pom.xml | 7 +-
.../filter2/dictionarylevel/DictionaryFilter.java | 8 +
.../format/converter/ParquetMetadataConverter.java | 14 +-
.../parquet/hadoop/ColumnChunkPageWriteStore.java | 4 +-
.../apache/parquet/hadoop/DirectCodecFactory.java | 4 +
.../java/org/apache/parquet/hadoop/DirectZstd.java | 144 +
.../parquet/hadoop/ParquetEmptyBlockException.java | 19 +-
.../apache/parquet/hadoop/ParquetFileReader.java | 53 +-
.../apache/parquet/hadoop/ParquetFileWriter.java | 21 +-
.../apache/parquet/hadoop/ParquetInputFormat.java | 2 +-
.../org/apache/parquet/hadoop/ParquetWriter.java | 33 +
.../codec/{SnappyCodec.java => Lz4RawCodec.java} | 61 +-
.../parquet/hadoop/codec/Lz4RawCompressor.java | 29 +-
.../parquet/hadoop/codec/Lz4RawDecompressor.java | 30 +-
...pyCompressor.java => NonBlockedCompressor.java} | 72 +-
...compressor.java => NonBlockedDecompressor.java} | 64 +-
.../parquet/hadoop/codec/SnappyCompressor.java | 138 +-
.../parquet/hadoop/codec/SnappyDecompressor.java | 134 +-
.../parquet/hadoop/metadata/FileMetaData.java | 38 +-
.../parquet/hadoop/metadata/ParquetMetadata.java | 10 +-
.../AesMode.java => hadoop/rewrite/MaskMode.java} | 21 +-
.../parquet/hadoop/rewrite/ParquetRewriter.java | 818 ++++++
.../parquet/hadoop/rewrite/RewriteOptions.java | 262 ++
.../parquet/hadoop/util/ColumnEncryptor.java | 255 +-
.../apache/parquet/hadoop/util/ColumnMasker.java | 174 +-
.../apache/parquet/hadoop/util/ColumnPruner.java | 98 +-
.../parquet/hadoop/util/CompressionConverter.java | 209 +-
.../dictionarylevel/DictionaryFilterTest.java | 12 +
.../apache/parquet/hadoop/TestBloomFiltering.java | 18 +-
.../parquet/hadoop/TestDirectCodecFactory.java | 2 -
.../hadoop/TestParquetReaderEmptyBlock.java | 170 ++
.../apache/parquet/hadoop/TestParquetWriter.java | 73 +-
.../hadoop/TestParquetWriterTruncation.java | 125 +
.../parquet/hadoop/TestStoreBloomFilter.java | 132 +
.../parquet/hadoop/codec/TestCompressionCodec.java | 177 ++
.../hadoop/codec/TestInteropReadLz4RawCodec.java | 129 +
.../hadoop/rewrite/ParquetRewriterTest.java | 697 +++++
...eterTest.java => CompressionConverterTest.java} | 2 +-
.../test/resources/test-empty-row-group_1.parquet | Bin 0 -> 191 bytes
.../test/resources/test-empty-row-group_2.parquet | Bin 0 -> 675 bytes
.../test/resources/test-empty-row-group_3.parquet | Bin 0 -> 781 bytes
parquet-jackson/pom.xml | 2 +-
parquet-pig-bundle/pom.xml | 2 +-
parquet-pig/pom.xml | 2 +-
parquet-plugins/parquet-encoding-vector/pom.xml | 127 +
.../bitpacking/ByteBitPacking512VectorLE.java | 3010 ++++++++++++++++++++
.../values/bitpacking/ParquetReadRouter.java | 133 +
.../column/values/bitpacking/VectorSupport.java | 16 +-
.../bitpacking/TestByteBitPacking512VectorLE.java | 169 ++
.../values/bitpacking/TestParquetReadRouter.java | 59 +
.../parquet-plugins-benchmarks}/pom.xml | 43 +-
.../benchmarks/ByteBitPackingVectorBenchmarks.java | 92 +
.../src/main/resources/log4j.properties | 0
parquet-protobuf/README.md | 2 +-
parquet-protobuf/pom.xml | 20 +-
.../apache/parquet/proto/ProtoSchemaConverter.java | 148 +-
.../apache/parquet/proto/ProtoWriteSupport.java | 32 +-
.../parquet/proto/ProtoSchemaConverterTest.java | 634 +++--
.../parquet/proto/ProtoWriteSupportTest.java | 254 +-
.../java/org/apache/parquet/proto/TestUtils.java | 5 +
parquet-protobuf/src/test/resources/BinaryTree.par | 50 +
parquet-protobuf/src/test/resources/Struct.par | 110 +
.../{TestProto3SchemaV1.proto => Trees.proto} | 25 +-
parquet-protobuf/src/test/resources/Value.par | 105 +
parquet-protobuf/src/test/resources/WideTree.par | 27 +
parquet-scala/pom.xml | 2 +-
parquet-thrift/pom.xml | 2 +-
pom.xml | 51 +-
126 files changed, 9036 insertions(+), 1707 deletions(-)
copy .github/workflows/{test.yml => vector-plugins.yml} (78%)
create mode 100644
parquet-avro/src/test/java/org/apache/parquet/avro/TestByteStreamSplitE2E.java
create mode 100644
parquet-cli/src/main/java/org/apache/parquet/cli/commands/RewriteCommand.java
copy
parquet-cli/src/main/java/org/apache/parquet/cli/commands/{CatCommand.java =>
ScanCommand.java} (60%)
copy
parquet-cli/src/test/java/org/apache/parquet/cli/commands/{TransCompressionCommandTest.java
=> RewriteCommandTest.java} (74%)
copy
parquet-cli/src/test/java/org/apache/parquet/cli/commands/{CatCommandTest.java
=> ScanCommandTest.java} (83%)
create mode 100644
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/DirectZstd.java
copy
parquet-pig/src/main/java/org/apache/parquet/pig/TupleConversionException.java
=>
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetEmptyBlockException.java
(74%)
copy
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/{SnappyCodec.java
=> Lz4RawCodec.java} (63%)
copy
parquet-common/src/main/java/org/apache/parquet/bytes/DirectByteBufferAllocator.java
=>
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/Lz4RawCompressor.java
(57%)
copy
parquet-column/src/main/java/org/apache/parquet/column/values/bloomfilter/XxHash.java
=>
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/Lz4RawDecompressor.java
(51%)
copy
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/{SnappyCompressor.java
=> NonBlockedCompressor.java} (69%)
copy
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/{SnappyDecompressor.java
=> NonBlockedDecompressor.java} (72%)
copy parquet-hadoop/src/main/java/org/apache/parquet/{crypto/AesMode.java =>
hadoop/rewrite/MaskMode.java} (69%)
mode change 100755 => 100644
create mode 100644
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/ParquetRewriter.java
create mode 100644
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/RewriteOptions.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetReaderEmptyBlock.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetWriterTruncation.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestStoreBloomFilter.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/codec/TestCompressionCodec.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/codec/TestInteropReadLz4RawCodec.java
create mode 100644
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/rewrite/ParquetRewriterTest.java
rename
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/util/{CompressionConveterTest.java
=> CompressionConverterTest.java} (99%)
create mode 100644
parquet-hadoop/src/test/resources/test-empty-row-group_1.parquet
create mode 100644
parquet-hadoop/src/test/resources/test-empty-row-group_2.parquet
create mode 100644
parquet-hadoop/src/test/resources/test-empty-row-group_3.parquet
create mode 100644 parquet-plugins/parquet-encoding-vector/pom.xml
create mode 100644
parquet-plugins/parquet-encoding-vector/src/main/java/org/apache/parquet/column/values/bitpacking/ByteBitPacking512VectorLE.java
create mode 100644
parquet-plugins/parquet-encoding-vector/src/main/java/org/apache/parquet/column/values/bitpacking/ParquetReadRouter.java
copy
parquet-encoding/src/main/java/org/apache/parquet/column/values/bitpacking/BytePackerFactory.java
=>
parquet-plugins/parquet-encoding-vector/src/main/java/org/apache/parquet/column/values/bitpacking/VectorSupport.java
(88%)
create mode 100644
parquet-plugins/parquet-encoding-vector/src/test/java/org/apache/parquet/column/values/bitpacking/TestByteBitPacking512VectorLE.java
create mode 100644
parquet-plugins/parquet-encoding-vector/src/test/java/org/apache/parquet/column/values/bitpacking/TestParquetReadRouter.java
copy {parquet-benchmarks =>
parquet-plugins/parquet-plugins-benchmarks}/pom.xml (77%)
create mode 100644
parquet-plugins/parquet-plugins-benchmarks/src/main/java/org/apache/parquet/plugins/benchmarks/ByteBitPackingVectorBenchmarks.java
copy {parquet-benchmarks =>
parquet-plugins/parquet-plugins-benchmarks}/src/main/resources/log4j.properties
(100%)
create mode 100644 parquet-protobuf/src/test/resources/BinaryTree.par
create mode 100644 parquet-protobuf/src/test/resources/Struct.par
copy parquet-protobuf/src/test/resources/{TestProto3SchemaV1.proto =>
Trees.proto} (71%)
create mode 100644 parquet-protobuf/src/test/resources/Value.par
create mode 100644 parquet-protobuf/src/test/resources/WideTree.par
