This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new aa2be5e19b Issue #2357 : Run Parquet file input and output on Beam
(#9211)
aa2be5e19b is described below
commit aa2be5e19b3accefa9c081f0d5febf9be0637584
Author: Matt Casters <[email protected]>
AuthorDate: Tue Nov 3 13:50:07 2026 +0100
Issue #2357 : Run Parquet file input and output on Beam (#9211)
* Issue #2357 : Run Parquet file input and output on Beam
Write one Parquet file with ParquetIO and read each named file back
onto the incoming row. Partitioned output and a selected Parquet type
are refused while the graph is built.
* Issue #2357 : Correct Parquet types and open files with Beam
* Issue #2357 : Keep the scheme of Beam Parquet output paths
---
assemblies/static/src/main/resources/LICENSE | 3 +
.../pipeline/transforms/parquet-file-input.adoc | 4 +
.../pipeline/transforms/parquet-file-output.adoc | 4 +
plugins/engines/beam/pom.xml | 41 +-
.../apache/hop/beam/core/fn/EmptyToNullRowFn.java | 35 ++
.../apache/hop/beam/core/fn/HopRowToParquetFn.java | 174 ++++++
.../hop/beam/core/fn/ParquetColumnMapping.java | 39 ++
.../apache/hop/beam/core/fn/ReadParquetFileFn.java | 315 ++++++++++
.../HopPipelineMetaToBeamPipelineConverter.java | 10 +-
.../handler/BeamParquetInputTransformHandler.java | 187 ++++++
.../handler/BeamParquetOutputTransformHandler.java | 469 ++++++++++++++
.../java/org/apache/hop/beam/util/BeamConst.java | 2 +
.../beam/src/main/resources/dependencies.xml | 1 +
.../pipeline/handler/BeamParquetFileIOTest.java | 672 +++++++++++++++++++++
.../parquet/transforms/output/ParquetOutput.java | 32 +-
.../transforms/output/ParquetWriteSupport.java | 10 +-
16 files changed, 1966 insertions(+), 32 deletions(-)
diff --git a/assemblies/static/src/main/resources/LICENSE
b/assemblies/static/src/main/resources/LICENSE
index 8ca9c2b87b..872463e86e 100644
--- a/assemblies/static/src/main/resources/LICENSE
+++ b/assemblies/static/src/main/resources/LICENSE
@@ -595,6 +595,9 @@ This product bundles beam-sdks-java-io-kafka 2.74.0, which
is available under a
This product bundles beam-sdks-java-io-mqtt 2.74.0, which is available under a
"Apache 2.0" license. For details, see LICENSE.
+This product bundles beam-sdks-java-io-parquet 2.74.0, which is available
under a
+"Apache 2.0" license. For details, see LICENSE.
+
This product bundles beam-sdks-java-io-snowflake 2.74.0, which is available
under a
"Apache 2.0" license. For details, see LICENSE.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
index a5d99f0f8b..0fb9181b69 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-input.adoc
@@ -140,3 +140,7 @@ You can pick another Hop type from the last column instead.
|LIST, MAP, VARIANT and other nested columns
2+|Not supported
|===
+
+== Beam
+
+On a Beam runner each file is read by one worker, and the configured columns
are appended to the row that named the file. The filename is opened through
Beam's file systems, so a local path or a `file://`, `gs://`, `s3://` or
`hdfs://` URI works. A null or empty filename is skipped. The field list or a
metadata filename is required. When the incoming collection is empty and "send
a null row when empty" is set, one null row is emitted. That option needs a
bounded input in the global windo [...]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
index 6f2208c156..822660f8d7 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
@@ -320,3 +320,7 @@ The nanoseconds beyond the microsecond are dropped.
|===
Other Hop types can't be written to Parquet.
+
+== Beam
+
+On a Beam runner the transform writes one Parquet file with `ParquetIO`. The
filename, including the date and time, is resolved when the pipeline is
submitted. The copy number and the split number are not extra files.
Partitioned output, and a field that sets its own Parquet type, are refused. A
JSON field and a UUID field are refused as well: on the local engine they
become a JSON column and a UUID column, and on Beam `ParquetIO` would write
both as plain strings. Dictionary page size a [...]
diff --git a/plugins/engines/beam/pom.xml b/plugins/engines/beam/pom.xml
index 8151ec8b5a..2381717597 100644
--- a/plugins/engines/beam/pom.xml
+++ b/plugins/engines/beam/pom.xml
@@ -1182,6 +1182,25 @@
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-mqtt</artifactId>
</dependency>
+
+ <!-- Parquet file input and output.
+ beam-sdks-java-io-parquet and the Parquet plugin both resolve to
Parquet
+ 1.18.1. dependencies.xml also puts tech/parquet on the Beam
loader, so
+ that loader sees two identical copies of the Parquet jars. That is
+ harmless while both stay on 1.18.1. Do not assume the plugin
ships a
+ newer parquet-avro. -->
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-parquet</artifactId>
+ <exclusions>
+ <!-- Transitive and unused here, and it conflicts with the
jackson version this
+ plugin already manages. -->
+ <exclusion>
+ <groupId>com.google.j2objc</groupId>
+ <artifactId>j2objc-annotations</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<!-- #2378: Snowflake COPY through GCS. JDBC version comes from
hop-libs-jdbc (4.3.0),
not Beam's 4.0.2. Snowpipe's ingest SDK is omitted because this
transform does not
call it. -->
@@ -1692,6 +1711,12 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-tech-parquet</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-transform-addsequence</artifactId>
@@ -1792,22 +1817,6 @@
</exclusions>
</dependency>
- <!-- ParquetIO is only referenced by ParquetIOViabilityTest. Test
scope keeps
- the Parquet jars out of the Beam assembly. -->
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-sdks-java-io-parquet</artifactId>
- <scope>test</scope>
- <exclusions>
- <!-- Transitive and unused here, and it conflicts with the
jackson version this
- plugin already manages. -->
- <exclusion>
- <groupId>com.google.j2objc</groupId>
- <artifactId>j2objc-annotations</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
</dependencies>
<build>
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/EmptyToNullRowFn.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/EmptyToNullRowFn.java
new file mode 100644
index 0000000000..ac924c7a3c
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/EmptyToNullRowFn.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.beam.core.HopRow;
+
+/** Emits one all-null row when a Parquet input received no rows. */
+@RequiredArgsConstructor
+public class EmptyToNullRowFn extends DoFn<Long, HopRow> {
+ private final int width;
+
+ @ProcessElement
+ public void process(@Element Long count, OutputReceiver<HopRow> output) {
+ if (count != null && count == 0L) {
+ output.output(new HopRow(new Object[width]));
+ }
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopRowToParquetFn.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopRowToParquetFn.java
new file mode 100644
index 0000000000..ecd559f433
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/HopRowToParquetFn.java
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Date;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.parquet.transforms.output.ParquetWriteSupport;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+
+/** Maps one Hop row onto the flat Avro record ParquetIO writes. */
+@RequiredArgsConstructor
+public class HopRowToParquetFn extends DoFn<HopRow, GenericRecord> {
+ private final String transformName;
+ private final String rowMetaJson;
+ private final String schemaJson;
+ private final List<String> sourceFields;
+
+ private transient Schema schema;
+ private transient IRowMeta rowMeta;
+ private transient int[] indexes;
+ private transient Counter readCounter;
+ private transient Counter outputCounter;
+ private transient Counter errorCounter;
+
+ @Setup
+ public void setup() {
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ outputCounter = Metrics.counter(Pipeline.METRIC_NAME_OUTPUT,
transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ schema = new Schema.Parser().parse(schemaJson);
+ rowMeta = JsonRowMeta.fromJson(rowMetaJson);
+ indexes = new int[sourceFields.size()];
+ for (int i = 0; i < sourceFields.size(); i++) {
+ indexes[i] = rowMeta.indexOfValue(sourceFields.get(i));
+ if (indexes[i] < 0) {
+ throw new HopRuntimeException("Field [" + sourceFields.get(i) + "]
was not found");
+ }
+ }
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to initialize Parquet output " +
transformName, e);
+ }
+ }
+
+ @ProcessElement
+ public void process(@Element HopRow row, OutputReceiver<GenericRecord>
output) {
+ readCounter.inc();
+ try {
+ GenericRecord record = new GenericData.Record(schema);
+ Object[] values = row.getRow();
+ for (int i = 0; i < schema.getFields().size(); i++) {
+ Schema.Field field = schema.getFields().get(i);
+ record.put(field.name(), value(values, indexes[i], field));
+ }
+ output.output(record);
+ outputCounter.inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException(
+ "Unable to convert a row for Parquet output " + transformName, e);
+ }
+ }
+
+ private Object value(Object[] values, int index, Schema.Field field) throws
Exception {
+ if (rowMeta.isNull(values, index)) {
+ return null;
+ }
+ Schema type = nonNull(field.schema());
+ return switch (type.getType()) {
+ case LONG -> longValue(values, index, type);
+ case DOUBLE -> rowMeta.getNumber(values, index);
+ case BOOLEAN -> rowMeta.getBoolean(values, index);
+ case STRING -> rowMeta.getString(values, index);
+ case BYTES -> bytesValue(values, index, type, field.name());
+ default ->
+ throw new HopRuntimeException("Parquet output cannot write Avro type
" + type.getType());
+ };
+ }
+
+ /**
+ * A date is timestamp-millis, so it stays in milliseconds. A timestamp is
timestamp-micros, so it
+ * is the same microsecond value {@link ParquetWriteSupport#epochValue}
writes locally.
+ */
+ private Object longValue(Object[] values, int index, Schema type) throws
Exception {
+ IValueMeta valueMeta = rowMeta.getValueMeta(index);
+ if (type.getLogicalType() instanceof LogicalTypes.TimestampMicros
+ || (valueMeta.getType() == IValueMeta.TYPE_TIMESTAMP
+ && !(type.getLogicalType() instanceof
LogicalTypes.TimestampMillis))) {
+ return ParquetWriteSupport.epochValue(
+ valueMeta, values[index], LogicalTypeAnnotation.timestampType(true,
TimeUnit.MICROS));
+ }
+ if (valueMeta.getType() == IValueMeta.TYPE_DATE
+ || valueMeta.getType() == IValueMeta.TYPE_TIMESTAMP
+ || type.getLogicalType() instanceof LogicalTypes.TimestampMillis) {
+ Date date = rowMeta.getDate(values, index);
+ return date == null ? null : date.getTime();
+ }
+ return rowMeta.getInteger(values, index);
+ }
+
+ /**
+ * A decimal logical type needs the unscaled two's-complement bytes of the
scaled big number. A
+ * binary field keeps {@link IRowMeta#getBinary}. Avro bytes are a {@link
ByteBuffer} either way.
+ */
+ private Object bytesValue(Object[] values, int index, Schema type, String
fieldName)
+ throws Exception {
+ if (type.getLogicalType() instanceof LogicalTypes.Decimal decimal) {
+ BigDecimal number = rowMeta.getBigNumber(values, index);
+ if (number == null) {
+ return null;
+ }
+ Binary binary =
+ ParquetWriteSupport.decimalBytes(
+ fieldName,
+ rowMeta.getValueMeta(index),
+ number,
+ (DecimalLogicalTypeAnnotation)
+ LogicalTypeAnnotation.decimalType(decimal.getScale(),
decimal.getPrecision()));
+ return ByteBuffer.wrap(binary.getBytes());
+ }
+ byte[] bytes = rowMeta.getBinary(values, index);
+ return bytes == null ? null : ByteBuffer.wrap(bytes);
+ }
+
+ private static Schema nonNull(Schema type) {
+ if (type.getType() != Schema.Type.UNION) {
+ return type;
+ }
+ for (Schema candidate : type.getTypes()) {
+ if (candidate.getType() != Schema.Type.NULL) {
+ return candidate;
+ }
+ }
+ return type;
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ParquetColumnMapping.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ParquetColumnMapping.java
new file mode 100644
index 0000000000..693344a052
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ParquetColumnMapping.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.io.Serializable;
+
+/** One Parquet column appended to an incoming Hop row. Type is an {@code
IValueMeta} id. */
+public class ParquetColumnMapping implements Serializable {
+ private final String columnName;
+ private final int hopType;
+
+ public ParquetColumnMapping(String columnName, int hopType) {
+ this.columnName = columnName;
+ this.hopType = hopType;
+ }
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public int getHopType() {
+ return hopType;
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ReadParquetFileFn.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ReadParquetFileFn.java
new file mode 100644
index 0000000000..6e80ae1a63
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/core/fn/ReadParquetFileFn.java
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.core.fn;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.nio.channels.SeekableByteChannel;
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.MatchResult;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hop.beam.core.BeamHop;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.core.exception.HopRuntimeException;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.io.DelegatingSeekableInputStream;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+
+/**
+ * Reads the Parquet file named by one incoming field and appends the
configured columns.
+ *
+ * <p>{@code ParquetIO.readFiles} returns records without the path, so the
source row would be lost.
+ * This reads the file on the worker that holds the row. The path is opened
with Beam {@link
+ * FileSystems}, so a local path or a {@code file://}, {@code gs://}, {@code
s3://} or {@code
+ * hdfs://} URI works. The Avro projection is the same one {@code
ParquetIO.read(schema)} would use,
+ * and each value is turned into the Hop type of the column.
+ */
+@RequiredArgsConstructor
+public class ReadParquetFileFn extends DoFn<HopRow, HopRow> {
+ private final String transformName;
+ private final String rowMetaJson;
+ private final String filenameField;
+ private final String schemaJson;
+ private final List<ParquetColumnMapping> columns;
+
+ private transient IRowMeta rowMeta;
+ private transient int filenameIndex;
+ private transient int inputSize;
+ private transient Schema schema;
+ private transient Counter readCounter;
+ private transient Counter writtenCounter;
+ private transient Counter errorCounter;
+
+ @Setup
+ public void setup() {
+ readCounter = Metrics.counter(Pipeline.METRIC_NAME_READ, transformName);
+ writtenCounter = Metrics.counter(Pipeline.METRIC_NAME_WRITTEN,
transformName);
+ errorCounter = Metrics.counter(Pipeline.METRIC_NAME_ERROR, transformName);
+ try {
+ BeamHop.init();
+ rowMeta = JsonRowMeta.fromJson(rowMetaJson);
+ filenameIndex = rowMeta.indexOfValue(filenameField);
+ if (filenameIndex < 0) {
+ throw new HopRuntimeException("Filename field '" + filenameField + "'
was not found");
+ }
+ inputSize = rowMeta.size();
+ schema = new Schema.Parser().parse(schemaJson);
+ Metrics.counter(Pipeline.METRIC_NAME_INIT, transformName).inc();
+ } catch (Exception e) {
+ errorCounter.inc();
+ throw new HopRuntimeException("Unable to initialize Parquet input " +
transformName, e);
+ }
+ }
+
+ @ProcessElement
+ public void process(@Element HopRow row, OutputReceiver<HopRow> output) {
+ String filename = null;
+ try {
+ filename = rowMeta.getString(row.getRow(), filenameIndex);
+ if (filename == null || filename.isEmpty()) {
+ return;
+ }
+ Configuration configuration = new Configuration(false);
+ AvroReadSupport.setRequestedProjection(configuration, schema);
+ try (ParquetReader<GenericRecord> reader = open(filename,
configuration)) {
+ GenericRecord record;
+ while ((record = reader.read()) != null) {
+ Object[] copy = Arrays.copyOf(row.getRow(), inputSize +
columns.size());
+ for (int i = 0; i < columns.size(); i++) {
+ copy[inputSize + i] =
+ hopValue(record.get(columns.get(i).getColumnName()),
columns.get(i));
+ }
+ output.output(new HopRow(copy));
+ readCounter.inc();
+ writtenCounter.inc();
+ }
+ }
+ } catch (Exception e) {
+ errorCounter.inc();
+ String where = filename == null ? "" : " '" + filename + "'";
+ throw new HopRuntimeException(
+ "Unable to read a Parquet file" + where + " for transform " +
transformName, e);
+ }
+ }
+
+ private static ParquetReader<GenericRecord> open(String filename,
Configuration configuration)
+ throws IOException {
+ MatchResult.Metadata metadata = FileSystems.matchSingleFileSpec(filename);
+ if (!metadata.isReadSeekEfficient()) {
+ throw new IOException("Parquet input cannot seek in " + filename);
+ }
+ ReadableByteChannel opened = FileSystems.open(metadata.resourceId());
+ if (!(opened instanceof SeekableByteChannel channel)) {
+ opened.close();
+ throw new IOException("Parquet input cannot seek in " + filename);
+ }
+ try {
+ return AvroParquetReader.<GenericRecord>builder(
+ new SeekableChannelInputFile(filename, channel))
+ .withConf(configuration)
+ .build();
+ } catch (Exception e) {
+ channel.close();
+ if (e instanceof IOException ioException) {
+ throw ioException;
+ }
+ throw new IOException("Unable to open Parquet file " + filename, e);
+ }
+ }
+
+ private Object hopValue(Object value, ParquetColumnMapping column) {
+ if (value == null) {
+ return null;
+ }
+ Schema.Field field = schema.getField(column.getColumnName());
+ if (field == null) {
+ throw new HopRuntimeException(
+ "Parquet column '" + column.getColumnName() + "' is not in the
schema");
+ }
+ LogicalType logicalType = nonNull(field.schema()).getLogicalType();
+ return switch (column.getHopType()) {
+ case IValueMeta.TYPE_STRING -> value.toString();
+ case IValueMeta.TYPE_INTEGER -> longValue(value);
+ case IValueMeta.TYPE_NUMBER -> doubleValue(value);
+ case IValueMeta.TYPE_BOOLEAN -> booleanValue(value);
+ case IValueMeta.TYPE_DATE -> dateValue(value, logicalType);
+ case IValueMeta.TYPE_TIMESTAMP -> timestampValue(value, logicalType);
+ case IValueMeta.TYPE_BIGNUMBER -> bigNumberValue(value, logicalType);
+ case IValueMeta.TYPE_BINARY -> bytesOf(value);
+ default -> value;
+ };
+ }
+
+ private static long longValue(Object value) {
+ if (value instanceof Number number) {
+ return number.longValue();
+ }
+ throw new HopRuntimeException("Parquet integer value is " +
value.getClass().getName());
+ }
+
+ private static double doubleValue(Object value) {
+ if (value instanceof Number number) {
+ return number.doubleValue();
+ }
+ throw new HopRuntimeException("Parquet number value is " +
value.getClass().getName());
+ }
+
+ private static boolean booleanValue(Object value) {
+ if (value instanceof Boolean bool) {
+ return bool;
+ }
+ throw new HopRuntimeException("Parquet boolean value is " +
value.getClass().getName());
+ }
+
+ private static Date dateValue(Object value, LogicalType logicalType) {
+ if (value instanceof java.sql.Timestamp timestamp) {
+ return new Date(timestamp.getTime());
+ }
+ if (value instanceof Date date) {
+ return date;
+ }
+ if (value instanceof Instant instant) {
+ return Date.from(instant);
+ }
+ long raw = longValue(value);
+ if (logicalType instanceof LogicalTypes.TimestampMicros) {
+ return new Date(Math.floorDiv(raw, 1_000L));
+ }
+ return new Date(raw);
+ }
+
+ private static java.sql.Timestamp timestampValue(Object value, LogicalType
logicalType) {
+ if (value instanceof java.sql.Timestamp timestamp) {
+ return timestamp;
+ }
+ if (value instanceof Instant instant) {
+ return java.sql.Timestamp.from(instant);
+ }
+ if (value instanceof Date date) {
+ return new java.sql.Timestamp(date.getTime());
+ }
+ long raw = longValue(value);
+ if (logicalType instanceof LogicalTypes.TimestampMillis) {
+ return new java.sql.Timestamp(raw);
+ }
+ long seconds = Math.floorDiv(raw, 1_000_000L);
+ int nanos = (int) (Math.floorMod(raw, 1_000_000L) * 1_000L);
+ return java.sql.Timestamp.from(Instant.ofEpochSecond(seconds, nanos));
+ }
+
+ private static BigDecimal bigNumberValue(Object value, LogicalType
logicalType) {
+ if (value instanceof BigDecimal decimal) {
+ return decimal;
+ }
+ if (value instanceof CharSequence text) {
+ return new BigDecimal(text.toString());
+ }
+ int scale = 0;
+ if (logicalType instanceof LogicalTypes.Decimal decimal) {
+ scale = decimal.getScale();
+ }
+ return new BigDecimal(new BigInteger(bytesOf(value)), scale);
+ }
+
+ private static byte[] bytesOf(Object value) {
+ if (value instanceof byte[] bytes) {
+ return bytes;
+ }
+ if (value instanceof ByteBuffer buffer) {
+ ByteBuffer duplicate = buffer.duplicate();
+ byte[] bytes = new byte[duplicate.remaining()];
+ duplicate.get(bytes);
+ return bytes;
+ }
+ throw new HopRuntimeException("Parquet binary value is " +
value.getClass().getName());
+ }
+
+ private static Schema nonNull(Schema type) {
+ if (type.getType() != Schema.Type.UNION) {
+ return type;
+ }
+ for (Schema candidate : type.getTypes()) {
+ if (candidate.getType() != Schema.Type.NULL) {
+ return candidate;
+ }
+ }
+ return type;
+ }
+
+ /**
+ * One seekable Beam channel as the Parquet {@link InputFile} {@code
ParquetIO} builds around
+ * {@code ReadableFile.openSeekable()}.
+ */
+ private static final class SeekableChannelInputFile implements InputFile {
+ private final String filename;
+ private final SeekableByteChannel channel;
+
+ private SeekableChannelInputFile(String filename, SeekableByteChannel
channel) {
+ this.filename = filename;
+ this.channel = channel;
+ }
+
+ @Override
+ public long getLength() throws IOException {
+ return channel.size();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ return new
DelegatingSeekableInputStream(Channels.newInputStream(channel)) {
+ @Override
+ public long getPos() throws IOException {
+ return channel.position();
+ }
+
+ @Override
+ public void seek(long newPos) throws IOException {
+ channel.position(newPos);
+ }
+ };
+ }
+
+ @Override
+ public String toString() {
+ return filename;
+ }
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
index 6f9cd84b05..79d7b784fe 100644
---
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/HopPipelineMetaToBeamPipelineConverter.java
@@ -48,6 +48,8 @@ import
org.apache.hop.beam.pipeline.handler.BeamAvroOutputTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamGenericTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamMemoryGroupByTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamMergeJoinTransformHandler;
+import org.apache.hop.beam.pipeline.handler.BeamParquetInputTransformHandler;
+import org.apache.hop.beam.pipeline.handler.BeamParquetOutputTransformHandler;
import org.apache.hop.beam.pipeline.handler.BeamRowGeneratorTransformHandler;
import org.apache.hop.beam.util.BeamConst;
import org.apache.hop.core.Const;
@@ -94,7 +96,9 @@ public class HopPipelineMetaToBeamPipelineConverter {
BeamConst.STRING_MEMORY_GROUP_BY_PLUGIN_ID,
BeamConst.STRING_ADD_SEQUENCE_PLUGIN_ID,
BeamConst.STRING_AVRO_FILE_INPUT_PLUGIN_ID,
- BeamConst.STRING_AVRO_OUTPUT_PLUGIN_ID);
+ BeamConst.STRING_AVRO_OUTPUT_PLUGIN_ID,
+ BeamConst.STRING_PARQUET_FILE_INPUT_PLUGIN_ID,
+ BeamConst.STRING_PARQUET_FILE_OUTPUT_PLUGIN_ID);
/**
* Transform meta classes that Beam refuses to run at all, mapped to the
user-facing reason. The
@@ -275,6 +279,10 @@ public class HopPipelineMetaToBeamPipelineConverter {
BeamConst.STRING_AVRO_FILE_INPUT_PLUGIN_ID, new
BeamAvroInputTransformHandler());
transformHandlers.put(
BeamConst.STRING_AVRO_OUTPUT_PLUGIN_ID, new
BeamAvroOutputTransformHandler());
+ transformHandlers.put(
+ BeamConst.STRING_PARQUET_FILE_INPUT_PLUGIN_ID, new
BeamParquetInputTransformHandler());
+ transformHandlers.put(
+ BeamConst.STRING_PARQUET_FILE_OUTPUT_PLUGIN_ID, new
BeamParquetOutputTransformHandler());
genericTransformHandler = new BeamGenericTransformHandler();
}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetInputTransformHandler.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetInputTransformHandler.java
new file mode 100644
index 0000000000..b3b3908710
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetInputTransformHandler.java
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.pipeline.handler;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.beam.core.fn.EmptyToNullRowFn;
+import org.apache.hop.beam.core.fn.ParquetColumnMapping;
+import org.apache.hop.beam.core.fn.ReadParquetFileFn;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.parquet.transforms.input.ParquetField;
+import org.apache.hop.parquet.transforms.input.ParquetInputMeta;
+import org.apache.hop.parquet.transforms.output.ParquetOutput;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+
+/**
+ * Issue #2357: Parquet file input on Beam.
+ *
+ * <p>Each incoming row names a file. The file is opened with Beam file
systems, so a local path or
+ * a {@code file://}, {@code gs://}, {@code s3://} or {@code hdfs://} URI
works. The configured
+ * columns are appended to that row. The field list or a metadata filename is
required, because the
+ * output row layout has to exist when the graph is built. Sending a null row
when the input is
+ * empty needs a bounded collection in the global window.
+ */
+public class BeamParquetInputTransformHandler extends BeamBaseTransformHandler
+ implements IBeamPipelineTransformHandler {
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+
+ if (input == null || previousTransforms == null ||
previousTransforms.isEmpty()) {
+ throw new HopException(
+ "Parquet input '" + transformMeta.getName() + "' needs one incoming
transform");
+ }
+ ParquetInputMeta meta = new ParquetInputMeta();
+ loadTransformMetadata(meta, transformMeta, metadataProvider, pipelineMeta);
+ String filenameField = variables.resolve(meta.getFilenameField());
+ if (Utils.isEmpty(filenameField)) {
+ throw new HopException(
+ "Parquet input '" + transformMeta.getName() + "' has no filename
field");
+ }
+ if (rowMeta == null || rowMeta.indexOfValue(filenameField) < 0) {
+ throw new HopException(
+ "Parquet input '"
+ + transformMeta.getName()
+ + "' cannot find filename field '"
+ + filenameField
+ + "'");
+ }
+
+ List<org.apache.hop.parquet.transforms.output.ParquetField> schemaFields =
new ArrayList<>();
+ List<IValueMeta> valueMetas = new ArrayList<>();
+ List<ParquetColumnMapping> columns = new ArrayList<>();
+ if (meta.getFields() != null && !meta.getFields().isEmpty()) {
+ for (ParquetField field : meta.getFields()) {
+ IValueMeta valueMeta = field.createValueMeta();
+ String column =
+ Utils.isEmpty(field.getSourceField()) ? field.getTargetField() :
field.getSourceField();
+ schemaFields.add(new
org.apache.hop.parquet.transforms.output.ParquetField(column, column));
+ valueMetas.add(valueMeta);
+ columns.add(new ParquetColumnMapping(column, valueMeta.getType()));
+ }
+ } else if (!Utils.isEmpty(meta.getMetadataFilename())) {
+ IRowMeta fileRow =
+ ParquetInputMeta.extractRowMeta(variables,
variables.resolve(meta.getMetadataFilename()));
+ for (int i = 0; i < fileRow.size(); i++) {
+ IValueMeta valueMeta = fileRow.getValueMeta(i);
+ schemaFields.add(
+ new org.apache.hop.parquet.transforms.output.ParquetField(
+ valueMeta.getName(), valueMeta.getName()));
+ valueMetas.add(valueMeta);
+ columns.add(new ParquetColumnMapping(valueMeta.getName(),
valueMeta.getType()));
+ }
+ }
+ if (columns.isEmpty()) {
+ throw new HopException(
+ "Parquet input '"
+ + transformMeta.getName()
+ + "' needs a field list or a metadata filename on Beam");
+ }
+
+ Schema schema = ParquetOutput.avroSchema(schemaFields, valueMetas);
+ PCollection<HopRow> read =
+ input
+ .apply(
+ transformMeta.getName() + " Read Parquet",
+ ParDo.of(
+ new ReadParquetFileFn(
+ transformMeta.getName(),
+ JsonRowMeta.toJson(rowMeta),
+ filenameField,
+ schema.toString(),
+ columns)))
+ .setCoder(new HopRowCoder());
+
+ if (meta.isSendingNullsRowWhenEmpty()) {
+ if (!(input.getWindowingStrategy().getWindowFn() instanceof
GlobalWindows)
+ || input.isBounded() != IsBounded.BOUNDED) {
+ throw new HopException(
+ "Parquet input '"
+ + transformMeta.getName()
+ + "' can send a null row when empty only for a bounded input
in the global window. "
+ + "Count.globally() fails while the graph is built for a
streaming or otherwise windowed input.");
+ }
+ int width = rowMeta.size() + columns.size();
+ PCollection<HopRow> whenEmpty =
+ input
+ .apply(transformMeta.getName() + " Count", Count.globally())
+ .apply(transformMeta.getName() + " Null row", ParDo.of(new
EmptyToNullRowFn(width)))
+ .setCoder(new HopRowCoder());
+ read =
+ PCollectionList.of(read)
+ .and(whenEmpty)
+ .apply(transformMeta.getName() + " Empty row",
Flatten.pCollections())
+ .setCoder(new HopRowCoder());
+ }
+
+ transformCollectionMap.put(transformMeta.getName(), read);
+ log.logBasic(
+ "Handled transform (PARQUET INPUT) : "
+ + transformMeta.getName()
+ + ", filename field "
+ + filenameField);
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetOutputTransformHandler.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetOutputTransformHandler.java
new file mode 100644
index 0000000000..b979618f84
--- /dev/null
+++
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/pipeline/handler/BeamParquetOutputTransformHandler.java
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.pipeline.handler;
+
+import java.net.URI;
+import java.nio.file.Paths;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.avro.coders.AvroCoder;
+import org.apache.beam.sdk.io.Compression;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.fn.HopRowToParquetFn;
+import org.apache.hop.beam.engines.IBeamPipelineEngineRunConfiguration;
+import org.apache.hop.beam.pipeline.IBeamPipelineTransformHandler;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.JsonRowMeta;
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.core.vfs.HopVfs;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.parquet.transforms.output.ParquetField;
+import org.apache.hop.parquet.transforms.output.ParquetOutput;
+import org.apache.hop.parquet.transforms.output.ParquetOutputMeta;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * Issue #2357: Parquet file output on Beam.
+ *
+ * <p>One file, written with {@code
FileIO.write().via(ParquetIO.sink(schema))}. The directory is
+ * split from the filename without {@code java.io.File}, so a {@code gs://} or
{@code s3://} URI
+ * keeps both slashes. Date and time are resolved when the pipeline is
submitted. Copy number and
+ * split number are not extra Beam shards. Partitioned output, a field with
its own Parquet type,
+ * and a JSON or UUID field are refused. JSON and UUID become their own
Parquet columns on the local
+ * engine. ParquetIO would write both as plain strings.
+ */
+public class BeamParquetOutputTransformHandler extends BeamBaseTransformHandler
+ implements IBeamPipelineTransformHandler {
+
+ @Override
+ public boolean isInput() {
+ return false;
+ }
+
+ @Override
+ public boolean isOutput() {
+ return false;
+ }
+
+ @Override
+ public void handleTransform(
+ ILogChannel log,
+ IVariables variables,
+ String runConfigurationName,
+ IBeamPipelineEngineRunConfiguration runConfiguration,
+ String dataSamplersJson,
+ IHopMetadataProvider metadataProvider,
+ PipelineMeta pipelineMeta,
+ TransformMeta transformMeta,
+ Map<String, PCollection<HopRow>> transformCollectionMap,
+ Pipeline pipeline,
+ IRowMeta rowMeta,
+ List<TransformMeta> previousTransforms,
+ PCollection<HopRow> input,
+ String parentLogChannelId)
+ throws HopException {
+
+ if (input == null || previousTransforms == null ||
previousTransforms.isEmpty()) {
+ throw new HopException(
+ "Parquet output '" + transformMeta.getName() + "' needs one incoming
transform");
+ }
+ ParquetOutputMeta meta = new ParquetOutputMeta();
+ loadTransformMetadata(meta, transformMeta, metadataProvider, pipelineMeta);
+ if (meta.isPartitioning()) {
+ throw new HopException(
+ "Parquet output '" + transformMeta.getName() + "' cannot partition
files on Beam");
+ }
+ if (rowMeta == null) {
+ throw new HopException(
+ "Parquet output '" + transformMeta.getName() + "' has no incoming
row layout");
+ }
+
+ List<ParquetField> fields = new ArrayList<>();
+ List<IValueMeta> valueMetas = new ArrayList<>();
+ List<String> sourceFields = new ArrayList<>();
+ if (meta.getFields() != null) {
+ for (ParquetField field : meta.getFields()) {
+ if (field == null) {
+ continue;
+ }
+ if (field.parquetFieldType() != null) {
+ throw new HopException(
+ "Parquet output '"
+ + transformMeta.getName()
+ + "' cannot write Parquet type '"
+ + field.getParquetType()
+ + "' on Beam");
+ }
+ if (rowMeta.indexOfValue(field.getSourceFieldName()) < 0) {
+ throw new HopException(
+ "Field [" + field.getSourceFieldName() + "] couldn't be found in
the input stream!");
+ }
+ IValueMeta valueMeta =
+
rowMeta.getValueMeta(rowMeta.indexOfValue(field.getSourceFieldName()));
+ if (valueMeta.getType() == IValueMeta.TYPE_JSON
+ || valueMeta.getType() == IValueMeta.TYPE_UUID) {
+ String kind = valueMeta.getType() == IValueMeta.TYPE_JSON ? "JSON" :
"UUID";
+ throw new HopException(
+ "Parquet output '"
+ + transformMeta.getName()
+ + "' cannot write "
+ + kind
+ + " field '"
+ + field.getSourceFieldName()
+ + "' on Beam. On the local engine a "
+ + kind
+ + " field is a "
+ + kind
+ + " column, and ParquetIO would write it as a plain
string.");
+ }
+ ParquetField named = new ParquetField(field);
+ if (Utils.isEmpty(named.getTargetFieldName())) {
+ named.setTargetFieldName(named.getSourceFieldName());
+ }
+ fields.add(named);
+ valueMetas.add(valueMeta);
+ sourceFields.add(field.getSourceFieldName());
+ }
+ }
+ if (fields.isEmpty()) {
+ throw new HopException(
+ "Parquet output '" + transformMeta.getName() + "' has no output
fields");
+ }
+
+ Schema schema = ParquetOutput.avroSchema(fields, valueMetas);
+ String filename = outputFile(meta, variables);
+ createParent(meta, filename);
+ // Submission builds the graph before BeamPipelineEngine registers
FileSystems. Register here
+ // so a scheme that has a Beam filesystem is parsed by it. If that fails,
the split below
+ // keeps the scheme by cutting the string on the last '/'.
+ registerFileSystems(pipeline);
+ OutputLocation location = outputLocation(filename);
+ String directory = location.directory;
+ String onlyName = location.filename;
+
+ org.apache.beam.sdk.io.parquet.ParquetIO.Sink sink =
+ org.apache.beam.sdk.io.parquet.ParquetIO.sink(schema);
+ CompressionCodecName codec = meta.getCompressionCodec();
+ if (codec != null) {
+ sink = sink.withCompressionCodec(codec);
+ }
+ int rowGroup = Const.toInt(variables.resolve(meta.getRowGroupSize()), -1);
+ if (rowGroup > 0) {
+ sink = sink.withRowGroupSize(rowGroup);
+ }
+ int page = Const.toInt(variables.resolve(meta.getDataPageSize()), -1);
+ if (page > 0) {
+ sink = sink.withPageSize(page);
+ }
+
+ PCollection<GenericRecord> records =
+ input
+ .apply(
+ transformMeta.getName() + " To Parquet",
+ ParDo.of(
+ new HopRowToParquetFn(
+ transformMeta.getName(),
+ JsonRowMeta.toJson(rowMeta),
+ schema.toString(),
+ sourceFields)))
+ .setCoder(AvroCoder.of(schema));
+ records.apply(
+ transformMeta.getName() + " Write",
+ FileIO.<GenericRecord>write()
+ .via(sink)
+ .to(directory)
+ .withNumShards(1)
+ .withIgnoreWindowing()
+ .withTempDirectory(directory)
+ .withNaming(new FixedName(onlyName)));
+
+ transformCollectionMap.put(transformMeta.getName(), input);
+ log.logBasic(
+ "Handled transform (PARQUET OUTPUT) : " + transformMeta.getName() + ",
" + filename);
+ }
+
+ static String outputFile(ParquetOutputMeta meta, IVariables variables)
throws HopException {
+ if (Utils.isEmpty(meta.getFilenameBase())) {
+ throw new HopException("Parquet output has no filename");
+ }
+ String filename = variables.resolve(meta.getFilenameBase());
+ Date date = new Date();
+ if (meta.isFilenameIncludingDate()) {
+ filename += "-" + new SimpleDateFormat("yyyyMMdd").format(date);
+ }
+ if (meta.isFilenameIncludingTime()) {
+ filename += "-" + new SimpleDateFormat("HHmmss").format(date);
+ }
+ if (meta.isFilenameIncludingDateTime()) {
+ filename +=
+ "-"
+ + new
SimpleDateFormat(variables.resolve(meta.getFilenameDateTimeFormat()))
+ .format(date);
+ }
+ String extension =
Const.NVL(variables.resolve(meta.getFilenameExtension()), "parquet");
+ CompressionCodecName codec =
+ meta.getCompressionCodec() == null
+ ? CompressionCodecName.UNCOMPRESSED
+ : meta.getCompressionCodec();
+ String compressionExtension = codec.getExtension();
+ if (meta.isFilenameCompressionBeforeExtension()) {
+ filename += compressionExtension;
+ filename += "." + extension;
+ } else {
+ filename += "." + extension;
+ filename += compressionExtension;
+ }
+ return filename;
+ }
+
+ /**
+ * Directory passed to {@code FileIO.write().to} and {@code
withTempDirectory}, and the file name
+ * inside it.
+ *
+ * <p>{@code new File("gs://bucket/out/data.parquet").getParent()} is {@code
gs:/bucket/out}. Beam
+ * then writes to that directory. {@link FileSystems#matchNewResource}
parses the path when that
+ * scheme is registered. Otherwise the path is split on the last {@code
'/'}, which keeps the
+ * scheme's {@code //}. A {@code file://} URI is turned into its local path
first, because the
+ * local filesystem would resolve {@code file://} against the working
directory.
+ */
+ static OutputLocation outputLocation(String filename) {
+ String path = localPath(filename);
+ if (path == null) {
+ path = filename;
+ }
+ OutputLocation fromBeam = locationFromBeam(path);
+ if (fromBeam != null) {
+ return fromBeam;
+ }
+ return locationFromSlash(path);
+ }
+
+ /**
+ * A {@code file://} URI names a local file, but {@code
LocalFileSystem.matchNewResource} parses
+ * it with {@code Paths.get}, which drops a slash and resolves it against
the working directory.
+ * The local path is what {@code FileIO} can write.
+ */
+ private static String localPath(String filename) {
+ int colon = filename.indexOf(':');
+ if (!isFileScheme(filename, colon)) {
+ return null;
+ }
+ try {
+ URI uri = URI.create(filename);
+ try {
+ return Paths.get(uri).toString();
+ } catch (IllegalArgumentException authority) {
+ String path = uri.getPath();
+ if (path == null || path.isEmpty()) {
+ return null;
+ }
+ return path;
+ }
+ } catch (RuntimeException ignored) {
+ return null;
+ }
+ }
+
+ private static void registerFileSystems(Pipeline pipeline) {
+ if (pipeline == null) {
+ return;
+ }
+ try {
+ FileSystems.registerFileSystemsOnce(pipeline.getOptions());
+ } catch (RuntimeException ignored) {
+ // A registrar that cannot start must not stop the write. The string
split keeps the scheme.
+ }
+ }
+
+ private static OutputLocation locationFromBeam(String path) {
+ try {
+ ResourceId resource = FileSystems.matchNewResource(path, false);
+ String name = resource.getFilename();
+ if (name == null || name.isEmpty()) {
+ return null;
+ }
+ String directory = resource.getCurrentDirectory().toString();
+ if (!beamSplitKeepsThePath(path, directory, name)) {
+ return null;
+ }
+ return new OutputLocation(withoutTrailingSeparator(directory), name);
+ } catch (RuntimeException ignored) {
+ return null;
+ }
+ }
+
+ /** Reject a parse that dropped {@code ://} or turned a relative path into
an absolute one. */
+ private static boolean beamSplitKeepsThePath(String path, String directory,
String name) {
+ if (directory == null || directory.isEmpty()) {
+ return false;
+ }
+ int scheme = path.indexOf("://");
+ if (scheme >= 0 && !directory.startsWith(path.substring(0, scheme + 3))) {
+ return false;
+ }
+ int slash = lastSeparator(path);
+ String expectedName = slash < 0 ? path : path.substring(slash + 1);
+ if (!expectedName.equals(name)) {
+ return false;
+ }
+ boolean pathAbsolute = path.startsWith("/") || isWindowsDrive(path);
+ boolean directoryAbsolute = directory.startsWith("/") ||
isWindowsDrive(directory);
+ return pathAbsolute || !directoryAbsolute;
+ }
+
+ private static OutputLocation locationFromSlash(String path) {
+ int slash = lastSeparator(path);
+ if (slash < 0) {
+ return new OutputLocation(".", path);
+ }
+ if (slash == 0) {
+ return new OutputLocation("/", path.substring(1));
+ }
+ return new OutputLocation(path.substring(0, slash), path.substring(slash +
1));
+ }
+
+ private static int lastSeparator(String path) {
+ int slash = path.lastIndexOf('/');
+ if (path.contains("://")) {
+ return slash;
+ }
+ return Math.max(slash, path.lastIndexOf('\\'));
+ }
+
+ private static String withoutTrailingSeparator(String directory) {
+ if (directory.length() <= 1) {
+ return directory;
+ }
+ char last = directory.charAt(directory.length() - 1);
+ if (last != '/' && last != '\\') {
+ return directory;
+ }
+ String stripped = directory.substring(0, directory.length() - 1);
+ // Leave "gs://" and a Windows drive root "C:\" alone.
+ if (stripped.endsWith(":/") || stripped.endsWith(":") ||
stripped.isEmpty()) {
+ return directory;
+ }
+ return stripped;
+ }
+
+ private static boolean isWindowsDrive(String path) {
+ return path.length() >= 2 && path.charAt(1) == ':' &&
Character.isLetter(path.charAt(0));
+ }
+
+ /**
+ * A path with no scheme, a Windows drive, or a {@code file://} URI. {@code
gs://}, {@code s3://}
+ * and the other remote schemes are not.
+ */
+ static boolean isLocalPath(String filename) {
+ if (filename == null || filename.isEmpty()) {
+ return false;
+ }
+ int colon = filename.indexOf(':');
+ if (colon < 0) {
+ return true;
+ }
+ if (isWindowsDrive(filename)) {
+ return true;
+ }
+ return isFileScheme(filename, colon);
+ }
+
+ private static boolean isFileScheme(String filename, int colon) {
+ return colon == 4 && filename.regionMatches(true, 0, "file", 0, 4);
+ }
+
+ /** Directory {@code FileIO} writes into, and the single file name inside
it. */
+ static final class OutputLocation {
+ final String directory;
+ final String filename;
+
+ private OutputLocation(String directory, String filename) {
+ this.directory = directory;
+ this.filename = filename;
+ }
+ }
+
+ private static void createParent(ParquetOutputMeta meta, String filename)
throws HopException {
+ if (!meta.isFilenameCreatingParentFolders()) {
+ return;
+ }
+ // Object stores have no real folders. Creating the parent through Hop VFS
runs on the machine
+ // that submits the pipeline and needs that machine's cloud credentials.
Only a local path, or
+ // a file:// URI, gets a parent folder.
+ if (!isLocalPath(filename)) {
+ return;
+ }
+ FileObject parent = null;
+ try {
+ parent = HopVfs.getFileObject(filename).getParent();
+ if (parent != null && !parent.exists()) {
+ parent.createFolder();
+ }
+ } catch (Exception e) {
+ throw new HopException("Unable to prepare the parent folder for " +
filename, e);
+ } finally {
+ if (parent != null) {
+ try {
+ parent.close();
+ } catch (Exception ignored) {
+ // closing the parent is best-effort
+ }
+ }
+ }
+ }
+
+ /** One shard, and the name the local transform would use without a copy or
split number. */
+ private static final class FixedName implements FileIO.Write.FileNaming {
+ private final String name;
+
+ private FixedName(String name) {
+ this.name = name;
+ }
+
+ @Override
+ public String getFilename(
+ BoundedWindow window,
+ PaneInfo pane,
+ int numShards,
+ int shardIndex,
+ Compression compression) {
+ return name;
+ }
+ }
+}
diff --git
a/plugins/engines/beam/src/main/java/org/apache/hop/beam/util/BeamConst.java
b/plugins/engines/beam/src/main/java/org/apache/hop/beam/util/BeamConst.java
index dc33957a9d..e041851c62 100644
--- a/plugins/engines/beam/src/main/java/org/apache/hop/beam/util/BeamConst.java
+++ b/plugins/engines/beam/src/main/java/org/apache/hop/beam/util/BeamConst.java
@@ -47,6 +47,8 @@ public class BeamConst {
public static final String STRING_ADD_SEQUENCE_PLUGIN_ID = "Sequence";
public static final String STRING_AVRO_FILE_INPUT_PLUGIN_ID =
"AvroFileInput";
public static final String STRING_AVRO_OUTPUT_PLUGIN_ID = "AvroOutput";
+ public static final String STRING_PARQUET_FILE_INPUT_PLUGIN_ID =
"ParquetFileInput";
+ public static final String STRING_PARQUET_FILE_OUTPUT_PLUGIN_ID =
"ParquetFileOutput";
/** Mask fields. Banned by plugin id so the Beam engine does not have to
load the transform. */
public static final String STRING_MASK_FIELDS_PLUGIN_ID = "MaskFields";
diff --git a/plugins/engines/beam/src/main/resources/dependencies.xml
b/plugins/engines/beam/src/main/resources/dependencies.xml
index 5a382399bc..dda60f7abe 100644
--- a/plugins/engines/beam/src/main/resources/dependencies.xml
+++ b/plugins/engines/beam/src/main/resources/dependencies.xml
@@ -19,6 +19,7 @@
<dependencies>
<folder>../../transforms/addsequence</folder>
<folder>../../tech/avro</folder>
+ <folder>../../tech/parquet</folder>
<folder>../../transforms/constant</folder>
<folder>../../transforms/filterrows</folder>
<folder>../../transforms/groupby</folder>
diff --git
a/plugins/engines/beam/src/test/java/org/apache/hop/beam/pipeline/handler/BeamParquetFileIOTest.java
b/plugins/engines/beam/src/test/java/org/apache/hop/beam/pipeline/handler/BeamParquetFileIOTest.java
new file mode 100644
index 0000000000..e224329e0c
--- /dev/null
+++
b/plugins/engines/beam/src/test/java/org/apache/hop/beam/pipeline/handler/BeamParquetFileIOTest.java
@@ -0,0 +1,672 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.beam.pipeline.handler;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricNameFilter;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.MetricResult;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hop.beam.core.HopRow;
+import org.apache.hop.beam.core.coder.HopRowCoder;
+import org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter;
+import org.apache.hop.beam.util.BeamConst;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.logging.LogChannel;
+import org.apache.hop.core.plugins.PluginRegistry;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaBase;
+import org.apache.hop.core.row.value.ValueMetaBigNumber;
+import org.apache.hop.core.row.value.ValueMetaBinary;
+import org.apache.hop.core.row.value.ValueMetaDate;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaJson;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.row.value.ValueMetaTimestamp;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.parquet.transforms.input.ParquetField;
+import org.apache.hop.parquet.transforms.input.ParquetInputMeta;
+import org.apache.hop.parquet.transforms.output.ParquetOutput;
+import org.apache.hop.parquet.transforms.output.ParquetOutputMeta;
+import org.apache.hop.parquet.transforms.output.ParquetPartitionField;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.io.LocalInputFile;
+import org.joda.time.Duration;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/** Issue #2357: Parquet file input and output on the Beam Direct runner. */
+class BeamParquetFileIOTest {
+
+ private static final List<String> SEEN = Collections.synchronizedList(new
ArrayList<>());
+ private static final List<Object[]> SEEN_ROWS =
Collections.synchronizedList(new ArrayList<>());
+ private static PipelineResult lastRead;
+
+ private static final Date WHEN = new Date(1_700_000_000_000L);
+ private static final byte[] PAYLOAD = new byte[] {9, 8, 7};
+ private static final BigDecimal AMOUNT = new BigDecimal("-12.34");
+
+ @TempDir Path tempDir;
+
+ @BeforeAll
+ static void initHopEnvironment() throws Exception {
+ HopEnvironment.init();
+ PluginRegistry.init();
+ }
+
+ @Test
+ void parquetPluginIdsAreExplicitBeamHandlers() {
+ assertTrue(
+
HopPipelineMetaToBeamPipelineConverter.EXPLICIT_HANDLER_PLUGIN_IDS.contains(
+ BeamConst.STRING_PARQUET_FILE_OUTPUT_PLUGIN_ID));
+ assertTrue(
+
HopPipelineMetaToBeamPipelineConverter.EXPLICIT_HANDLER_PLUGIN_IDS.contains(
+ BeamConst.STRING_PARQUET_FILE_INPUT_PLUGIN_ID));
+ }
+
+ @Test
+ void avroSchemaIsTheHopRecord() throws Exception {
+ ParquetOutputMeta meta = fileMeta(tempDir.resolve("schema.parquet"));
+ List<org.apache.hop.parquet.transforms.output.ParquetField> fields =
meta.getFields();
+ List<IValueMeta> types = List.of(new ValueMetaString("name"), new
ValueMetaInteger("id"));
+ Schema schema = ParquetOutput.avroSchema(fields, types);
+ assertEquals("ApacheHopParquetSchema", schema.getName());
+ assertNotNull(schema.getField("name"));
+ assertNotNull(schema.getField("id"));
+ assertEquals(Schema.Type.UNION, schema.getField("id").schema().getType());
+ }
+
+ @Test
+ void writeThenReadKeepsTheIncomingRow() throws Exception {
+ Path file = tempDir.resolve("rows.parquet");
+ writeTwoRows(file);
+ List<String> seen = new
ArrayList<>(readBack(file.toAbsolutePath().toString(), stringFields()));
+ Collections.sort(seen);
+ assertEquals(List.of("keep-me|alice|1", "keep-me|bob|2"), seen);
+ }
+
+ @Test
+ void remoteOutputDirectoryKeepsTheScheme() {
+ BeamParquetOutputTransformHandler.OutputLocation gs =
+
BeamParquetOutputTransformHandler.outputLocation("gs://bucket/out/data.parquet");
+ assertEquals("gs://bucket/out", gs.directory);
+ assertEquals("data.parquet", gs.filename);
+
+ BeamParquetOutputTransformHandler.OutputLocation s3 =
+
BeamParquetOutputTransformHandler.outputLocation("s3://bkt/x/y.parquet");
+ assertEquals("s3://bkt/x", s3.directory);
+ assertEquals("y.parquet", s3.filename);
+
+ BeamParquetOutputTransformHandler.OutputLocation file =
+
BeamParquetOutputTransformHandler.outputLocation("file:///tmp/out/data.parquet");
+ assertEquals("/tmp/out", file.directory);
+ assertEquals("data.parquet", file.filename);
+
+
assertTrue(BeamParquetOutputTransformHandler.isLocalPath("/tmp/out/data.parquet"));
+
assertTrue(BeamParquetOutputTransformHandler.isLocalPath("out/data.parquet"));
+
assertTrue(BeamParquetOutputTransformHandler.isLocalPath("file:///tmp/out/data.parquet"));
+
assertFalse(BeamParquetOutputTransformHandler.isLocalPath("gs://bucket/out/data.parquet"));
+
assertFalse(BeamParquetOutputTransformHandler.isLocalPath("s3://bkt/x/y.parquet"));
+ }
+
+ @Test
+ void fileUriIsReadBack() throws Exception {
+ Path file = tempDir.resolve("uri.parquet");
+ writeTwoRows(file);
+ String uri = file.toUri().toString();
+ assertTrue(uri.startsWith("file:"), uri);
+ List<String> seen = new ArrayList<>(readBack(uri, stringFields()));
+ Collections.sort(seen);
+ assertEquals(List.of("keep-me|alice|1", "keep-me|bob|2"), seen);
+ assertEquals(2, counter(lastRead,
org.apache.hop.pipeline.Pipeline.METRIC_NAME_READ, "read"));
+ assertEquals(
+ 2, counter(lastRead,
org.apache.hop.pipeline.Pipeline.METRIC_NAME_WRITTEN, "read"));
+ assertEquals(0, counter(lastRead,
org.apache.hop.pipeline.Pipeline.METRIC_NAME_OUTPUT, "read"));
+ }
+
+ @Test
+ void fileUriIsWrittenAndReadBack() throws Exception {
+ Path file = tempDir.resolve("from-uri.parquet");
+ String uri = file.toUri().toString();
+ assertTrue(uri.startsWith("file:"), uri);
+ ParquetOutputMeta meta = fileMeta(file);
+ String base =
+ uri.endsWith(".parquet") ? uri.substring(0, uri.length() -
".parquet".length()) : uri;
+ meta.setFilenameBase(base);
+
+ Pipeline pipeline = Pipeline.create();
+ writeGraph(meta, pipeline);
+ pipeline.run().waitUntilFinish();
+
+ assertTrue(Files.exists(file), uri);
+ List<String> seen = new ArrayList<>(readBack(uri, stringFields()));
+ Collections.sort(seen);
+ assertEquals(List.of("keep-me|alice|1", "keep-me|bob|2"), seen);
+ }
+
+ @Test
+ void dateTimestampDecimalAndBinaryRoundTrip() throws Exception {
+ Path file = tempDir.resolve("typed.parquet");
+ Timestamp after = new Timestamp(1_000L);
+ after.setNanos(123_456_789);
+ Timestamp before = new Timestamp(-1_000L);
+ before.setNanos(999_999_000);
+ Pipeline pipeline = Pipeline.create();
+ writeGraph(
+ typedMeta(file),
+ pipeline,
+ typedRow(),
+ List.of(
+ new HopRow(new Object[] {"alice", 1L, WHEN, after, AMOUNT,
PAYLOAD}),
+ new HopRow(
+ new Object[] {
+ "bob", 2L, new Date(0L), before, new BigDecimal("0.01"), new
byte[] {1}
+ })));
+ pipeline.run().waitUntilFinish();
+
+ try (ParquetReader<GenericRecord> reader =
+ AvroParquetReader.<GenericRecord>builder(new LocalInputFile(file))
+ .withConf(new Configuration(false))
+ .build()) {
+ GenericRecord first = reader.read();
+ GenericRecord second = reader.read();
+ assertNotNull(first);
+ assertNotNull(second);
+ GenericRecord alice = "alice".equals(String.valueOf(first.get("name")))
? first : second;
+ GenericRecord bob = alice == first ? second : first;
+ assertEquals(1_123_456L, storedEpoch(alice.get("ts"), true));
+ assertEquals(-1L, storedEpoch(bob.get("ts"), true));
+ assertEquals(WHEN.getTime(), storedEpoch(alice.get("when"), false));
+ assertEquals(0, AMOUNT.compareTo(storedDecimal(alice.get("amount"), 2)));
+ assertArrayEquals(PAYLOAD, storedBytes(alice.get("payload")));
+ }
+
+ readRows(file.toAbsolutePath().toString(), typedInputFields(),
typedFileRow());
+ assertEquals(2, SEEN_ROWS.size());
+ Object[] alice = rowNamed("alice");
+ Object[] bob = rowNamed("bob");
+ assertEquals("keep-me", alice[1]);
+ assertEquals(1L, alice[3]);
+ assertInstanceOf(Long.class, alice[3]);
+ assertInstanceOf(Date.class, alice[4]);
+ assertFalse(alice[4] instanceof Timestamp);
+ assertEquals(WHEN, alice[4]);
+ assertEquals(Timestamp.from(Instant.ofEpochSecond(1, 123_456_000)),
alice[5]);
+ assertEquals(AMOUNT, alice[6]);
+ assertArrayEquals(PAYLOAD, (byte[]) alice[7]);
+ assertEquals(Timestamp.from(Instant.ofEpochSecond(-1, 999_999_000)),
bob[5]);
+ assertEquals(new BigDecimal("0.01"), bob[6]);
+ assertArrayEquals(new byte[] {1}, (byte[]) bob[7]);
+ }
+
+ @Test
+ void partitionFieldIsRefused() {
+ ParquetOutputMeta meta = fileMeta(tempDir.resolve("part.parquet"));
+ meta.setPartitionFields(List.of(new ParquetPartitionField("region")));
+ HopException refused =
+ assertThrows(HopException.class, () -> writeGraph(meta,
Pipeline.create()));
+ assertTrue(refused.getMessage().contains("partition"));
+ }
+
+ @Test
+ void explicitParquetTypeIsRefused() {
+ ParquetOutputMeta meta =
fileMeta(tempDir.resolve("typed-refused.parquet"));
+ meta.getFields().get(0).setParquetType("JSON");
+ HopException refused =
+ assertThrows(HopException.class, () -> writeGraph(meta,
Pipeline.create()));
+ assertTrue(refused.getMessage().contains("JSON"));
+ }
+
+ @Test
+ void jsonAndUuidFieldsAreRefused() {
+ ParquetOutputMeta jsonMeta = fileMeta(tempDir.resolve("json.parquet"));
+ jsonMeta.setFields(
+ List.of(new
org.apache.hop.parquet.transforms.output.ParquetField("doc", "doc")));
+ RowMeta jsonRow = new RowMeta();
+ jsonRow.addValueMeta(new ValueMetaJson("doc"));
+ HopException json =
+ assertThrows(
+ HopException.class,
+ () ->
+ writeGraph(
+ jsonMeta,
+ Pipeline.create(),
+ jsonRow,
+ List.of(new HopRow(new Object[] {"{}"}))));
+ assertTrue(json.getMessage().contains("JSON"), json.getMessage());
+
+ ParquetOutputMeta uuidMeta = fileMeta(tempDir.resolve("uuid.parquet"));
+ uuidMeta.setFields(
+ List.of(new
org.apache.hop.parquet.transforms.output.ParquetField("id", "id")));
+ RowMeta uuidRow = new RowMeta();
+ uuidRow.addValueMeta(new UuidValueMeta("id"));
+ HopException uuid =
+ assertThrows(
+ HopException.class,
+ () ->
+ writeGraph(
+ uuidMeta,
+ Pipeline.create(),
+ uuidRow,
+ List.of(new HopRow(new Object[]
{"00112233-4455-6677-8899-aabbccddeeff"}))));
+ assertTrue(uuid.getMessage().contains("UUID"), uuid.getMessage());
+ }
+
+ @Test
+ void inputWithoutFieldsOrMetadataFileIsRefused() {
+ ParquetInputMeta meta = new ParquetInputMeta();
+ meta.setFilenameField("file");
+ HopException missing =
+ assertThrows(
+ HopException.class,
+ () -> {
+ Pipeline pipeline = Pipeline.create();
+ PCollection<HopRow> rows =
+ pipeline.apply(
+ Create.of(new HopRow(new Object[] {"x"})).withCoder(new
HopRowCoder()));
+ new BeamParquetInputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileInput", "read", meta),
+ new HashMap<>(),
+ pipeline,
+ fileRow(),
+ List.of(upstream()),
+ rows,
+ null);
+ });
+ assertTrue(missing.getMessage().contains("metadata"));
+ }
+
+ @Test
+ void nullRowWhenEmptyIsAcceptedInTheGlobalWindow() throws Exception {
+ ParquetInputMeta meta = inputMeta(stringFields());
+ meta.setSendingNullsRowWhenEmpty(true);
+ Pipeline pipeline = Pipeline.create();
+ PCollection<HopRow> rows =
+ pipeline.apply(
+ Create.of(new HopRow(new Object[] {"x", "keep"})).withCoder(new
HopRowCoder()));
+ new BeamParquetInputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileInput", "read", meta),
+ new HashMap<>(),
+ pipeline,
+ fileRow(),
+ List.of(upstream()),
+ rows,
+ null);
+ }
+
+ @Test
+ void nullRowWhenEmptyRefusesAWindowedInput() {
+ ParquetInputMeta meta = inputMeta(stringFields());
+ meta.setSendingNullsRowWhenEmpty(true);
+ Pipeline pipeline = Pipeline.create();
+ PCollection<HopRow> rows =
+ pipeline
+ .apply(Create.of(new HopRow(new Object[] {"x",
"keep"})).withCoder(new HopRowCoder()))
+ .apply(Window.into(FixedWindows.of(Duration.standardMinutes(1))));
+ HopException refused =
+ assertThrows(
+ HopException.class,
+ () ->
+ new BeamParquetInputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileInput", "read", meta),
+ new HashMap<>(),
+ pipeline,
+ fileRow(),
+ List.of(upstream()),
+ rows,
+ null));
+ assertTrue(refused.getMessage().contains("global window"),
refused.getMessage());
+ }
+
+ private void writeTwoRows(Path file) throws Exception {
+ Pipeline pipeline = Pipeline.create();
+ writeGraph(fileMeta(file), pipeline);
+ pipeline.run().waitUntilFinish();
+ }
+
+ private static void writeGraph(ParquetOutputMeta meta, Pipeline pipeline)
throws Exception {
+ writeGraph(
+ meta,
+ pipeline,
+ valueRow(),
+ List.of(new HopRow(new Object[] {"alice", 1L}), new HopRow(new
Object[] {"bob", 2L})));
+ }
+
+ private static void writeGraph(
+ ParquetOutputMeta meta, Pipeline pipeline, IRowMeta rowMeta,
List<HopRow> rows)
+ throws Exception {
+ new BeamParquetOutputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileOutput", "write", meta),
+ new HashMap<>(),
+ pipeline,
+ rowMeta,
+ List.of(upstream()),
+ pipeline.apply(Create.of(rows).withCoder(new HopRowCoder())),
+ null);
+ }
+
+ private static List<String> readBack(String filename, List<ParquetField>
fields)
+ throws Exception {
+ SEEN.clear();
+ Pipeline pipeline = Pipeline.create();
+ PCollection<HopRow> rows =
+ pipeline.apply(
+ Create.of(new HopRow(new Object[] {filename,
"keep-me"})).withCoder(new HopRowCoder()));
+ HashMap<String, PCollection<HopRow>> map = new HashMap<>();
+ new BeamParquetInputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileInput", "read", inputMeta(fields)),
+ map,
+ pipeline,
+ fileRow(),
+ List.of(upstream()),
+ rows,
+ null);
+ map.get("read").apply(ParDo.of(new CollectRecord()));
+ PipelineResult result = pipeline.run();
+ result.waitUntilFinish();
+ lastRead = result;
+ return new ArrayList<>(SEEN);
+ }
+
+ private static void readRows(String filename, List<ParquetField> fields,
IRowMeta rowMeta)
+ throws Exception {
+ SEEN_ROWS.clear();
+ Pipeline pipeline = Pipeline.create();
+ PCollection<HopRow> rows =
+ pipeline.apply(
+ Create.of(new HopRow(new Object[] {filename,
"keep-me"})).withCoder(new HopRowCoder()));
+ HashMap<String, PCollection<HopRow>> map = new HashMap<>();
+ new BeamParquetInputTransformHandler()
+ .handleTransform(
+ LogChannel.GENERAL,
+ new Variables(),
+ "direct",
+ null,
+ null,
+ new MemoryMetadataProvider(),
+ new PipelineMeta(),
+ new TransformMeta("ParquetFileInput", "read", inputMeta(fields)),
+ map,
+ pipeline,
+ rowMeta,
+ List.of(upstream()),
+ rows,
+ null);
+ map.get("read").apply(ParDo.of(new CollectRow()));
+ pipeline.run().waitUntilFinish();
+ }
+
+ private static Object[] rowNamed(String name) {
+ for (Object[] row : SEEN_ROWS) {
+ if (name.equals(row[2])) {
+ return row;
+ }
+ }
+ throw new AssertionError("missing row " + name);
+ }
+
+ private static long storedEpoch(Object value, boolean micros) {
+ if (value instanceof Instant instant) {
+ if (micros) {
+ return instant.getEpochSecond() * 1_000_000L + instant.getNano() /
1_000L;
+ }
+ return instant.toEpochMilli();
+ }
+ if (value instanceof Number number) {
+ return number.longValue();
+ }
+ throw new AssertionError(value.getClass().getName());
+ }
+
+ private static BigDecimal storedDecimal(Object value, int scale) {
+ if (value instanceof BigDecimal decimal) {
+ return decimal;
+ }
+ return new BigDecimal(new BigInteger(storedBytes(value)), scale);
+ }
+
+ private static byte[] storedBytes(Object value) {
+ if (value instanceof byte[] bytes) {
+ return bytes;
+ }
+ if (value instanceof ByteBuffer buffer) {
+ ByteBuffer duplicate = buffer.duplicate();
+ byte[] bytes = new byte[duplicate.remaining()];
+ duplicate.get(bytes);
+ return bytes;
+ }
+ throw new AssertionError(value.getClass().getName());
+ }
+
+ private static long counter(PipelineResult result, String namespace, String
name) {
+ MetricQueryResults results =
+ result
+ .metrics()
+ .queryMetrics(
+ MetricsFilter.builder()
+ .addNameFilter(MetricNameFilter.named(namespace, name))
+ .build());
+ long count = 0;
+ for (MetricResult<Long> metric : results.getCounters()) {
+ count += metric.getAttempted();
+ }
+ return count;
+ }
+
+ static class CollectRecord extends DoFn<HopRow, Void> {
+ @ProcessElement
+ public void process(@Element HopRow row) {
+ Object[] values = row.getRow();
+ SEEN.add(values[1] + "|" + values[2] + "|" + values[3]);
+ }
+ }
+
+ static class CollectRow extends DoFn<HopRow, Void> {
+ @ProcessElement
+ public void process(@Element HopRow row) {
+ SEEN_ROWS.add(row.getRow().clone());
+ }
+ }
+
+ /** Stands in for the UUID value type plugin, which is not on this test
classpath. */
+ private static final class UuidValueMeta extends ValueMetaBase {
+ private UuidValueMeta(String name) {
+ super(name, IValueMeta.TYPE_UUID);
+ }
+ }
+
+ private static ParquetOutputMeta fileMeta(Path file) {
+ ParquetOutputMeta meta = new ParquetOutputMeta();
+ applyFile(meta, file);
+ meta.setFields(
+ List.of(
+ new org.apache.hop.parquet.transforms.output.ParquetField("name",
"name"),
+ new org.apache.hop.parquet.transforms.output.ParquetField("id",
"id")));
+ return meta;
+ }
+
+ private static ParquetOutputMeta typedMeta(Path file) {
+ ParquetOutputMeta meta = new ParquetOutputMeta();
+ applyFile(meta, file);
+ meta.setFields(
+ List.of(
+ new org.apache.hop.parquet.transforms.output.ParquetField("name",
"name"),
+ new org.apache.hop.parquet.transforms.output.ParquetField("id",
"id"),
+ new org.apache.hop.parquet.transforms.output.ParquetField("when",
"when"),
+ new org.apache.hop.parquet.transforms.output.ParquetField("ts",
"ts"),
+ new
org.apache.hop.parquet.transforms.output.ParquetField("amount", "amount"),
+ new
org.apache.hop.parquet.transforms.output.ParquetField("payload", "payload")));
+ return meta;
+ }
+
+ private static void applyFile(ParquetOutputMeta meta, Path file) {
+ String path = file.toAbsolutePath().toString();
+ String base =
+ path.endsWith(".parquet") ? path.substring(0, path.length() -
".parquet".length()) : path;
+ meta.setFilenameBase(base);
+ meta.setFilenameExtension("parquet");
+ meta.setFilenameIncludingCopyNr(false);
+ meta.setFilenameIncludingSplitNr(false);
+ meta.setFilenameIncludingDate(false);
+ meta.setFilenameIncludingTime(false);
+ meta.setFilenameIncludingDateTime(false);
+ meta.setFilenameCreatingParentFolders(true);
+ }
+
+ private static List<ParquetField> stringFields() {
+ return List.of(
+ new ParquetField("name", "name", "String", null, null, null),
+ new ParquetField("id", "id", "Integer", null, null, null));
+ }
+
+ private static List<ParquetField> typedInputFields() {
+ return List.of(
+ new ParquetField("name", "name", "String", null, null, null),
+ new ParquetField("id", "id", "Integer", null, null, null),
+ new ParquetField("when", "when", "Date", null, null, null),
+ new ParquetField("ts", "ts", "Timestamp", null, null, null),
+ new ParquetField("amount", "amount", "BigNumber", null, "10", "2"),
+ new ParquetField("payload", "payload", "Binary", null, null, null));
+ }
+
+ private static ParquetInputMeta inputMeta(List<ParquetField> fields) {
+ ParquetInputMeta meta = new ParquetInputMeta();
+ meta.setFilenameField("file");
+ meta.setFields(fields);
+ return meta;
+ }
+
+ private static TransformMeta upstream() {
+ TransformMeta upstream = new TransformMeta();
+ upstream.setName("upstream");
+ return upstream;
+ }
+
+ private static IRowMeta valueRow() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("name"));
+ rowMeta.addValueMeta(new ValueMetaInteger("id"));
+ return rowMeta;
+ }
+
+ private static IRowMeta typedRow() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("name"));
+ rowMeta.addValueMeta(new ValueMetaInteger("id"));
+ rowMeta.addValueMeta(new ValueMetaDate("when"));
+ rowMeta.addValueMeta(new ValueMetaTimestamp("ts"));
+ ValueMetaBigNumber amount = new ValueMetaBigNumber("amount");
+ amount.setLength(10);
+ amount.setPrecision(2);
+ rowMeta.addValueMeta(amount);
+ rowMeta.addValueMeta(new ValueMetaBinary("payload"));
+ return rowMeta;
+ }
+
+ private static IRowMeta fileRow() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("file"));
+ rowMeta.addValueMeta(new ValueMetaString("tag"));
+ return rowMeta;
+ }
+
+ private static IRowMeta typedFileRow() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("file"));
+ rowMeta.addValueMeta(new ValueMetaString("tag"));
+ return rowMeta;
+ }
+}
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutput.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutput.java
index 541d36f5d7..eb5cffdb61 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutput.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutput.java
@@ -251,28 +251,38 @@ public class ParquetOutput extends
BaseTransform<ParquetOutputMeta, ParquetOutpu
* every split or partition file.
*/
private MessageType buildSchema() throws HopException {
+ List<IValueMeta> valueMetas = new ArrayList<>();
+ for (int i = 0; i < data.outputFields.size(); i++) {
+
valueMetas.add(getInputRowMeta().getValueMeta(data.sourceFieldIndexes.get(i)));
+ }
+ // Convert from Avro to Parquet schema, then apply a Parquet type selected
on a field.
+ //
+ return applySelectedTypes(
+ withParquetOnlyTypes(
+ new AvroSchemaConverter().convert(avroSchema(data.outputFields,
valueMetas))));
+ }
+
+ /**
+ * The Avro record {@link #buildSchema()} converts to Parquet. Every field
is a nullable union.
+ * Beam Parquet output uses this schema directly and does not apply a
selected Parquet type.
+ */
+ public static Schema avroSchema(List<ParquetField> fields, List<IValueMeta>
valueMetas)
+ throws HopException {
SchemaBuilder.FieldAssembler<Schema> fieldAssembler =
SchemaBuilder.record("ApacheHopParquetSchema").fields();
-
- // Build the Parquet Schema: every field is a nullable union of the Avro
type for the Hop type.
- for (int i = 0; i < data.outputFields.size(); i++) {
- ParquetField field = data.outputFields.get(i);
- IValueMeta valueMeta =
getInputRowMeta().getValueMeta(data.sourceFieldIndexes.get(i));
+ for (int i = 0; i < fields.size(); i++) {
fieldAssembler =
fieldAssembler
- .name(field.getTargetFieldName())
+ .name(fields.get(i).getTargetFieldName())
.type()
.unionOf()
.nullType()
.and()
- .type(avroType(valueMeta))
+ .type(avroType(valueMetas.get(i)))
.endUnion()
.noDefault();
}
- // Convert from Avro to Parquet schema, then apply a Parquet type selected
on a field.
- //
- return applySelectedTypes(
- withParquetOnlyTypes(new
AvroSchemaConverter().convert(fieldAssembler.endRecord())));
+ return fieldAssembler.endRecord();
}
/**
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteSupport.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteSupport.java
index cca10a0f78..1b7ef59d6c 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteSupport.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteSupport.java
@@ -155,9 +155,10 @@ public class ParquetWriteSupport extends
WriteSupport<RowMetaAndData> {
/**
* A date or timestamp as the number the column holds: microseconds for a
TIMESTAMP(MICROS)
* column, milliseconds otherwise. The value meta takes care of lazy (binary
string) storage and
- * the date mask.
+ * the date mask. Beam Parquet output calls this so both writers store the
same instant.
*/
- static long epochValue(IValueMeta valueMeta, Object valueData,
LogicalTypeAnnotation logicalType)
+ public static long epochValue(
+ IValueMeta valueMeta, Object valueData, LogicalTypeAnnotation
logicalType)
throws HopException {
if (logicalType instanceof TimestampLogicalTypeAnnotation timestamp
&& timestamp.getUnit() == TimeUnit.MICROS) {
@@ -173,9 +174,10 @@ public class ParquetWriteSupport extends
WriteSupport<RowMetaAndData> {
/**
* A big number as the unscaled two's complement bytes of a DECIMAL column,
rounded to the scale
- * of the column the way the field rounds.
+ * of the column the way the field rounds. Beam Parquet output calls this so
both writers store
+ * the same bytes.
*/
- static Binary decimalBytes(
+ public static Binary decimalBytes(
String fieldName,
IValueMeta valueMeta,
BigDecimal value,