This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 06c1f666c3 Issue #8298 : Put Parquet Output options on four tabs so
the dialog fits 1080p (#8300)
06c1f666c3 is described below
commit 06c1f666c3a4bec16dcc572f26ac84d99deb4b9d
Author: Matt Casters <[email protected]>
AuthorDate: Wed Sep 9 20:48:01 2026 +0200
Issue #8298 : Put Parquet Output options on four tabs so the dialog fits
1080p (#8300)
The File, Options, Partitioning, and Fields tabs use GuiCompositeWidgets
with a scrolled composite so the shell stays within a 1080p window at
100% zoom. Partitioning behaviour is unchanged; only the layout is new.
---
.../parquet/parquet-output-dialog-fields-tab.png | Bin 0 -> 82138 bytes
.../parquet/parquet-output-dialog-file-tab.png | Bin 0 -> 124604 bytes
.../parquet/parquet-output-dialog-options-tab.png | Bin 0 -> 103927 bytes
.../parquet-output-dialog-partitioning-tab.png | Bin 0 -> 94142 bytes
.../pipeline/transforms/parquet-file-output.adoc | 58 +-
.../transforms/output/ParquetOutputDialog.java | 779 +++++++--------------
.../transforms/output/ParquetOutputMeta.java | 551 +++++++--------
.../transforms/output/ParquetTypeFilename.java | 42 ++
.../parquet/transforms/output/ParquetVersion.java | 7 +-
.../transforms/output/ParquetWriteMode.java | 7 +-
.../output/messages/messages_en_US.properties | 22 +
.../transforms/output/ParquetOutputMetaTest.java | 19 +
.../transforms/output/ParquetVersionTest.java | 6 +
.../transforms/output/ParquetWriteModeTest.java | 44 ++
14 files changed, 683 insertions(+), 852 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-fields-tab.png
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-fields-tab.png
new file mode 100644
index 0000000000..3e60ce13d5
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-fields-tab.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-file-tab.png
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-file-tab.png
new file mode 100644
index 0000000000..0f00ae648f
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-file-tab.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-options-tab.png
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-options-tab.png
new file mode 100644
index 0000000000..3679b650cc
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-options-tab.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-partitioning-tab.png
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-partitioning-tab.png
new file mode 100644
index 0000000000..59486e2522
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/tech/parquet/parquet-output-dialog-partitioning-tab.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
index 1eccd4e01b..1e3480401c 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/parquet-file-output.adoc
@@ -44,6 +44,8 @@ For more information on this see:
http://parquet.apache.org/[Apache Parquet].
== Options
+The dialog is split across four tabs so it fits a 1080p screen at 100% zoom:
**File**, **Options**, **Partitioning**, and **Fields**.
+
Notes:
* The date optionally referenced in the output file name(s) will be the start
of the pipeline execution.
@@ -53,6 +55,10 @@ Notes:
This happens when all rows are written.
To not run out of memory make sure to specify a split size.
+=== File
+
+image:tech/parquet/parquet-output-dialog-file-tab.png[Parquet file output File
tab, width="90%"]
+
[options="header"]
|===
|Option|Description
@@ -71,7 +77,7 @@ Write to a local folder : `/my/folder/customer-data`
|Extension
|This is the extension of the file.
-Usually this is simply `snappy`
+Usually this is simply `parquet`
|Include date?
|Check this box if you want to include the date in the filename with mask
`yyyMMdd`
@@ -92,14 +98,26 @@ Specify a split size larger than 0 and this is then the
number of rows per file.
The file part (split) number will be included in the filename to make sure
that the same file is not being overwritten.
The split number is formatted with mask `0000`
+|Create parent folders?
+|Create missing parent folders of the output path.
+
|Include compression codec before extension?
|When enabled (the default for new transforms), the compression codec is
placed before the file extension (for example `file.snappy.parquet`), which
matches the naming used by Spark and other Parquet tools.
When disabled, the compression codec is appended after the extension (for
example `file.parquet.snappy`).
Existing pipelines that do not have this option set keep the previous behavior.
+|===
+
+=== Options
+
+image:tech/parquet/parquet-output-dialog-options-tab.png[Parquet file output
Options tab, width="90%"]
+
+[options="header"]
+|===
+|Option|Description
|Compression codec
|Here you can indicate which compression codec you want to use.
-The default is SNAPPY for Apache Snappy compression.
+The default is UNCOMPRESSED.
|Version
|Choose the protocol version of Parquet (1.0 or 2.0)
@@ -112,9 +130,45 @@ The default is SNAPPY for Apache Snappy compression.
|Dictionary page size
|The data dictionary page size on a 1kB boundary (default is 1048576)
+|===
+
+=== Partitioning
+
+image:tech/parquet/parquet-output-dialog-partitioning-tab.png[Parquet file
output Partitioning tab, width="90%"]
+
+Leave **Partition by** empty to write a single file set, as before.
+
+When one or more fields are listed, the transform writes Hive-style
`name=value` folders under the base file name, and those fields are not stored
inside the Parquet files.
+
+[options="header"]
+|===
+|Option|Description
+
+|Existing data
+|What to do when a partition folder already exists: append, overwrite that
partition, fail if it exists, or overwrite the whole base folder.
+This option is enabled when at least one partition field is set.
+
+|Maximum open partitions
+|How many partition files may be written at the same time.
+Each open file buffers up to one row group in memory, so a lower number uses
less memory but writes more files.
+This option is enabled when at least one partition field is set.
+
+|Partition by
+|Incoming fields used as directory levels.
+Leave empty to write a single file set.
+|===
+
+=== Fields
+
+image:tech/parquet/parquet-output-dialog-fields-tab.png[Parquet file output
Fields tab, width="90%"]
+
+[options="header"]
+|===
+|Option|Description
|Fields
|You can specify which fields to write and in which order.
You can use the "Get Fields" button to populate the dialog.
+Leave empty to output all input fields.
|===
\ No newline at end of file
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputDialog.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputDialog.java
index 47b21e7e4f..7f3bc5cfc3 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputDialog.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputDialog.java
@@ -17,30 +17,33 @@
package org.apache.hop.parquet.transforms.output;
+import java.util.ArrayList;
+import java.util.List;
import org.apache.hop.core.Const;
import org.apache.hop.core.logging.LogChannel;
import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
import org.apache.hop.ui.core.dialog.ErrorDialog;
-import org.apache.hop.ui.core.gui.WindowProperty;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgetsAdapter;
import org.apache.hop.ui.core.widget.ColumnInfo;
+import org.apache.hop.ui.core.widget.ComboVar;
import org.apache.hop.ui.core.widget.NamingSchemeTypes;
import org.apache.hop.ui.core.widget.TableView;
import org.apache.hop.ui.core.widget.TextVar;
import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
-import org.apache.parquet.hadoop.metadata.CompressionCodecName;
import org.eclipse.swt.SWT;
import org.eclipse.swt.layout.FormAttachment;
import org.eclipse.swt.layout.FormData;
-import org.eclipse.swt.layout.FormLayout;
import org.eclipse.swt.widgets.Button;
import org.eclipse.swt.widgets.Combo;
+import org.eclipse.swt.widgets.Composite;
import org.eclipse.swt.widgets.Control;
-import org.eclipse.swt.widgets.Group;
import org.eclipse.swt.widgets.Label;
import org.eclipse.swt.widgets.Shell;
import org.eclipse.swt.widgets.TableItem;
@@ -49,34 +52,10 @@ public class ParquetOutputDialog extends
BaseTransformDialog {
public static final Class<?> PKG = ParquetOutputMeta.class;
- protected ParquetOutputMeta input;
-
- private TextVar wFilenameBase;
- private TextVar wFilenameExtension;
- private Button wFilenameIncludeDate;
- private Button wFilenameIncludeTime;
- private Button wFilenameIncludeDateTime;
- private Label wlFilenameDateTimeFormat;
- private TextVar wFilenameDateTimeFormat;
- private Button wFilenameIncludeCopyNr;
- private Button wFilenameIncludeSplitNr;
- private Label wlFilenameSplitSize;
- private TextVar wFilenameSplitSize;
- private Button wFilenameCreateFolders;
- private Button wFilenameCompressionBeforeExtension;
- private Combo wCompressionCodec;
- private Combo wVersion;
- private TextVar wRowGroupSize;
- private TextVar wDataPageSize;
- private TextVar wDictionaryPageSize;
+ private final ParquetOutputMeta input;
+ private GuiCompositeWidgets widgets;
private TableView wFields;
private TableView wPartitionFields;
- private Combo wWriteMode;
- private TextVar wMaxOpenPartitions;
- private Label wlWriteMode;
- private Label wlMaxOpenPartitions;
-
- private String returnValue;
public ParquetOutputDialog(
Shell parent,
@@ -91,355 +70,79 @@ public class ParquetOutputDialog extends
BaseTransformDialog {
public String open() {
createShell(BaseMessages.getString(PKG, "ParquetOutput.Name"));
+ changed = input.hasChanged();
+
buildButtonBar().ok(e -> ok()).get(e -> getFields()).cancel(e ->
cancel()).build();
- Control lastControl = wSpacer;
-
- Group wFileGroup = new Group(shell, SWT.SHADOW_ETCHED_IN);
- wFileGroup.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameGroup.Label"));
- wFileGroup.setLayout(new FormLayout());
- PropsUi.setLook(wFileGroup);
- FormData fdFileGroup = new FormData();
- fdFileGroup.left = new FormAttachment(0, 0);
- fdFileGroup.right = new FormAttachment(100, 0);
- fdFileGroup.top = new FormAttachment(lastControl, margin);
- wFileGroup.setLayoutData(fdFileGroup);
-
- Label wlFilenameBase = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameBase.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameBase.Label"));
- PropsUi.setLook(wlFilenameBase);
- FormData fdlFilenameBase = new FormData();
- fdlFilenameBase.left = new FormAttachment(0, 0);
- fdlFilenameBase.top = new FormAttachment(0, margin);
- fdlFilenameBase.right = new FormAttachment(middle, -margin);
- wlFilenameBase.setLayoutData(fdlFilenameBase);
- wFilenameBase =
- new TextVar(variables, wFileGroup, SWT.SINGLE | SWT.LEFT | SWT.BORDER)
- .enableNamingSchemes(NamingSchemeTypes.FILE);
- PropsUi.setLook(wFilenameBase);
- FormData fdFilenameBase = new FormData();
- fdFilenameBase.left = new FormAttachment(middle, 0);
- fdFilenameBase.top = new FormAttachment(wlFilenameBase, 0, SWT.CENTER);
- fdFilenameBase.right = new FormAttachment(90, 0);
- wFilenameBase.setLayoutData(fdFilenameBase);
- lastControl = wFilenameBase;
-
- Button wbFilename = new Button(wFileGroup, SWT.PUSH | SWT.CENTER);
- PropsUi.setLook(wbFilename);
- wbFilename.setText(BaseMessages.getString(PKG, "System.Button.Browse"));
- FormData fdbFilename = new FormData();
- fdbFilename.left = new FormAttachment(wFilenameBase, 0);
- fdbFilename.top = new FormAttachment(wFilenameBase, 0, SWT.CENTER);
- wbFilename.setLayoutData(fdbFilename);
-
- wbFilename.addListener(
- SWT.Selection,
- e ->
- BaseDialog.presentFileDialog(
- true,
- shell,
- wFilenameBase,
- variables,
- new String[] {".parquet", "*"},
- new String[] {
- BaseMessages.getString(PKG,
"ParquetOutputDialog.extension.Label"),
- BaseMessages.getString(PKG, "System.FileType.AllFiles")
- },
- true));
-
- Label wlFilenameExtension = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameExtension.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameExtension.Label"));
- PropsUi.setLook(wlFilenameExtension);
- FormData fdlFilenameExtension = new FormData();
- fdlFilenameExtension.left = new FormAttachment(0, 0);
- fdlFilenameExtension.right = new FormAttachment(middle, -margin);
- fdlFilenameExtension.top = new FormAttachment(lastControl, margin);
- wlFilenameExtension.setLayoutData(fdlFilenameExtension);
- wFilenameExtension = new TextVar(variables, wFileGroup, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
- PropsUi.setLook(wFilenameExtension);
- FormData fdFilenameExtension = new FormData();
- fdFilenameExtension.left = new FormAttachment(middle, 0);
- fdFilenameExtension.top = new FormAttachment(wlFilenameExtension, 0,
SWT.CENTER);
- fdFilenameExtension.right = new FormAttachment(100, 0);
- wFilenameExtension.setLayoutData(fdFilenameExtension);
- lastControl = wFilenameExtension;
-
- Label wlFilenameIncludeDate = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameIncludeDate.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameIncludeDate.Label"));
- PropsUi.setLook(wlFilenameIncludeDate);
- FormData fdlFilenameIncludeDate = new FormData();
- fdlFilenameIncludeDate.left = new FormAttachment(0, 0);
- fdlFilenameIncludeDate.right = new FormAttachment(middle, -margin);
- fdlFilenameIncludeDate.top = new FormAttachment(lastControl, margin);
- wlFilenameIncludeDate.setLayoutData(fdlFilenameIncludeDate);
- wFilenameIncludeDate = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameIncludeDate);
- FormData fdFilenameIncludeDate = new FormData();
- fdFilenameIncludeDate.left = new FormAttachment(middle, 0);
- fdFilenameIncludeDate.top = new FormAttachment(wlFilenameIncludeDate, 0,
SWT.CENTER);
- fdFilenameIncludeDate.right = new FormAttachment(100, 0);
- wFilenameIncludeDate.setLayoutData(fdFilenameIncludeDate);
- lastControl = wlFilenameIncludeDate;
-
- Label wlFilenameIncludeTime = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameIncludeTime.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameIncludeTime.Label"));
- PropsUi.setLook(wlFilenameIncludeTime);
- FormData fdlFilenameIncludeTime = new FormData();
- fdlFilenameIncludeTime.left = new FormAttachment(0, 0);
- fdlFilenameIncludeTime.right = new FormAttachment(middle, -margin);
- fdlFilenameIncludeTime.top = new FormAttachment(lastControl, margin);
- wlFilenameIncludeTime.setLayoutData(fdlFilenameIncludeTime);
- wFilenameIncludeTime = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameIncludeTime);
- FormData fdFilenameIncludeTime = new FormData();
- fdFilenameIncludeTime.left = new FormAttachment(middle, 0);
- fdFilenameIncludeTime.top = new FormAttachment(wlFilenameIncludeTime, 0,
SWT.CENTER);
- fdFilenameIncludeTime.right = new FormAttachment(100, 0);
- wFilenameIncludeTime.setLayoutData(fdFilenameIncludeTime);
- lastControl = wlFilenameIncludeTime;
-
- Label wlFilenameIncludeDateTime = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameIncludeDateTime.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameIncludeDateTime.Label"));
- PropsUi.setLook(wlFilenameIncludeDateTime);
- FormData fdlFilenameIncludeDateTime = new FormData();
- fdlFilenameIncludeDateTime.left = new FormAttachment(0, 0);
- fdlFilenameIncludeDateTime.right = new FormAttachment(middle, -margin);
- fdlFilenameIncludeDateTime.top = new FormAttachment(lastControl, margin);
- wlFilenameIncludeDateTime.setLayoutData(fdlFilenameIncludeDateTime);
- wFilenameIncludeDateTime = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameIncludeDateTime);
- FormData fdFilenameIncludeDateTime = new FormData();
- fdFilenameIncludeDateTime.left = new FormAttachment(middle, 0);
- fdFilenameIncludeDateTime.top = new
FormAttachment(wlFilenameIncludeDateTime, 0, SWT.CENTER);
- fdFilenameIncludeDateTime.right = new FormAttachment(100, 0);
- wFilenameIncludeDateTime.setLayoutData(fdFilenameIncludeDateTime);
- wFilenameIncludeDateTime.addListener(SWT.Selection, e -> enableFields());
- lastControl = wlFilenameIncludeDateTime;
-
- wlFilenameDateTimeFormat = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameDateTimeFormat.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameDateTimeFormat.Label"));
- PropsUi.setLook(wlFilenameDateTimeFormat);
- FormData fdlFilenameDateTimeFormat = new FormData();
- fdlFilenameDateTimeFormat.left = new FormAttachment(0, 0);
- fdlFilenameDateTimeFormat.right = new FormAttachment(middle, -margin);
- fdlFilenameDateTimeFormat.top = new FormAttachment(lastControl, margin);
- wlFilenameDateTimeFormat.setLayoutData(fdlFilenameDateTimeFormat);
- wFilenameDateTimeFormat =
- new TextVar(variables, wFileGroup, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- PropsUi.setLook(wFilenameDateTimeFormat);
- FormData fdFilenameDateTimeFormat = new FormData();
- fdFilenameDateTimeFormat.left = new FormAttachment(middle, 0);
- fdFilenameDateTimeFormat.top = new
FormAttachment(wlFilenameDateTimeFormat, 0, SWT.CENTER);
- fdFilenameDateTimeFormat.right = new FormAttachment(100, 0);
- wFilenameDateTimeFormat.setLayoutData(fdFilenameDateTimeFormat);
- lastControl = wFilenameDateTimeFormat;
-
- Label wlFilenameIncludeCopyNr = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameIncludeCopyNr.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameIncludeCopyNr.Label"));
- PropsUi.setLook(wlFilenameIncludeCopyNr);
- FormData fdlFilenameIncludeCopyNr = new FormData();
- fdlFilenameIncludeCopyNr.left = new FormAttachment(0, 0);
- fdlFilenameIncludeCopyNr.right = new FormAttachment(middle, -margin);
- fdlFilenameIncludeCopyNr.top = new FormAttachment(lastControl, margin);
- wlFilenameIncludeCopyNr.setLayoutData(fdlFilenameIncludeCopyNr);
- wFilenameIncludeCopyNr = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameIncludeCopyNr);
- FormData fdFilenameIncludeCopyNr = new FormData();
- fdFilenameIncludeCopyNr.left = new FormAttachment(middle, 0);
- fdFilenameIncludeCopyNr.top = new FormAttachment(wlFilenameIncludeCopyNr,
0, SWT.CENTER);
- fdFilenameIncludeCopyNr.right = new FormAttachment(100, 0);
- wFilenameIncludeCopyNr.setLayoutData(fdFilenameIncludeCopyNr);
- lastControl = wlFilenameIncludeCopyNr;
-
- Label wlFilenameIncludeSplitNr = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameIncludeSplitNr.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameIncludeSplitNr.Label"));
- PropsUi.setLook(wlFilenameIncludeSplitNr);
- FormData fdlFilenameIncludeSplitNr = new FormData();
- fdlFilenameIncludeSplitNr.left = new FormAttachment(0, 0);
- fdlFilenameIncludeSplitNr.right = new FormAttachment(middle, -margin);
- fdlFilenameIncludeSplitNr.top = new FormAttachment(lastControl, margin);
- wlFilenameIncludeSplitNr.setLayoutData(fdlFilenameIncludeSplitNr);
- wFilenameIncludeSplitNr = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameIncludeSplitNr);
- FormData fdFilenameIncludeSplitNr = new FormData();
- fdFilenameIncludeSplitNr.left = new FormAttachment(middle, 0);
- fdFilenameIncludeSplitNr.top = new
FormAttachment(wlFilenameIncludeSplitNr, 0, SWT.CENTER);
- fdFilenameIncludeSplitNr.right = new FormAttachment(100, 0);
- wFilenameIncludeSplitNr.setLayoutData(fdFilenameIncludeSplitNr);
- wFilenameIncludeSplitNr.addListener(SWT.Selection, e -> enableFields());
- lastControl = wlFilenameIncludeSplitNr;
-
- wlFilenameSplitSize = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameSplitSize.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameSplitSize.Label"));
- PropsUi.setLook(wlFilenameSplitSize);
- FormData fdlFilenameSplitSize = new FormData();
- fdlFilenameSplitSize.left = new FormAttachment(0, 0);
- fdlFilenameSplitSize.right = new FormAttachment(middle, -margin);
- fdlFilenameSplitSize.top = new FormAttachment(lastControl, margin);
- wlFilenameSplitSize.setLayoutData(fdlFilenameSplitSize);
- wFilenameSplitSize = new TextVar(variables, wFileGroup, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
- wFilenameSplitSize.enableExpandedInteger();
- PropsUi.setLook(wFilenameSplitSize);
- FormData fdFilenameSplitSize = new FormData();
- fdFilenameSplitSize.left = new FormAttachment(middle, 0);
- fdFilenameSplitSize.top = new FormAttachment(wlFilenameSplitSize, 0,
SWT.CENTER);
- fdFilenameSplitSize.right = new FormAttachment(100, 0);
- wFilenameSplitSize.setLayoutData(fdFilenameSplitSize);
- lastControl = wFilenameSplitSize;
-
- Label wlFilenameCreateFolders = new Label(wFileGroup, SWT.RIGHT);
- wlFilenameCreateFolders.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.FilenameCreateFolders.Label"));
- PropsUi.setLook(wlFilenameCreateFolders);
- FormData fdlFilenameCreateFolders = new FormData();
- fdlFilenameCreateFolders.left = new FormAttachment(0, 0);
- fdlFilenameCreateFolders.right = new FormAttachment(middle, -margin);
- fdlFilenameCreateFolders.top = new FormAttachment(lastControl, margin);
- wlFilenameCreateFolders.setLayoutData(fdlFilenameCreateFolders);
- wFilenameCreateFolders = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameCreateFolders);
- FormData fdFilenameCreateFolders = new FormData();
- fdFilenameCreateFolders.left = new FormAttachment(middle, 0);
- fdFilenameCreateFolders.top = new FormAttachment(wlFilenameCreateFolders,
0, SWT.CENTER);
- fdFilenameCreateFolders.right = new FormAttachment(100, 0);
- wFilenameCreateFolders.setLayoutData(fdFilenameCreateFolders);
- lastControl = wlFilenameCreateFolders;
-
- Label wlFilenameCompressionBeforeExtension = new Label(wFileGroup,
SWT.RIGHT);
- wlFilenameCompressionBeforeExtension.setText(
- BaseMessages.getString(
- PKG,
"ParquetOutputDialog.FilenameCompressionBeforeExtension.Label"));
- PropsUi.setLook(wlFilenameCompressionBeforeExtension);
- FormData fdlFilenameCompressionBeforeExtension = new FormData();
- fdlFilenameCompressionBeforeExtension.left = new FormAttachment(0, 0);
- fdlFilenameCompressionBeforeExtension.right = new FormAttachment(middle,
-margin);
- fdlFilenameCompressionBeforeExtension.top = new
FormAttachment(lastControl, margin);
-
wlFilenameCompressionBeforeExtension.setLayoutData(fdlFilenameCompressionBeforeExtension);
- wFilenameCompressionBeforeExtension = new Button(wFileGroup, SWT.CHECK);
- PropsUi.setLook(wFilenameCompressionBeforeExtension);
- FormData fdFilenameCompressionBeforeExtension = new FormData();
- fdFilenameCompressionBeforeExtension.left = new FormAttachment(middle, 0);
- fdFilenameCompressionBeforeExtension.top =
- new FormAttachment(wlFilenameCompressionBeforeExtension, 0,
SWT.CENTER);
- fdFilenameCompressionBeforeExtension.right = new FormAttachment(100, 0);
-
wFilenameCompressionBeforeExtension.setLayoutData(fdFilenameCompressionBeforeExtension);
-
- // End of the file group
- //
- lastControl = wFileGroup;
-
- Label wlCompressionCodec = new Label(shell, SWT.RIGHT);
- wlCompressionCodec.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.CompressionCodec.Label"));
- PropsUi.setLook(wlCompressionCodec);
- FormData fdlCompressionCodec = new FormData();
- fdlCompressionCodec.left = new FormAttachment(0, 0);
- fdlCompressionCodec.right = new FormAttachment(middle, -margin);
- fdlCompressionCodec.top = new FormAttachment(lastControl, margin);
- wlCompressionCodec.setLayoutData(fdlCompressionCodec);
- wCompressionCodec = new Combo(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- for (CompressionCodecName codecName : CompressionCodecName.values()) {
- wCompressionCodec.add(codecName.name());
- }
- PropsUi.setLook(wCompressionCodec);
- FormData fdCompressionCodec = new FormData();
- fdCompressionCodec.left = new FormAttachment(middle, 0);
- fdCompressionCodec.top = new FormAttachment(wlCompressionCodec, 0,
SWT.CENTER);
- fdCompressionCodec.right = new FormAttachment(100, 0);
- wCompressionCodec.setLayoutData(fdCompressionCodec);
- lastControl = wCompressionCodec;
-
- Label wlVersion = new Label(shell, SWT.RIGHT);
- wlVersion.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.Version.Label"));
- PropsUi.setLook(wlVersion);
- FormData fdlVersion = new FormData();
- fdlVersion.left = new FormAttachment(0, 0);
- fdlVersion.right = new FormAttachment(middle, -margin);
- fdlVersion.top = new FormAttachment(lastControl, margin);
- wlVersion.setLayoutData(fdlVersion);
- wVersion = new Combo(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- for (ParquetVersion version : ParquetVersion.values()) {
- wVersion.add(version.getDescription());
- }
- PropsUi.setLook(wVersion);
- FormData fdVersion = new FormData();
- fdVersion.left = new FormAttachment(middle, 0);
- fdVersion.top = new FormAttachment(wlVersion, 0, SWT.CENTER);
- fdVersion.right = new FormAttachment(100, 0);
- wVersion.setLayoutData(fdVersion);
- lastControl = wVersion;
-
- Label wlRowGroupSize = new Label(shell, SWT.RIGHT);
- wlRowGroupSize.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.RowGroupSize.Label"));
- PropsUi.setLook(wlRowGroupSize);
- FormData fdlRowGroupSize = new FormData();
- fdlRowGroupSize.left = new FormAttachment(0, 0);
- fdlRowGroupSize.right = new FormAttachment(middle, -margin);
- fdlRowGroupSize.top = new FormAttachment(lastControl, margin);
- wlRowGroupSize.setLayoutData(fdlRowGroupSize);
- wRowGroupSize = new TextVar(variables, shell, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
- wRowGroupSize.enableExpandedInteger();
- PropsUi.setLook(wRowGroupSize);
- FormData fdRowGroupSize = new FormData();
- fdRowGroupSize.left = new FormAttachment(middle, 0);
- fdRowGroupSize.top = new FormAttachment(wlRowGroupSize, 0, SWT.CENTER);
- fdRowGroupSize.right = new FormAttachment(100, 0);
- wRowGroupSize.setLayoutData(fdRowGroupSize);
- lastControl = wRowGroupSize;
-
- Label wlDataPageSize = new Label(shell, SWT.RIGHT);
- wlDataPageSize.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.DataPageSize.Label"));
- PropsUi.setLook(wlDataPageSize);
- FormData fdlDataPageSize = new FormData();
- fdlDataPageSize.left = new FormAttachment(0, 0);
- fdlDataPageSize.right = new FormAttachment(middle, -margin);
- fdlDataPageSize.top = new FormAttachment(lastControl, margin);
- wlDataPageSize.setLayoutData(fdlDataPageSize);
- wDataPageSize = new TextVar(variables, shell, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
- wDataPageSize.enableExpandedInteger();
- PropsUi.setLook(wDataPageSize);
- FormData fdDataPageSize = new FormData();
- fdDataPageSize.left = new FormAttachment(middle, 0);
- fdDataPageSize.top = new FormAttachment(wlDataPageSize, 0, SWT.CENTER);
- fdDataPageSize.right = new FormAttachment(100, 0);
- wDataPageSize.setLayoutData(fdDataPageSize);
- lastControl = wDataPageSize;
-
- Label wlDictionaryPageSize = new Label(shell, SWT.RIGHT);
- wlDictionaryPageSize.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.DictionaryPageSize.Label"));
- PropsUi.setLook(wlDictionaryPageSize);
- FormData fdlDictionaryPageSize = new FormData();
- fdlDictionaryPageSize.left = new FormAttachment(0, 0);
- fdlDictionaryPageSize.right = new FormAttachment(middle, -margin);
- fdlDictionaryPageSize.top = new FormAttachment(lastControl, margin);
- wlDictionaryPageSize.setLayoutData(fdlDictionaryPageSize);
- wDictionaryPageSize = new TextVar(variables, shell, SWT.SINGLE | SWT.LEFT
| SWT.BORDER);
- wDictionaryPageSize.enableExpandedInteger();
- PropsUi.setLook(wDictionaryPageSize);
- FormData fdDictionaryPageSize = new FormData();
- fdDictionaryPageSize.left = new FormAttachment(middle, 0);
- fdDictionaryPageSize.top = new FormAttachment(wlDictionaryPageSize, 0,
SWT.CENTER);
- fdDictionaryPageSize.right = new FormAttachment(100, 0);
- wDictionaryPageSize.setLayoutData(fdDictionaryPageSize);
- lastControl = wDictionaryPageSize;
-
- Label wlPartitionFields = new Label(shell, SWT.LEFT);
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ ParquetOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input,
+ w -> {
+ // Extra-group builders run during createCompositeWidgets, before
+ // addScrolledComposite returns. Keep the field assigned so they
can
+ // look up widgets already placed on the same tab.
+ widgets = w;
+ w.registerExtraGroup(
+ BaseMessages.getString(PKG,
"ParquetOutputMeta.Group.Partitioning"),
+ "0300",
+ null,
+ this::addPartitionFieldsTable);
+ w.registerExtraGroup(
+ BaseMessages.getString(PKG,
"ParquetOutputMeta.Group.Fields"),
+ "0400",
+ null,
+ this::addFieldsTable);
+ });
+ widgets.setWidgetsListener(
+ new GuiCompositeWidgetsAdapter() {
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control changedWidget,
String widgetId) {
+ if (!loading) {
+ input.setChanged();
+ }
+ if
(ParquetOutputMeta.WIDGET_FILENAME_INCLUDE_DATETIME.equals(widgetId)
+ ||
ParquetOutputMeta.WIDGET_FILENAME_INCLUDE_SPLIT_NR.equals(widgetId)) {
+ enableFields();
+ }
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {
+ persistTables();
+ persistDescribedEnums();
+ }
+ });
+
+ applyDescribedEnumCombos();
+ enableExpandedIntegers();
+ populateTables();
+ setFieldComboValues();
+ enableFields();
+ input.setChanged(changed);
+
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
+ return transformName;
+ }
+
+ private void addPartitionFieldsTable(Composite parent) {
+ Control last =
widgets.getWidgetsMap().get(ParquetOutputMeta.WIDGET_MAX_OPEN_PARTITIONS);
+
+ Label wlPartitionFields = new Label(parent, SWT.LEFT);
wlPartitionFields.setText(
BaseMessages.getString(PKG,
"ParquetOutputDialog.PartitionFields.Label"));
PropsUi.setLook(wlPartitionFields);
FormData fdlPartitionFields = new FormData();
fdlPartitionFields.left = new FormAttachment(0, 0);
fdlPartitionFields.right = new FormAttachment(100, 0);
- fdlPartitionFields.top = new FormAttachment(lastControl, margin);
+ fdlPartitionFields.top =
+ last == null ? new FormAttachment(0, 0) : new FormAttachment(last,
margin);
wlPartitionFields.setLayoutData(fdlPartitionFields);
ColumnInfo[] partitionColumns =
@@ -452,74 +155,34 @@ public class ParquetOutputDialog extends
BaseTransformDialog {
wPartitionFields =
new TableView(
variables,
- shell,
- SWT.BORDER,
+ parent,
+ SWT.BORDER | SWT.FULL_SELECTION | SWT.MULTI,
partitionColumns,
- input.getPartitionFields().size(),
+ input.getPartitionFields() == null ? 0 :
input.getPartitionFields().size(),
false,
- null,
+ e -> {
+ if (!loading) {
+ input.setChanged();
+ }
+ enableFields();
+ },
props);
- PropsUi.setLook(wPartitionFields);
FormData fdPartitionFields = new FormData();
fdPartitionFields.left = new FormAttachment(0, 0);
fdPartitionFields.top = new FormAttachment(wlPartitionFields, margin);
fdPartitionFields.right = new FormAttachment(100, 0);
- fdPartitionFields.height = (int) (100 * props.getZoomFactor());
+ fdPartitionFields.bottom = new FormAttachment(100, 0);
wPartitionFields.setLayoutData(fdPartitionFields);
- wPartitionFields.addModifyListener(e -> enableFields());
- lastControl = wPartitionFields;
-
- wlWriteMode = new Label(shell, SWT.RIGHT);
- wlWriteMode.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.WriteMode.Label"));
- wlWriteMode.setToolTipText(
- BaseMessages.getString(PKG, "ParquetOutputDialog.WriteMode.ToolTip"));
- PropsUi.setLook(wlWriteMode);
- FormData fdlWriteMode = new FormData();
- fdlWriteMode.left = new FormAttachment(0, 0);
- fdlWriteMode.right = new FormAttachment(middle, -margin);
- fdlWriteMode.top = new FormAttachment(lastControl, margin);
- wlWriteMode.setLayoutData(fdlWriteMode);
- wWriteMode = new Combo(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER |
SWT.READ_ONLY);
- wWriteMode.setItems(ParquetWriteMode.getDescriptions());
- wWriteMode.setToolTipText(BaseMessages.getString(PKG,
"ParquetOutputDialog.WriteMode.ToolTip"));
- PropsUi.setLook(wWriteMode);
- FormData fdWriteMode = new FormData();
- fdWriteMode.left = new FormAttachment(middle, 0);
- fdWriteMode.top = new FormAttachment(wlWriteMode, 0, SWT.CENTER);
- fdWriteMode.right = new FormAttachment(100, 0);
- wWriteMode.setLayoutData(fdWriteMode);
- lastControl = wWriteMode;
-
- wlMaxOpenPartitions = new Label(shell, SWT.RIGHT);
- wlMaxOpenPartitions.setText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.MaxOpenPartitions.Label"));
- wlMaxOpenPartitions.setToolTipText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.MaxOpenPartitions.ToolTip"));
- PropsUi.setLook(wlMaxOpenPartitions);
- FormData fdlMaxOpenPartitions = new FormData();
- fdlMaxOpenPartitions.left = new FormAttachment(0, 0);
- fdlMaxOpenPartitions.right = new FormAttachment(middle, -margin);
- fdlMaxOpenPartitions.top = new FormAttachment(lastControl, margin);
- wlMaxOpenPartitions.setLayoutData(fdlMaxOpenPartitions);
- wMaxOpenPartitions = new TextVar(variables, shell, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
- wMaxOpenPartitions.enableExpandedInteger();
- wMaxOpenPartitions.setToolTipText(
- BaseMessages.getString(PKG,
"ParquetOutputDialog.MaxOpenPartitions.ToolTip"));
- PropsUi.setLook(wMaxOpenPartitions);
- FormData fdMaxOpenPartitions = new FormData();
- fdMaxOpenPartitions.left = new FormAttachment(middle, 0);
- fdMaxOpenPartitions.top = new FormAttachment(wlMaxOpenPartitions, 0,
SWT.CENTER);
- fdMaxOpenPartitions.right = new FormAttachment(100, 0);
- wMaxOpenPartitions.setLayoutData(fdMaxOpenPartitions);
- lastControl = wMaxOpenPartitions;
-
- Label wlFields = new Label(shell, SWT.LEFT);
+ }
+
+ private void addFieldsTable(Composite parent) {
+ Label wlFields = new Label(parent, SWT.LEFT);
wlFields.setText(BaseMessages.getString(PKG,
"ParquetOutputDialog.Fields.Label"));
PropsUi.setLook(wlFields);
FormData fdlFields = new FormData();
fdlFields.left = new FormAttachment(0, 0);
- fdlFields.right = new FormAttachment(middle, -margin);
- fdlFields.top = new FormAttachment(lastControl, margin);
+ fdlFields.right = new FormAttachment(100, 0);
+ fdlFields.top = new FormAttachment(0, 0);
wlFields.setLayoutData(fdlFields);
ColumnInfo[] columns =
@@ -537,144 +200,194 @@ public class ParquetOutputDialog extends
BaseTransformDialog {
columns[1].setNamingSchemeType(NamingSchemeTypes.HOP_FIELD);
wFields =
new TableView(
- variables, shell, SWT.BORDER, columns, input.getFields().size(),
false, null, props);
- PropsUi.setLook(wFields);
+ variables,
+ parent,
+ SWT.BORDER | SWT.FULL_SELECTION | SWT.MULTI,
+ columns,
+ input.getFields() == null ? 0 : input.getFields().size(),
+ false,
+ e -> {
+ if (!loading) {
+ input.setChanged();
+ }
+ },
+ props);
FormData fdFields = new FormData();
fdFields.left = new FormAttachment(0, 0);
fdFields.top = new FormAttachment(wlFields, margin);
fdFields.right = new FormAttachment(100, 0);
- fdFields.bottom = new FormAttachment(100, -50);
+ fdFields.bottom = new FormAttachment(100, 0);
wFields.setLayoutData(fdFields);
+ }
- getData();
- focusTransformName();
- BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
- return returnValue;
+ private void applyDescribedEnumCombos() {
+ widgets.setComboValues(ParquetOutputMeta.WIDGET_VERSION,
ParquetVersion.getDescriptions());
+ setComboText(ParquetOutputMeta.WIDGET_VERSION,
input.getVersionDescription());
+ widgets.setComboValues(ParquetOutputMeta.WIDGET_WRITE_MODE,
ParquetWriteMode.getDescriptions());
+ setComboText(ParquetOutputMeta.WIDGET_WRITE_MODE,
input.getWriteModeDescription());
}
- private void enableFields() {
-
wlFilenameDateTimeFormat.setEnabled(wFilenameIncludeDateTime.getSelection());
-
wFilenameDateTimeFormat.setEnabled(wFilenameIncludeDateTime.getSelection());
-
- wlFilenameSplitSize.setEnabled(wFilenameIncludeSplitNr.getSelection());
- wFilenameSplitSize.setEnabled(wFilenameIncludeSplitNr.getSelection());
-
- // The write mode and the open-partition limit only mean something while
partitioning.
- boolean partitioning = !wPartitionFields.getNonEmptyItems().isEmpty();
- wlWriteMode.setEnabled(partitioning);
- wWriteMode.setEnabled(partitioning);
- wlMaxOpenPartitions.setEnabled(partitioning);
- wMaxOpenPartitions.setEnabled(partitioning);
+ private void enableExpandedIntegers() {
+ enableExpandedInteger(ParquetOutputMeta.WIDGET_FILE_SPLIT_SIZE);
+ enableExpandedInteger(ParquetOutputMeta.WIDGET_ROW_GROUP_SIZE);
+ enableExpandedInteger(ParquetOutputMeta.WIDGET_DATA_PAGE_SIZE);
+ enableExpandedInteger(ParquetOutputMeta.WIDGET_DICTIONARY_PAGE_SIZE);
+ enableExpandedInteger(ParquetOutputMeta.WIDGET_MAX_OPEN_PARTITIONS);
}
- private void getFields() {
- // Populate the wFields grid
- //
- try {
- IRowMeta rowMeta = pipelineMeta.getPrevTransformFields(variables,
transformName);
- BaseTransformDialog.getFieldsFromPrevious(
- rowMeta, wFields, 2, new int[] {1, 2}, new int[0], -1, -1, true,
null);
- } catch (Exception e) {
- new ErrorDialog(shell, "Error", "Error getting fields", e);
+ private void enableExpandedInteger(String widgetId) {
+ Control control = widgets.getWidgetsMap().get(widgetId);
+ if (control instanceof TextVar textVar) {
+ textVar.enableExpandedInteger();
+ }
+ }
+
+ private void populateTables() {
+ if (wFields != null && !wFields.isDisposed()) {
+ wFields.clearAll();
+ if (input.getFields() != null) {
+ for (ParquetField field : input.getFields()) {
+ TableItem item = new TableItem(wFields.table, SWT.NONE);
+ item.setText(1, Const.NVL(field.getSourceFieldName(), ""));
+ item.setText(2, Const.NVL(field.getTargetFieldName(), ""));
+ }
+ }
+ wFields.optimizeTableView();
+ }
+
+ if (wPartitionFields != null && !wPartitionFields.isDisposed()) {
+ wPartitionFields.clearAll();
+ if (input.getPartitionFields() != null) {
+ for (ParquetPartitionField field : input.getPartitionFields()) {
+ TableItem item = new TableItem(wPartitionFields.table, SWT.NONE);
+ item.setText(1, Const.NVL(field.getName(), ""));
+ }
+ }
+ wPartitionFields.optimizeTableView();
}
}
- private void getData() {
+ private void persistTables() {
+ if (wFields != null && !wFields.isDisposed()) {
+ List<ParquetField> fields = new ArrayList<>();
+ for (TableItem item : wFields.getNonEmptyItems()) {
+ fields.add(new ParquetField(item.getText(1), item.getText(2)));
+ }
+ input.setFields(fields);
+ }
+ if (wPartitionFields != null && !wPartitionFields.isDisposed()) {
+ List<ParquetPartitionField> partitionFields = new ArrayList<>();
+ for (TableItem item : wPartitionFields.getNonEmptyItems()) {
+ partitionFields.add(new ParquetPartitionField(item.getText(1)));
+ }
+ input.setPartitionFields(partitionFields);
+ }
+ }
+
+ private void persistDescribedEnums() {
+ input.setVersion(
+
ParquetVersion.getVersionFromDescription(getComboText(ParquetOutputMeta.WIDGET_VERSION)));
+ input.setWriteMode(
+
ParquetWriteMode.getModeFromDescription(getComboText(ParquetOutputMeta.WIDGET_WRITE_MODE)));
+ }
+
+ private void setFieldComboValues() {
try {
IRowMeta fields = pipelineMeta.getPrevTransformFields(variables,
transformName);
- wFields.getColumns()[0].setComboValues(fields.getFieldNames());
- wPartitionFields.getColumns()[0].setComboValues(fields.getFieldNames());
+ String[] names = fields == null ? new String[0] : fields.getFieldNames();
+ if (wFields != null && !wFields.isDisposed()) {
+ wFields.getColumns()[0].setComboValues(names);
+ }
+ if (wPartitionFields != null && !wPartitionFields.isDisposed()) {
+ wPartitionFields.getColumns()[0].setComboValues(names);
+ }
} catch (Exception e) {
LogChannel.UI.logError("Error getting source fields", e);
}
- wFilenameBase.setText(Const.NVL(input.getFilenameBase(), ""));
- wFilenameExtension.setText(Const.NVL(input.getFilenameExtension(), ""));
- wFilenameIncludeDate.setSelection(input.isFilenameIncludingDate());
- wFilenameIncludeTime.setSelection(input.isFilenameIncludingTime());
- wFilenameIncludeDateTime.setSelection(input.isFilenameIncludingDateTime());
-
wFilenameDateTimeFormat.setText(Const.NVL(input.getFilenameDateTimeFormat(),
""));
- wFilenameIncludeCopyNr.setSelection(input.isFilenameIncludingCopyNr());
- wFilenameIncludeSplitNr.setSelection(input.isFilenameIncludingSplitNr());
- wFilenameSplitSize.setText(Const.NVL(input.getFileSplitSize(), ""));
-
wFilenameCreateFolders.setSelection(input.isFilenameCreatingParentFolders());
-
wFilenameCompressionBeforeExtension.setSelection(input.isFilenameCompressionBeforeExtension());
- wCompressionCodec.setText(input.getCompressionCodec().name());
- wVersion.setText(input.getVersion().getDescription());
- wRowGroupSize.setText(Const.NVL(input.getRowGroupSize(), ""));
- wDataPageSize.setText(Const.NVL(input.getDataPageSize(), ""));
- wDictionaryPageSize.setText(Const.NVL(input.getDictionaryPageSize(), ""));
- for (int i = 0; i < input.getFields().size(); i++) {
- ParquetField field = input.getFields().get(i);
- TableItem item = wFields.table.getItem(i);
- item.setText(1, Const.NVL(field.getSourceFieldName(), ""));
- item.setText(2, Const.NVL(field.getTargetFieldName(), ""));
+ }
+
+ private void enableFields() {
+ boolean includeDateTime =
isChecked(ParquetOutputMeta.WIDGET_FILENAME_INCLUDE_DATETIME);
+ setEnabled(ParquetOutputMeta.WIDGET_FILENAME_DATETIME_FORMAT,
includeDateTime);
+
+ boolean includeSplit =
isChecked(ParquetOutputMeta.WIDGET_FILENAME_INCLUDE_SPLIT_NR);
+ setEnabled(ParquetOutputMeta.WIDGET_FILE_SPLIT_SIZE, includeSplit);
+
+ boolean partitioning =
+ wPartitionFields != null
+ && !wPartitionFields.isDisposed()
+ && !wPartitionFields.getNonEmptyItems().isEmpty();
+ setEnabled(ParquetOutputMeta.WIDGET_WRITE_MODE, partitioning);
+ setEnabled(ParquetOutputMeta.WIDGET_MAX_OPEN_PARTITIONS, partitioning);
+ }
+
+ private boolean isChecked(String widgetId) {
+ Control control = widgets.getWidgetsMap().get(widgetId);
+ return control instanceof Button button && button.getSelection();
+ }
+
+ private void setEnabled(String widgetId, boolean enabled) {
+ Control label = widgets.getLabelsMap().get(widgetId);
+ if (label != null && !label.isDisposed()) {
+ label.setEnabled(enabled);
}
- wFields.optimizeTableView();
+ Control widget = widgets.getWidgetsMap().get(widgetId);
+ if (widget != null && !widget.isDisposed()) {
+ widget.setEnabled(enabled);
+ }
+ }
- for (int i = 0; i < input.getPartitionFields().size(); i++) {
- ParquetPartitionField field = input.getPartitionFields().get(i);
- TableItem item = wPartitionFields.table.getItem(i);
- item.setText(1, Const.NVL(field.getName(), ""));
+ private void setComboText(String widgetId, String text) {
+ String value = Const.NVL(text, "");
+ Control control = widgets.getWidgetsMap().get(widgetId);
+ if (control instanceof ComboVar comboVar) {
+ comboVar.setText(value);
+ } else if (control instanceof Combo combo) {
+ combo.setText(value);
}
- wPartitionFields.optimizeTableView();
- wWriteMode.setText(
- input.getWriteMode() == null
- ? ParquetWriteMode.Append.getDescription()
- : input.getWriteMode().getDescription());
- wMaxOpenPartitions.setText(Const.NVL(input.getMaxOpenPartitions(), ""));
+ }
- enableFields();
+ private String getComboText(String widgetId) {
+ Control control = widgets.getWidgetsMap().get(widgetId);
+ if (control instanceof ComboVar comboVar) {
+ return Const.NVL(comboVar.getText(), "");
+ }
+ if (control instanceof Combo combo) {
+ return Const.NVL(combo.getText(), "");
+ }
+ return "";
}
- private void ok() {
- returnValue = wTransformName.getText();
-
- input.setFilenameBase(wFilenameBase.getText());
- input.setFilenameExtension(wFilenameExtension.getText());
- input.setFilenameIncludingDate(wFilenameIncludeDate.getSelection());
- input.setFilenameIncludingTime(wFilenameIncludeTime.getSelection());
-
input.setFilenameIncludingDateTime(wFilenameIncludeDateTime.getSelection());
- input.setFilenameDateTimeFormat(wFilenameDateTimeFormat.getText());
- input.setFilenameIncludingCopyNr(wFilenameIncludeCopyNr.getSelection());
- input.setFilenameIncludingSplitNr(wFilenameIncludeSplitNr.getSelection());
- input.setFileSplitSize(wFilenameSplitSize.getText());
-
input.setFilenameCreatingParentFolders(wFilenameCreateFolders.getSelection());
-
input.setFilenameCompressionBeforeExtension(wFilenameCompressionBeforeExtension.getSelection());
-
- CompressionCodecName codec = CompressionCodecName.UNCOMPRESSED;
+ private void getFields() {
try {
- codec = CompressionCodecName.valueOf(wCompressionCodec.getText());
+ IRowMeta rowMeta = pipelineMeta.getPrevTransformFields(variables,
transformName);
+ BaseTransformDialog.getFieldsFromPrevious(
+ rowMeta, wFields, 2, new int[] {1, 2}, new int[0], -1, -1, true,
null);
} catch (Exception e) {
- // Uncompressed it is.
- }
- input.setCompressionCodec(codec);
-
-
input.setVersion(ParquetVersion.getVersionFromDescription(wVersion.getText()));
- input.setRowGroupSize(wRowGroupSize.getText());
- input.setDataPageSize(wDataPageSize.getText());
- input.setDictionaryPageSize(wDictionaryPageSize.getText());
- input.getFields().clear();
- for (TableItem item : wFields.getNonEmptyItems()) {
- input.getFields().add(new ParquetField(item.getText(1),
item.getText(2)));
- }
- input.getPartitionFields().clear();
- for (TableItem item : wPartitionFields.getNonEmptyItems()) {
- input.getPartitionFields().add(new
ParquetPartitionField(item.getText(1)));
+ new ErrorDialog(
+ shell,
+ BaseMessages.getString(PKG,
"ParquetOutputDialog.FailedToGetFields.Title"),
+ BaseMessages.getString(PKG,
"ParquetOutputDialog.FailedToGetFields.Message"),
+ e);
}
-
input.setWriteMode(ParquetWriteMode.getModeFromDescription(wWriteMode.getText()));
- input.setMaxOpenPartitions(wMaxOpenPartitions.getText());
- input.setChanged();
- dispose();
}
private void cancel() {
- returnValue = null;
+ transformName = null;
+ input.setChanged(changed);
dispose();
}
- @Override
- public void dispose() {
- props.setScreen(new WindowProperty(shell));
- shell.dispose();
+ private void ok() {
+ if (Utils.isEmpty(wTransformName.getText())) {
+ return;
+ }
+
+ widgets.getWidgetsContents(input,
ParquetOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ persistTables();
+ persistDescribedEnums();
+ transformName = wTransformName.getText();
+ input.setChanged();
+ dispose();
}
}
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputMeta.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputMeta.java
index a523a40931..027f11ba26 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputMeta.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetOutputMeta.java
@@ -19,12 +19,20 @@ package org.apache.hop.parquet.transforms.output;
import java.util.ArrayList;
import java.util.List;
+import lombok.Getter;
+import lombok.Setter;
import org.apache.hop.core.annotations.Transform;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
import org.apache.hop.metadata.api.HopMetadataProperty;
import org.apache.hop.pipeline.transform.BaseTransformMeta;
import org.apache.parquet.column.ParquetProperties;
import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+@Getter
+@Setter
@Transform(
id = "ParquetFileOutput",
image = "parquet_output.svg",
@@ -33,35 +41,165 @@ import
org.apache.parquet.hadoop.metadata.CompressionCodecName;
categoryDescription =
"i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.Output",
documentationUrl = "/pipeline/transforms/parquet-file-output.html",
keywords = "i18n::ParquetOutputMeta.keyword")
+@GuiPlugin
public class ParquetOutputMeta extends BaseTransformMeta<ParquetOutput,
ParquetOutputData> {
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID =
"PARQUET_OUTPUT_DIALOG_OPTIONS";
+
+ public static final String GROUP_FILE = "i18n::ParquetOutputMeta.Group.File";
+ public static final String GROUP_OPTIONS =
"i18n::ParquetOutputMeta.Group.Options";
+ public static final String GROUP_PARTITIONING =
"i18n::ParquetOutputMeta.Group.Partitioning";
+ public static final String GROUP_FIELDS =
"i18n::ParquetOutputMeta.Group.Fields";
+
+ public static final String WIDGET_FILENAME_BASE = "filenameBase";
+ public static final String WIDGET_FILENAME_EXTENSION = "filenameExtension";
+ public static final String WIDGET_FILENAME_INCLUDE_DATE =
"filenameIncludingDate";
+ public static final String WIDGET_FILENAME_INCLUDE_TIME =
"filenameIncludingTime";
+ public static final String WIDGET_FILENAME_INCLUDE_DATETIME =
"filenameIncludingDateTime";
+ public static final String WIDGET_FILENAME_DATETIME_FORMAT =
"filenameDateTimeFormat";
+ public static final String WIDGET_FILENAME_INCLUDE_COPY_NR =
"filenameIncludingCopyNr";
+ public static final String WIDGET_FILENAME_INCLUDE_SPLIT_NR =
"filenameIncludingSplitNr";
+ public static final String WIDGET_FILE_SPLIT_SIZE = "fileSplitSize";
+ public static final String WIDGET_FILENAME_CREATE_FOLDERS =
"filenameCreatingParentFolders";
+ public static final String WIDGET_FILENAME_COMPRESSION_BEFORE_EXTENSION =
+ "filenameCompressionBeforeExtension";
+ public static final String WIDGET_COMPRESSION_CODEC = "compressionCodec";
+ public static final String WIDGET_VERSION = "version";
+ public static final String WIDGET_ROW_GROUP_SIZE = "rowGroupSize";
+ public static final String WIDGET_DATA_PAGE_SIZE = "dataPageSize";
+ public static final String WIDGET_DICTIONARY_PAGE_SIZE =
"dictionaryPageSize";
+ public static final String WIDGET_WRITE_MODE = "writeMode";
+ public static final String WIDGET_MAX_OPEN_PARTITIONS = "maxOpenPartitions";
+
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_BASE,
+ order = "0100",
+ type = GuiElementType.FILENAME,
+ typeFilename = ParquetTypeFilename.class,
+ namingSchemeType = "file",
+ label = "i18n::ParquetOutputDialog.FilenameBase.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameBase.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_base")
private String filenameBase;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_EXTENSION,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.FilenameExtension.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameExtension.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_ext")
private String filenameExtension;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_INCLUDE_DATE,
+ order = "0300",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameIncludeDate.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameIncludeDate.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_include_date")
private boolean filenameIncludingDate;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_INCLUDE_TIME,
+ order = "0400",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameIncludeTime.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameIncludeTime.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_include_time")
private boolean filenameIncludingTime;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_INCLUDE_DATETIME,
+ order = "0500",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameIncludeDateTime.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameIncludeDateTime.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_include_datetime")
private boolean filenameIncludingDateTime;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_DATETIME_FORMAT,
+ order = "0600",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.FilenameDateTimeFormat.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameDateTimeFormat.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_datetime_format")
private String filenameDateTimeFormat;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_INCLUDE_COPY_NR,
+ order = "0700",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameIncludeCopyNr.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameIncludeCopyNr.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_include_copy")
private boolean filenameIncludingCopyNr;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_INCLUDE_SPLIT_NR,
+ order = "0800",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameIncludeSplitNr.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameIncludeSplitNr.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_include_split")
private boolean filenameIncludingSplitNr;
+ @GuiWidgetElement(
+ id = WIDGET_FILE_SPLIT_SIZE,
+ order = "0900",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.FilenameSplitSize.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameSplitSize.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_split_size")
private String fileSplitSize;
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_CREATE_FOLDERS,
+ order = "1000",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::ParquetOutputDialog.FilenameCreateFolders.Label",
+ toolTip = "i18n::ParquetOutputMeta.FilenameCreateFolders.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_create_parent_folders")
private boolean filenameCreatingParentFolders;
@@ -71,21 +209,84 @@ public class ParquetOutputMeta extends
BaseTransformMeta<ParquetOutput, ParquetO
* file.parquet.snappy). New transforms default to true; pipelines loaded
without this property
* keep false for backward compatibility.
*/
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_COMPRESSION_BEFORE_EXTENSION,
+ order = "1100",
+ type = GuiElementType.CHECKBOX,
+ label =
"i18n::ParquetOutputDialog.FilenameCompressionBeforeExtension.Label",
+ toolTip =
"i18n::ParquetOutputMeta.FilenameCompressionBeforeExtension.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_FILE,
+ groupOrder = "0100")
@HopMetadataProperty(key = "filename_compression_before_extension")
private boolean filenameCompressionBeforeExtension;
+ @GuiWidgetElement(
+ id = WIDGET_COMPRESSION_CODEC,
+ order = "0100",
+ type = GuiElementType.COMBO,
+ variables = false,
+ label = "i18n::ParquetOutputDialog.CompressionCodec.Label",
+ toolTip = "i18n::ParquetOutputMeta.CompressionCodec.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_OPTIONS,
+ groupOrder = "0200")
@HopMetadataProperty(key = "compression_codec")
private CompressionCodecName compressionCodec;
+ @GuiWidgetElement(
+ id = WIDGET_VERSION,
+ order = "0200",
+ type = GuiElementType.COMBO,
+ variables = false,
+ getterMethod = "getVersionDescription",
+ label = "i18n::ParquetOutputDialog.Version.Label",
+ toolTip = "i18n::ParquetOutputMeta.Version.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_OPTIONS,
+ groupOrder = "0200")
@HopMetadataProperty(key = "version", storeWithCode = true)
private ParquetVersion version;
+ @GuiWidgetElement(
+ id = WIDGET_ROW_GROUP_SIZE,
+ order = "0300",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.RowGroupSize.Label",
+ toolTip = "i18n::ParquetOutputMeta.RowGroupSize.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_OPTIONS,
+ groupOrder = "0200")
@HopMetadataProperty(key = "row_group_size")
private String rowGroupSize;
+ @GuiWidgetElement(
+ id = WIDGET_DATA_PAGE_SIZE,
+ order = "0400",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.DataPageSize.Label",
+ toolTip = "i18n::ParquetOutputMeta.DataPageSize.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_OPTIONS,
+ groupOrder = "0200")
@HopMetadataProperty(key = "data_page_size")
private String dataPageSize;
+ @GuiWidgetElement(
+ id = WIDGET_DICTIONARY_PAGE_SIZE,
+ order = "0500",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.DictionaryPageSize.Label",
+ toolTip = "i18n::ParquetOutputMeta.DictionaryPageSize.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_OPTIONS,
+ groupOrder = "0200")
@HopMetadataProperty(key = "dictionary_page_size")
private String dictionaryPageSize;
@@ -95,9 +296,31 @@ public class ParquetOutputMeta extends
BaseTransformMeta<ParquetOutput, ParquetO
@HopMetadataProperty(groupKey = "partition_fields", key = "partition_field")
private List<ParquetPartitionField> partitionFields;
+ @GuiWidgetElement(
+ id = WIDGET_WRITE_MODE,
+ order = "0100",
+ type = GuiElementType.COMBO,
+ variables = false,
+ getterMethod = "getWriteModeDescription",
+ label = "i18n::ParquetOutputDialog.WriteMode.Label",
+ toolTip = "i18n::ParquetOutputDialog.WriteMode.ToolTip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_PARTITIONING,
+ groupOrder = "0300")
@HopMetadataProperty(key = "write_mode", storeWithCode = true)
private ParquetWriteMode writeMode;
+ @GuiWidgetElement(
+ id = WIDGET_MAX_OPEN_PARTITIONS,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ label = "i18n::ParquetOutputDialog.MaxOpenPartitions.Label",
+ toolTip = "i18n::ParquetOutputDialog.MaxOpenPartitions.ToolTip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.TABS,
+ group = GROUP_PARTITIONING,
+ groupOrder = "0300")
@HopMetadataProperty(key = "max_open_partitions")
private String maxOpenPartitions;
@@ -136,7 +359,12 @@ public class ParquetOutputMeta extends
BaseTransformMeta<ParquetOutput, ParquetO
this.rowGroupSize = m.rowGroupSize;
this.dataPageSize = m.dataPageSize;
this.dictionaryPageSize = m.dictionaryPageSize;
- this.fields = m.fields;
+ this.fields = new ArrayList<>();
+ if (m.fields != null) {
+ for (ParquetField field : m.fields) {
+ this.fields.add(new ParquetField(field));
+ }
+ }
this.partitionFields = new ArrayList<>();
if (m.partitionFields != null) {
for (ParquetPartitionField f : m.partitionFields) {
@@ -148,291 +376,16 @@ public class ParquetOutputMeta extends
BaseTransformMeta<ParquetOutput, ParquetO
}
/**
- * Gets filenameBase
- *
- * @return value of filenameBase
- */
- public String getFilenameBase() {
- return filenameBase;
- }
-
- /**
- * @param filenameBase The filenameBase to set
- */
- public void setFilenameBase(String filenameBase) {
- this.filenameBase = filenameBase;
- }
-
- /**
- * Gets filenameExtension
- *
- * @return value of filenameExtension
- */
- public String getFilenameExtension() {
- return filenameExtension;
- }
-
- /**
- * @param filenameExtension The filenameExtension to set
- */
- public void setFilenameExtension(String filenameExtension) {
- this.filenameExtension = filenameExtension;
- }
-
- /**
- * Gets filenameIncludingDate
- *
- * @return value of filenameIncludingDate
- */
- public boolean isFilenameIncludingDate() {
- return filenameIncludingDate;
- }
-
- /**
- * @param filenameIncludingDate The filenameIncludingDate to set
- */
- public void setFilenameIncludingDate(boolean filenameIncludingDate) {
- this.filenameIncludingDate = filenameIncludingDate;
- }
-
- /**
- * Gets filenameIncludingTime
- *
- * @return value of filenameIncludingTime
- */
- public boolean isFilenameIncludingTime() {
- return filenameIncludingTime;
- }
-
- /**
- * @param filenameIncludingTime The filenameIncludingTime to set
- */
- public void setFilenameIncludingTime(boolean filenameIncludingTime) {
- this.filenameIncludingTime = filenameIncludingTime;
- }
-
- /**
- * Gets filenameIncludingDateTime
- *
- * @return value of filenameIncludingDateTime
- */
- public boolean isFilenameIncludingDateTime() {
- return filenameIncludingDateTime;
- }
-
- /**
- * @param filenameIncludingDateTime The filenameIncludingDateTime to set
- */
- public void setFilenameIncludingDateTime(boolean filenameIncludingDateTime) {
- this.filenameIncludingDateTime = filenameIncludingDateTime;
- }
-
- /**
- * Gets filenameDateTimeFormat
- *
- * @return value of filenameDateTimeFormat
- */
- public String getFilenameDateTimeFormat() {
- return filenameDateTimeFormat;
- }
-
- /**
- * @param filenameDateTimeFormat The filenameDateTimeFormat to set
- */
- public void setFilenameDateTimeFormat(String filenameDateTimeFormat) {
- this.filenameDateTimeFormat = filenameDateTimeFormat;
- }
-
- /**
- * Gets filenameIncludingCopyNr
- *
- * @return value of filenameIncludingCopyNr
- */
- public boolean isFilenameIncludingCopyNr() {
- return filenameIncludingCopyNr;
- }
-
- /**
- * @param filenameIncludingCopyNr The filenameIncludingCopyNr to set
- */
- public void setFilenameIncludingCopyNr(boolean filenameIncludingCopyNr) {
- this.filenameIncludingCopyNr = filenameIncludingCopyNr;
- }
-
- /**
- * Gets filenameIncludingSplitNr
- *
- * @return value of filenameIncludingSplitNr
- */
- public boolean isFilenameIncludingSplitNr() {
- return filenameIncludingSplitNr;
- }
-
- /**
- * @param filenameIncludingSplitNr The filenameIncludingSplitNr to set
- */
- public void setFilenameIncludingSplitNr(boolean filenameIncludingSplitNr) {
- this.filenameIncludingSplitNr = filenameIncludingSplitNr;
- }
-
- /**
- * Gets filenameIncludingSplitSize
- *
- * @return value of filenameIncludingSplitSize
- */
- public String getFileSplitSize() {
- return fileSplitSize;
- }
-
- /**
- * @param fileSplitSize The filenameIncludingSplitSize to set
- */
- public void setFileSplitSize(String fileSplitSize) {
- this.fileSplitSize = fileSplitSize;
- }
-
- /**
- * Gets filenameCreatingParentFolders
- *
- * @return value of filenameCreatingParentFolders
- */
- public boolean isFilenameCreatingParentFolders() {
- return filenameCreatingParentFolders;
- }
-
- /**
- * @param filenameCreatingParentFolders The filenameCreatingParentFolders to
set
+ * Combo text for {@link #version}. {@code GuiCompositeWidgets} shows enum
{@code toString()}
+ * values by default; this getter feeds the user-facing description instead.
*/
- public void setFilenameCreatingParentFolders(boolean
filenameCreatingParentFolders) {
- this.filenameCreatingParentFolders = filenameCreatingParentFolders;
+ public String getVersionDescription() {
+ return version == null ? "" : version.getDescription();
}
- /**
- * Gets filenameCompressionBeforeExtension
- *
- * @return value of filenameCompressionBeforeExtension
- */
- public boolean isFilenameCompressionBeforeExtension() {
- return filenameCompressionBeforeExtension;
- }
-
- /**
- * @param filenameCompressionBeforeExtension The
filenameCompressionBeforeExtension to set
- */
- public void setFilenameCompressionBeforeExtension(boolean
filenameCompressionBeforeExtension) {
- this.filenameCompressionBeforeExtension =
filenameCompressionBeforeExtension;
- }
-
- /**
- * Gets compressionCodec
- *
- * @return value of compressionCodec
- */
- public CompressionCodecName getCompressionCodec() {
- return compressionCodec;
- }
-
- /**
- * @param compressionCodec The compressionCodec to set
- */
- public void setCompressionCodec(CompressionCodecName compressionCodec) {
- this.compressionCodec = compressionCodec;
- }
-
- /**
- * Gets version
- *
- * @return value of version
- */
- public ParquetVersion getVersion() {
- return version;
- }
-
- /**
- * @param version The version to set
- */
- public void setVersion(ParquetVersion version) {
- this.version = version;
- }
-
- /**
- * Gets rowGroupSize
- *
- * @return value of rowGroupSize
- */
- public String getRowGroupSize() {
- return rowGroupSize;
- }
-
- /**
- * @param rowGroupSize The rowGroupSize to set
- */
- public void setRowGroupSize(String rowGroupSize) {
- this.rowGroupSize = rowGroupSize;
- }
-
- /**
- * Gets dataPageSize
- *
- * @return value of dataPageSize
- */
- public String getDataPageSize() {
- return dataPageSize;
- }
-
- /**
- * @param dataPageSize The dataPageSize to set
- */
- public void setDataPageSize(String dataPageSize) {
- this.dataPageSize = dataPageSize;
- }
-
- /**
- * Gets dictionaryPageSize
- *
- * @return value of dictionaryPageSize
- */
- public String getDictionaryPageSize() {
- return dictionaryPageSize;
- }
-
- /**
- * @param dictionaryPageSize The dictionaryPageSize to set
- */
- public void setDictionaryPageSize(String dictionaryPageSize) {
- this.dictionaryPageSize = dictionaryPageSize;
- }
-
- /**
- * Gets fields
- *
- * @return value of fields
- */
- public List<ParquetField> getFields() {
- return fields;
- }
-
- /**
- * @param fields The fields to set
- */
- public void setFields(List<ParquetField> fields) {
- this.fields = fields;
- }
-
- /**
- * Gets partitionFields
- *
- * @return value of partitionFields
- */
- public List<ParquetPartitionField> getPartitionFields() {
- return partitionFields;
- }
-
- /**
- * @param partitionFields The partitionFields to set
- */
- public void setPartitionFields(List<ParquetPartitionField> partitionFields) {
- this.partitionFields = partitionFields;
+ /** Combo text for {@link #writeMode}. Same reason as {@link
#getVersionDescription()}. */
+ public String getWriteModeDescription() {
+ return writeMode == null ? "" : writeMode.getDescription();
}
/**
@@ -452,36 +405,4 @@ public class ParquetOutputMeta extends
BaseTransformMeta<ParquetOutput, ParquetO
}
return false;
}
-
- /**
- * Gets writeMode
- *
- * @return value of writeMode
- */
- public ParquetWriteMode getWriteMode() {
- return writeMode;
- }
-
- /**
- * @param writeMode The writeMode to set
- */
- public void setWriteMode(ParquetWriteMode writeMode) {
- this.writeMode = writeMode;
- }
-
- /**
- * Gets maxOpenPartitions
- *
- * @return value of maxOpenPartitions
- */
- public String getMaxOpenPartitions() {
- return maxOpenPartitions;
- }
-
- /**
- * @param maxOpenPartitions The maxOpenPartitions to set
- */
- public void setMaxOpenPartitions(String maxOpenPartitions) {
- this.maxOpenPartitions = maxOpenPartitions;
- }
}
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetTypeFilename.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetTypeFilename.java
new file mode 100644
index 0000000000..163a72cdb5
--- /dev/null
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetTypeFilename.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.parquet.transforms.output;
+
+import org.apache.hop.core.gui.plugin.ITypeFilename;
+import org.apache.hop.i18n.BaseMessages;
+
+public class ParquetTypeFilename implements ITypeFilename {
+ private static final Class<?> PKG = ParquetOutputMeta.class;
+
+ @Override
+ public String getDefaultFileExtension() {
+ return ".parquet";
+ }
+
+ @Override
+ public String[] getFilterExtensions() {
+ return new String[] {"*.parquet", "*"};
+ }
+
+ @Override
+ public String[] getFilterNames() {
+ return new String[] {
+ BaseMessages.getString(PKG, "ParquetOutputDialog.extension.Label"),
+ BaseMessages.getString("System.FileType.AllFiles")
+ };
+ }
+}
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetVersion.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetVersion.java
index 7488fe414e..267f0446fb 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetVersion.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetVersion.java
@@ -41,8 +41,13 @@ public enum ParquetVersion implements IEnumHasCode {
}
public static ParquetVersion getVersionFromDescription(String description) {
+ if (description == null || description.isEmpty()) {
+ return Version1;
+ }
for (ParquetVersion version : values()) {
- if (version.getDescription().equals(description)) {
+ if (version.getDescription().equals(description)
+ || version.name().equals(description)
+ || version.getCode().equals(description)) {
return version;
}
}
diff --git
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteMode.java
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteMode.java
index bb7a45aacd..826869dedb 100644
---
a/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteMode.java
+++
b/plugins/tech/parquet/src/main/java/org/apache/hop/parquet/transforms/output/ParquetWriteMode.java
@@ -56,8 +56,13 @@ public enum ParquetWriteMode implements IEnumHasCode {
}
public static ParquetWriteMode getModeFromDescription(String description) {
+ if (description == null || description.isEmpty()) {
+ return Append;
+ }
for (ParquetWriteMode mode : values()) {
- if (mode.getDescription().equals(description)) {
+ if (mode.getDescription().equals(description)
+ || mode.name().equals(description)
+ || mode.getCode().equals(description)) {
return mode;
}
}
diff --git
a/plugins/tech/parquet/src/main/resources/org/apache/hop/parquet/transforms/output/messages/messages_en_US.properties
b/plugins/tech/parquet/src/main/resources/org/apache/hop/parquet/transforms/output/messages/messages_en_US.properties
index 2f568cf488..db52a86ad3 100644
---
a/plugins/tech/parquet/src/main/resources/org/apache/hop/parquet/transforms/output/messages/messages_en_US.properties
+++
b/plugins/tech/parquet/src/main/resources/org/apache/hop/parquet/transforms/output/messages/messages_en_US.properties
@@ -50,4 +50,26 @@ ParquetOutputDialog.Version.Label=Version
ParquetOutputDialog.WriteMode.Label=Existing data
ParquetOutputDialog.WriteMode.ToolTip=What to do with data already present in
a partition folder.
ParquetOutputDialog.extension.Label=Parquet files
+ParquetOutputDialog.FailedToGetFields.Title=Get fields failed
+ParquetOutputDialog.FailedToGetFields.Message=Unable to get fields from
previous transforms
+ParquetOutputMeta.Group.File=File
+ParquetOutputMeta.Group.Options=Options
+ParquetOutputMeta.Group.Partitioning=Partitioning
+ParquetOutputMeta.Group.Fields=Fields
+ParquetOutputMeta.FilenameBase.Tooltip=Base path and file name of the Parquet
output, for example '${PROJECT_HOME}/out/data'
+ParquetOutputMeta.FilenameExtension.Tooltip=File extension, usually parquet
+ParquetOutputMeta.FilenameIncludeDate.Tooltip=Add the pipeline start date to
the file name with mask yyyyMMdd
+ParquetOutputMeta.FilenameIncludeTime.Tooltip=Add the pipeline start time to
the file name with mask HHmmss
+ParquetOutputMeta.FilenameIncludeDateTime.Tooltip=Add a custom date-time stamp
to the file name
+ParquetOutputMeta.FilenameDateTimeFormat.Tooltip=Date-time pattern used when
Include date-time format is enabled
+ParquetOutputMeta.FilenameIncludeCopyNr.Tooltip=Add the transform copy number
so parallel copies do not write the same file
+ParquetOutputMeta.FilenameIncludeSplitNr.Tooltip=Split the output into
multiple files and include the part number in the file name
+ParquetOutputMeta.FilenameSplitSize.Tooltip=Number of rows written to each
split file
+ParquetOutputMeta.FilenameCreateFolders.Tooltip=Create missing parent folders
of the output path
+ParquetOutputMeta.FilenameCompressionBeforeExtension.Tooltip=Place the
compression codec before the file extension (file.snappy.parquet). When
disabled the codec is appended after the extension (file.parquet.snappy)
+ParquetOutputMeta.CompressionCodec.Tooltip=Parquet page compression codec
+ParquetOutputMeta.Version.Tooltip=Parquet file format version
+ParquetOutputMeta.RowGroupSize.Tooltip=Number of rows in a Parquet row group
+ParquetOutputMeta.DataPageSize.Tooltip=Data page size in bytes
+ParquetOutputMeta.DictionaryPageSize.Tooltip=Dictionary page size in bytes
ParquetOutputMeta.keyword=parquet,columnar,write,file,apache
diff --git
a/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetOutputMetaTest.java
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetOutputMetaTest.java
index 6924a66926..694ed7042b 100644
---
a/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetOutputMetaTest.java
+++
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetOutputMetaTest.java
@@ -53,6 +53,12 @@ class ParquetOutputMetaTest {
assertFalse(meta.isFilenameIncludingDateTime());
assertTrue(meta.isFilenameCompressionBeforeExtension());
assertTrue(meta.getFields().isEmpty());
+ assertTrue(meta.getPartitionFields().isEmpty());
+ assertFalse(meta.isPartitioning());
+ assertEquals(ParquetWriteMode.Append, meta.getWriteMode());
+ assertEquals("10", meta.getMaxOpenPartitions());
+ assertEquals("Parquet 2.0", meta.getVersionDescription());
+ assertEquals(ParquetWriteMode.Append.getDescription(),
meta.getWriteModeDescription());
}
@Test
@@ -75,6 +81,8 @@ class ParquetOutputMetaTest {
assertEquals(ParquetVersion.Version1, copy.getVersion());
assertEquals(1, copy.getFields().size());
assertEquals("id", copy.getFields().get(0).getSourceFieldName());
+ copy.getFields().get(0).setSourceFieldName("changed");
+ assertEquals("id", original.getFields().get(0).getSourceFieldName());
}
@Test
@@ -98,6 +106,9 @@ class ParquetOutputMetaTest {
meta.setDictionaryPageSize("256");
meta.getFields().add(new ParquetField("id", "id"));
meta.getFields().add(new ParquetField("name", "name"));
+ meta.getPartitionFields().add(new ParquetPartitionField("region"));
+ meta.setWriteMode(ParquetWriteMode.OverwritePartitions);
+ meta.setMaxOpenPartitions("4");
String xml =
XmlHandler.openTag(TransformMeta.XML_TAG)
@@ -143,5 +154,13 @@ class ParquetOutputMetaTest {
expected.getFields().get(i).getTargetFieldName(),
actual.getFields().get(i).getTargetFieldName());
}
+ assertEquals(expected.getPartitionFields().size(),
actual.getPartitionFields().size());
+ for (int i = 0; i < expected.getPartitionFields().size(); i++) {
+ assertEquals(
+ expected.getPartitionFields().get(i).getName(),
+ actual.getPartitionFields().get(i).getName());
+ }
+ assertEquals(expected.getWriteMode(), actual.getWriteMode());
+ assertEquals(expected.getMaxOpenPartitions(),
actual.getMaxOpenPartitions());
}
}
diff --git
a/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetVersionTest.java
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetVersionTest.java
index 5a0ce6e236..5329373527 100644
---
a/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetVersionTest.java
+++
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetVersionTest.java
@@ -42,6 +42,12 @@ class ParquetVersionTest {
void testGetVersionFromDescription() {
assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription("Parquet 1.0"));
assertEquals(ParquetVersion.Version2,
ParquetVersion.getVersionFromDescription("Parquet 2.0"));
+ assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription("Version1"));
+ assertEquals(ParquetVersion.Version2,
ParquetVersion.getVersionFromDescription("Version2"));
+ assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription("1.0"));
+ assertEquals(ParquetVersion.Version2,
ParquetVersion.getVersionFromDescription("2.0"));
assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription("unknown"));
+ assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription(""));
+ assertEquals(ParquetVersion.Version1,
ParquetVersion.getVersionFromDescription(null));
}
}
diff --git
a/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetWriteModeTest.java
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetWriteModeTest.java
new file mode 100644
index 0000000000..fc7faec779
--- /dev/null
+++
b/plugins/tech/parquet/src/test/java/org/apache/hop/parquet/transforms/output/ParquetWriteModeTest.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.parquet.transforms.output;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import org.junit.jupiter.api.Test;
+
+/** Unit test for {@link ParquetWriteMode} */
+class ParquetWriteModeTest {
+
+ @Test
+ void testGetModeFromDescription() {
+ assertEquals(ParquetWriteMode.Append,
ParquetWriteMode.getModeFromDescription("append"));
+ assertEquals(
+ ParquetWriteMode.OverwritePartitions,
+ ParquetWriteMode.getModeFromDescription("overwrite_partitions"));
+ assertEquals(
+ ParquetWriteMode.FailIfExists,
ParquetWriteMode.getModeFromDescription("fail_if_exists"));
+ assertEquals(
+ ParquetWriteMode.OverwriteAll,
ParquetWriteMode.getModeFromDescription("overwrite_all"));
+ assertEquals(
+ ParquetWriteMode.OverwritePartitions,
+ ParquetWriteMode.getModeFromDescription("OverwritePartitions"));
+ assertEquals(ParquetWriteMode.Append,
ParquetWriteMode.getModeFromDescription("unknown"));
+ assertEquals(ParquetWriteMode.Append,
ParquetWriteMode.getModeFromDescription(""));
+ assertEquals(ParquetWriteMode.Append,
ParquetWriteMode.getModeFromDescription(null));
+ }
+}
