bamaer commented on code in PR #9446:
URL: https://github.com/apache/hop/pull/9446#discussion_r4554986287
##########
plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMeta.java:
##########
@@ -184,16 +243,28 @@ public void getFields(
IHopMetadataProvider metadataProvider)
throws HopTransformException {
+ List<NormaliserField> fields = getFieldsToNormalise(row);
+
int maxlen = 0;
- for (NormaliserField field : normaliserFields) {
+ for (NormaliserField field : fields) {
if (field.getValue().length() > maxlen) {
maxlen = field.getValue().length();
}
}
- // Take the normalised fields from the input before adding anything to it.
+ // Take the normalised fields from the input before changing it.
//
- List<IValueMeta> normalisedValueMetas = getNormalisedValueMetas(row);
+ List<IValueMeta> normalisedValueMetas = getNormalisedValueMetas(row,
fields);
+
+ // Remove the fields that get normalised before adding the new ones: a new
field may have the
+ // name of one of them, value for example, and would otherwise be renamed
to value_1.
+ //
+ for (NormaliserField field : fields) {
+ int idx = row.indexOfValue(field.getName());
+ if (idx >= 0) {
+ row.removeValueMeta(idx);
+ }
+ }
Review Comment:
**Blocker: this renames output fields of existing pipelines.**
Moving the removal before the add also changes the listed-fields case.
Normalising `amount` and `amount_q2` into new field `amount` gives `id,
quarter, amount_1` on main and `id, quarter, amount` with this PR. Transforms
downstream that use `amount_1` (Select values, Filter, Table output mappings,
...) break after the upgrade.
Suggested fix: remove first only in the new all-fields mode, which has no
old behaviour to keep, and keep the old order for listed fields:
```java
boolean allFields = isNormalisingAllFields();
if (allFields) {
removeFields(row, fields);
}
// ... add type field and normalised fields as now ...
if (!allFields) {
removeFields(row, fields);
}
```
The listed half of `aNewFieldKeepsItsNameWhenANormalisedFieldHadIt` then
expects `value_1`. I tried this locally and the other tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]