bamaer opened a new issue, #8791: URL: https://github.com/apache/hop/issues/8791
### What would you like to happen? Follow-up to #2582 / #8748. The Mask fields transform works, but its `DATABASE` storage mode and its error handling have problems that matter when it's used for GDPR work. ### Problems - **Original values are stored in plain text.** The mapping table keeps each input value in `source_key`, so the table can be used to re-identify people. That column is also limited to 255 characters, so longer values fail or get truncated, depending on the database. - **Error rows can contain unmasked PII.** `MaskingEngine.apply` changes the row one field at a time. If one field fails, `putError` sends a row where the fields before it are masked and that field and the ones after it are not. - **Two people can get the same masked value.** `allocateSequence` reads `next_value` and then writes it back plus one, without locking the row. Two Hop processes using the same table can read the same number. Nothing prevents the duplicate: there is no unique constraint on `(pattern_name, masked_value)`. - **Stores can be shared across different databases.** `MaskingRuntime` shares stores by connection name, schema and table only. On Hop Server, two projects with a connection of the same name (or different variable values) share whichever store opened first. - **The lookup key depends on formatting.** The key is the value's formatted text (`valueMeta.getString`), so it changes with a number's conversion mask. Values that differ only in case or spaces get different masked values. ### Proposed changes - [ ] Store an HMAC-SHA256 of the input value instead of the value itself, with the secret in a password field or variable on the pattern. Keep reading existing tables that hold plain-text keys, or document how to migrate them. - [ ] Mask a copy of the row, or null out the remaining configured fields, before sending a row to error handling. - [ ] Reserve sequence numbers safely: increment first and then read the new value, or use `SELECT … FOR UPDATE`. Add a unique constraint on the masked value. - [ ] Share stores by the resolved connection (URL and user), not by the connection name. - [ ] Add optional trim / ignore-case settings to the pattern, and build the key from the raw value instead of its formatted text. ### Out of scope (separate issues) - Partial masking, hashing as an output type, date shifting. - Performance of the database store (batching, reserving blocks of numbers). ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
