mengyejiang opened a new issue, #11068:
URL: https://github.com/apache/paimon/issues/11068

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   ### Paimon version
   
   1.2.0
   
   ### Engine and environment
   
   - Apache Flink 1.16.2
   - Java 8
   - Kafka source containing Oracle GoldenGate (OGG) JSON messages
   - Paimon action: kafka_sync_database
   
   ### Description
   
   When ingesting OGG JSON messages, JSON null values are converted to the 
literal string "null" during record parsing.
   
   For a STRING column, the value is therefore stored as the string "null" 
instead of SQL NULL.
   
   The affected code is AbstractJsonRecordParser.extractRowData(), which is 
used by OggRecordParser.
   
   ### Minimal example
   
   An OGG insert message:
   
   {
     "table": "TEST.CUSTOMER",
     "op_type": "I",
     "op_ts": "2026-10-15 09:13:37.019234",
     "current_ts": "2026-10-15T09:13:41.537004",
     "pos": "00000627270001984891",
     "primary_keys": ["ID"],
     "before": null,
     "after": {
       "ID": "1",
       "NAME": null,
       "EMPTY_VALUE": "",
       "LITERAL_NULL": "null"
     }
   }
   
   ### Expected behavior
   
   The parser should preserve the distinction between:
   
   - JSON null → Java null → SQL NULL
   - Empty string "" → empty string
   - Literal string "null" → literal string "null"
   
   ### Actual behavior
   
   The JSON null value in NAME is converted to the literal string "null".
   
   ### Root cause
   
   In AbstractJsonRecordParser.extractRowData(), the JSON record is converted 
to Map<String, Object>. JSON null is correctly represented as Java null at this 
stage.
   
   However, the subsequent conversion uses:
   
       Objects.toString(entry.getValue())
   
   Objects.toString(null) returns the literal string "null".
   
   Simply changing this expression to Objects.toString(value, null) is 
insufficient because Collectors.toMap does not accept null values.
   
   ### Proposed fix
   
   Use a map collection approach that supports null values and explicitly 
preserve Java null.
   
   Keep the existing behavior for non-null basic values and JSON serialization 
of complex values.
   
   Add regression tests covering:
   
   - JSON null
   - Empty strings
   - The literal string "null"
   
   I have prepared a local fix and a parser regression test on the 
release-1.2.0 baseline. The regression test passes.
   
   ### Target branch
   
   My goal is to contribute this fix to release-1.2.
   
   Would a PR directly targeting release-1.2 be accepted, or should the fix be 
submitted to master first and then backported?
   
   ### Willingness to contribute
   
   I am willing to submit a PR with the fix and regression tests.
   
   ### Compute Engine
   
   Apache Flink 1.16.2
   
   ### Minimal reproduce step
   
   1. Use Apache Paimon 1.2.0 with Flink 1.16.2 and Java 8.
   
   2. Create a Kafka topic named `ogg_null_test` and produce this OGG JSON 
message:
   
   ```json
   {
     "table": "TEST.CUSTOMER",
     "op_type": "I",
     "op_ts": "2026-10-15 09:13:37.019234",
     "current_ts": "2026-10-15T09:13:41.537004",
     "pos": "00000627270001984891",
     "primary_keys": ["ID"],
     "before": null,
     "after": {
       "ID": "1",
       "NAME": null,
       "EMPTY_VALUE": "",
       "LITERAL_NULL": "null"
     }
   }
   ```
   
   3. Run the Paimon `kafka_sync_database` action against this topic, using 
`value.format=ogg-json` and `scan.startup.mode=earliest-offset`.
   
   4. Wait for a successful checkpoint, then query the resulting Paimon table 
using Flink SQL:
   
   ```sql
   SELECT
       ID,
       NAME IS NULL AS name_is_null,
       NAME = 'null' AS name_is_literal_null,
       EMPTY_VALUE = '' AS empty_value_preserved,
       LITERAL_NULL = 'null' AS literal_null_preserved
   FROM customer
   WHERE ID = '1';
   ```
   
   The table and column names may need to be adjusted to match the configured 
case conversion.
   
   Expected:
   - `name_is_null`: true
   - `name_is_literal_null`: NULL
   - `empty_value_preserved`: true
   - `literal_null_preserved`: true
   
   Actual:
   - `name_is_null`: false
   - `name_is_literal_null`: true
   
   The JSON null in `NAME` is converted to the literal string `"null"` by 
`AbstractJsonRecordParser.extractRowData()`, which calls 
`Objects.toString(value)`.
   
   ### What doesn't meet your expectations?
   
   I expect JSON null values in OGG messages to be preserved as SQL NULL when 
written to Paimon.
   
   However, with Paimon 1.2.0, a source field such as `"NAME": null` is written 
to a STRING column as the literal string "null". Therefore, `NAME IS NULL` 
returns false, while `NAME = 'null'` returns true.
   
   The parser should preserve the distinction between JSON null, an empty 
string "", and the literal string "null".
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to