cshuo opened a new pull request, #20761:
URL: https://github.com/apache/hudi/pull/20761

   ### Describe the issue this Pull Request addresses
   
   Closes https://github.com/apache/hudi/issues/20606.
   
   With the Spark Connect backend, `query_lakehouse` loses information when 
converting collected rows to JSON. Structs lose their field names, `TIMESTAMP` 
and `TIMESTAMP_NTZ` both appear without a UTC offset, and clients cannot 
distinguish a decimal value such as `"1.10"` from a string with the same text.
   
   ### Summary and Changelog
   
   #### feat(gateway): preserve Spark Connect query types in JSON results 
(`b16f437d901`)
   
   - Encode Arrow values using the schema returned by the same Spark execution. 
Include `column_types` in column order, even for empty results, and apply the 
existing response byte limit to both rows and type metadata.
   - Return named structs as JSON objects, including inside arrays. When field 
names repeat, return values in field order so both values survive.
   - Preserve decimal precision and scale with exact strings. Encode 
`TIMESTAMP` as ISO-8601 in UTC with `+00:00` and `TIMESTAMP_NTZ` as local date 
and time without an offset.
   - Encode binary values as base64. Maps with string or integer keys use JSON 
objects, matching Trino for integer keys. Other map keys use `[key, value]` 
pairs so clients can read structured keys.
   
   ### Impact
   
   The response format changes for Spark Connect clients, including MCP 
clients. For example, a struct previously returned as `["Beijing", "South 
Station"]` now appears as `{"city":"Beijing","station":"South Station"}`. 
Clients that accessed named struct fields by position should use field names.
   
   ### Risk Level
   
   Medium. Existing clients may depend on positional struct values or the 
previous timestamp format. All 797 gateway offline tests passed. On Spark 
Connect 4.1.3, five live tests passed, including checks for timestamps across a 
daylight-saving transition and distinct `BIGINT` map keys above JavaScript’s 
safe integer range. Four tests were skipped because their environment 
prerequisites were unavailable.
   
   ### Documentation Update
   
   The gateway README documents `column_types`, decimal and timestamp 
encodings, and the JSON differences between Spark Connect and Trino for 
structs, binary values, and maps.
   
   ### Contributor's checklist
   
   - [ ] Read through the [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to