tomnewton opened a new issue, #39840:
URL: https://github.com/apache/beam/issues/39840
### What would you like to happen?
### Problem
ClickHouseIO has no notion of ClickHouse's fixed-point `Decimal` family —
`Decimal(P, S)`,
`Decimal(P)`, bare `Decimal`, and the width aliases `Decimal32(S)`,
`Decimal64(S)`,
`Decimal128(S)`, `Decimal256(S)`. The column-type parser rejects these type
strings, so
`ClickHouseIO.getTableSchema()` throws on the `DESCRIBE TABLE` output and
pipelines cannot
write to any table containing a Decimal column — a blocker for monetary and
other
fixed-point data.
### Proposed solution
1. **Schema model**
- Add `TypeName.DECIMAL`.
- Extend `ColumnType` with a `scale` field alongside the existing
`precision` field
(introduced for `DateTime64`), plus a validated factory
`ColumnType.decimal(int precision, int scale)` (P in [1, 76], S in [0,
P]).
2. **Parser**
- Grammar rules for `Decimal[(P[, S])]` (bare `Decimal` = `Decimal(10,
0)`,
`Decimal(P)` = `Decimal(P, 0)`, matching ClickHouse) and the width
aliases
`Decimal32/64/128/256(S)` (precisions 9/18/38/76), reachable through
`Nullable(...)` and `Array(...)`.
3. **Beam field type mapping**
- `Decimal(P, S)` maps to the `FixedPrecisionNumeric.of(P, S)` logical
type
(base type `DECIMAL`, values are `BigDecimal`), preserving the declared
precision/scale through `getEquivalentSchema` — the same logical type
JdbcIO
uses for NUMERIC columns.
4. **Writer**
- Serialize as ClickHouse's RowBinary representation: the unscaled value
as a
little-endian signed integer whose width (32/64/128/256 bits) is chosen
from the
declared precision, exactly per ClickHouse's `Decimal(P, S)` →
`Decimal32/64/128/256`
synonym rule.
5. **Defaults**
- Support `DEFAULT <literal>` on Decimal columns in
`ColumnType.parseDefaultExpression` — without it, schema fetch fails
for any table
with a defaulted Decimal column.
6. **Tests**
- Parser/mapping unit tests, byte-level writer tests, and round-trip
tests against the
ClickHouse test container for all four width buckets,
`Nullable(Decimal)`,
`Array(Decimal)`, DEFAULT handling, and `getTableSchema` on the
server's canonicalized
`DESCRIBE` output.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
