zeroshade commented on issue #46555:
URL: https://github.com/apache/arrow/issues/46555#issuecomment-2904995318
So, I used that `DecimalValue` type for how I return the decimal types:
```go
type DecimalValue[T decimal.DecimalTypes] struct {
Scale uint8
Value decimal.Num[T]
}
```
But the builder currently has:
```go
func (b *Builder) AppendDecimal4(scale uint8, v decimal.Decimal32) error
{...}
func (b *Builder) AppendDecimal8(scale uint8, v decimal.Decimal64) error
{...}
func (b *Builder) AppendDecimal16(scale uint8, v decimal.Decimal128) error
{...}
```
> ( can float/double cannot distinct it fits the smallest type? Perhaps not?
)
I'll try that, shouldn't be difficult to set it up for that.
> Also, I've list a part for output and internal buffer handling. Arena like
data structure would be used here. Previously I've think how to do think of
reusing buffer and making build array of Variant vectorization. But now I found
it's complex to vectorize building a so complex type. So maybe I should focous
on reuse buffer and decrease dynamic buffer allocation
Another reason why I avoided the "child builder" construction is to improve
buffer reuse and reduce dynamic allocation. There's a single buffer in the
builder that everything writes to. `FinishObject` / `FinishArray` are called,
the buffer is expanded and the size of the header is calculated. The raw data
is then shifted by that many bytes and the header is written in the new space.
That way we aren't creating a separate buffer for each child builder that we
then have to copy back to the main one (which is what
https://github.com/apache/parquet-java/blob/1f1e07bbf750fba228851c2d63470c3da5726831/parquet-variant/src/main/java/org/apache/parquet/variant/VariantObjectBuilder.java
does)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
