xiangfu0 opened a new pull request, #18950:
URL: https://github.com/apache/pinot/pull/18950
## Summary
Adds a new `pinot-bson` input-format plugin for reading MongoDB **BSON**
data, for both batch and streaming ingestion. It mirrors the structure of the
existing `pinot-json` plugin (reader + shared extractor + stream decoder) since
a decoded BSON document (`org.bson.Document`) is itself a `Map<String, Object>`.
## What's included
- **`BSONRecordReader`** — batch reader for `mongodump`-style files (a
concatenation of framed, length-prefixed BSON documents). GZIP-compressed files
are supported transparently. A corrupt document surfaces as a skippable
`IOException` so a single bad record doesn't abort the whole segment.
- **`BSONMessageDecoder`** — stream decoder for one BSON document per
message (e.g. a MongoDB change-stream/oplog document forwarded to Kafka).
- **`BSONRecordExtractor`** — maps BSON's type system onto the standard
`RecordExtractor` contract, so the downstream data-type transformer coerces
values to the declared column type:
| BSON type | Output |
|---|---|
| Double / Int32 / Int64 / Boolean / String | boxed pass-through |
| Document (embedded) | `Map<String, Object>` |
| Array | `Object[]` |
| ObjectId | `String` (24-char hex) |
| DateTime | `java.sql.Timestamp` |
| Decimal128 | `BigDecimal` (`NaN`/`Infinity` → `null`) |
| Binary | `byte[]` |
| other (rare/deprecated/internal) | `toString()` fallback |
- **Registration**: `BSON` added to the `FileFormat` enum and
`RecordReaderFactory`, so `dataFormat: bson` works in batch ingestion without
specifying a reader class.
Uses the standalone `org.mongodb:bson` library — **no** MongoDB driver and
**no** network dependency.
## Usage
Batch ingestion job spec:
```yaml
recordReaderSpec:
dataFormat: 'bson'
className: 'org.apache.pinot.plugin.inputformat.bson.BSONRecordReader'
```
Stream (table) config:
```json
"stream.kafka.decoder.class.name":
"org.apache.pinot.plugin.inputformat.bson.BSONMessageDecoder"
```
## Testing
- `BSONRecordExtractorTest` (20) — scalars, ObjectId, Date, Decimal128
(incl. NaN/Infinity), Binary, arrays, nested documents, null handling, fallback.
- `BSONRecordReaderTest` (7) — extends the shared `AbstractRecordReaderTest`
(happy path + gzip + rewind over 10k records) plus raw-frame negative/recovery
cases (empty file, truncated prefix, invalid length, truncated body, and
skip-then-continue past a corrupt frame).
- `BSONMessageDecoderTest` (5) — scalars + logical types, nested doc/array,
field projection, offset/length decode, corrupt payload.
`checkstyle`, `license`, and `spotless` all pass.
## Release notes
New input format `bson` for reading MongoDB BSON data (batch `.bson` files
and streaming). Bundles the Apache-2.0 licensed `org.mongodb:bson` library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
