xiangfu0 commented on code in PR #18953: URL: https://github.com/apache/pinot/pull/18953#discussion_r3556958596
########## pinot-plugins/pinot-input-format/pinot-json/src/main/java/org/apache/pinot/plugin/inputformat/json/format/SqliteJsonbPayloadParser.java: ########## @@ -0,0 +1,383 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.plugin.inputformat.json.format; + +import com.google.common.collect.Maps; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + + +/// Parses the <a href="https://sqlite.org/jsonb.html";>SQLite JSONB</a> binary format (SQLite 3.45+). +/// +/// Every element is a 1–9 byte header followed by a payload. The header's first byte packs the element type in +/// the low nibble and a payload-size descriptor in the high nibble; descriptors 12–15 mean the size is a big- +/// endian `uint8` / `uint16` / `uint32` / `uint64` in the following bytes, and 0–11 are the size itself. +/// +/// Numbers are stored as their ASCII text, so integers narrow to `Integer` / `Long` / `BigInteger` and floats +/// to `Double`, matching the text-JSON value contract. `TEXTJ` strings/labels are JSON-unescaped (invalid +/// escapes are rejected); `TEXT5` additionally accepts JSON5 escapes. +/// +/// The decoder expects one row per payload, so the top-level element must be an `OBJECT`. +class SqliteJsonbPayloadParser implements JsonPayloadParser { + + // Element types (low nibble of the header's first byte). + private static final int TYPE_NULL = 0; + private static final int TYPE_TRUE = 1; + private static final int TYPE_FALSE = 2; + private static final int TYPE_INT = 3; // canonical decimal integer, ASCII + private static final int TYPE_INT5 = 4; // JSON5 integer (hex / leading sign), ASCII + private static final int TYPE_FLOAT = 5; // canonical float, ASCII + private static final int TYPE_FLOAT5 = 6; // JSON5 float, ASCII + private static final int TYPE_TEXT = 7; // raw text, no escapes + private static final int TYPE_TEXTJ = 8; // text with JSON escapes + private static final int TYPE_TEXT5 = 9; // text with JSON5 escapes + private static final int TYPE_TEXTRAW = 10; // raw text needing quoting, no escapes + private static final int TYPE_ARRAY = 11; + private static final int TYPE_OBJECT = 12; + + private static final BigInteger INT_MIN = BigInteger.valueOf(Integer.MIN_VALUE); + private static final BigInteger INT_MAX = BigInteger.valueOf(Integer.MAX_VALUE); + private static final BigInteger LONG_MIN = BigInteger.valueOf(Long.MIN_VALUE); + private static final BigInteger LONG_MAX = BigInteger.valueOf(Long.MAX_VALUE); + + @Override + public boolean matches(byte[] payload, int offset, int length) { + // Top-level element must be an OBJECT (low nibble == 12) to produce a row. This never collides with text + // JSON, whose first byte is '{' (0x7B) or '[' (0x5B) -- both low nibble 0x0B (ARRAY), never 0x0C. + return length >= 1 && (payload[offset] & 0x0F) == TYPE_OBJECT; + } + + @Override + public Map<String, Object> parse(byte[] payload, int offset, int length) { + int limit = offset + length; + Cursor cursor = new Cursor(payload, offset, limit); + Object value = readElement(cursor, limit); Review Comment: Fixed in `4b7401f` — `parse()` enforces SQLite's exact-fill rule (rejects a short-declared top-level element and trailing bytes), with regression tests at the parser and `JSONMessageDecoder` levels. `68af742` additionally moves the same rule into `matches()`, so AUTO no longer claims such a payload at all. _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
