This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 3b95f424fe4 Speed up JSON-heavy ingestion transforms: non-JSON
fast-path + jsonExtractObject parse-once helper (#18711)
3b95f424fe4 is described below
commit 3b95f424fe4de2d1d6aaf6d0160053c60541ea0c
Author: Xiang Fu <[email protected]>
AuthorDate: Tue Jun 9 12:28:37 2026 -0700
Speed up JSON-heavy ingestion transforms: non-JSON fast-path +
jsonExtractObject parse-once helper (#18711)
* Skip JSON parse for non-JSON input in jsonPath* default-value functions
The default-value overloads of the jsonPath* scalar functions
(jsonPathString/Long/Double(..., default) and jsonPathArrayDefaultEmpty)
parse their argument with Jackson and catch any exception to return the
default. For columns that are sometimes structured JSON and sometimes
plain text (e.g. a log `message` field), every plain-text value throws a
JsonParseException whose fillInStackTrace() dominates the ingestion hot
path. On one realtime log table, filling these stack traces accounted for
roughly 30% of consume-loop CPU.
Add a conservative pre-check (canExtractJsonPath) that returns the
caller's default without invoking the parser when the input is null or a
string that cannot begin a JSON value. Inputs that could begin a JSON
value (object/array/string/number/true/false/null literal) are still
handed to the parser, so behavior - including the exceptions raised by the
throwing two-arg overloads, which are intentionally left unchanged - is
preserved. The change is behavior-preserving: the skipped inputs already
returned the default, just via a thrown-and-caught exception.
Validated with a JMH benchmark of the realtime transform pipeline: the
fix alone raises transform throughput ~1.7x on a JSON log-ingestion
workload (and ~3.3x combined with parse-once transform configs).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Add jsonExtractObject for parse-once JSON ingestion transforms
Add a null-safe scalar that parses a JSON document into a reusable
Map/List once (or passes through an already-parsed container), returning
null - without throwing - for null, scalar, or non-JSON input. It
delegates to the existing jsonStringToListOrMap for strings and
generalizes it to accept Object so it also handles values the decoder or
an upstream transform already parsed.
This enables a "parse-once" transform pattern: materialize an
intermediate object column once and point many JSONPATHSTRING extractions
at it (jsonPath navigates the returned object without re-parsing), instead
of re-parsing the same source document for every extracted field. Because
it never throws, it is safe for columns that are only sometimes JSON
(e.g. a log `message` field) - plain-text rows yield null and fall
through to the downstream default.
Measured on a realtime log-ingestion workload (message JSON ~25% of the
time): parsing `message` once via jsonExtractObject instead of per-field
lifts transform throughput ~7% and end-to-end (transform + index, schema
unchanged) ~9% over re-parsing per field.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
.../common/function/scalar/JsonFunctions.java | 73 +++++++++++++++++-
.../pinot/common/function/JsonFunctionsTest.java | 87 ++++++++++++++++++++++
2 files changed, 159 insertions(+), 1 deletion(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java
b/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java
index 280ab25fb0b..2a5b3516630 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java
@@ -105,6 +105,38 @@ public class JsonFunctions {
return PARSE_CONTEXT.parse(object).read(jsonPath, NO_PREDICATES);
}
+ /**
+ * Returns {@code false} when the input is known to be non-extractable by a
json path without invoking the JSON
+ * parser: a {@code null} value, or a string whose first non-whitespace
character cannot begin a JSON value. Used
+ * by the default-value {@code jsonPath*} overloads to skip parsing
plain-text input (e.g. raw log lines) that
+ * would otherwise throw a {@link com.jayway.jsonpath.InvalidJsonException}
whose {@code fillInStackTrace()}
+ * dominates the hot ingestion path. It is intentionally conservative: any
string that could begin a JSON value
+ * (object, array, string, number, or a {@code true}/{@code false}/{@code
null} literal) is still handed to the
+ * parser, so the parser's behavior - including any exception it raises - is
unchanged for those inputs. Returning
+ * the caller's default for the skipped inputs is equivalent to the prior
behavior, where the parse exception was
+ * caught and the default returned.
+ */
+ private static boolean canExtractJsonPath(@Nullable Object object) {
+ if (object == null) {
+ return false;
+ }
+ if (!(object instanceof String)) {
+ // Already-parsed Map/List/etc. - handled by jsonPath() directly without
re-parsing.
+ return true;
+ }
+ String s = (String) object;
+ for (int i = 0, n = s.length(); i < n; i++) {
+ char c = s.charAt(i);
+ if (c == ' ' || c == '\t' || c == '\n' || c == '\r') {
+ continue;
+ }
+ // First non-whitespace character of any valid JSON value.
+ return c == '{' || c == '[' || c == '"' || c == '-' || (c >= '0' && c <=
'9') || c == 't' || c == 'f'
+ || c == 'n';
+ }
+ return false; // empty / all-whitespace is not valid JSON
+ }
+
/**
* Extract object array based on Json path
*/
@@ -119,8 +151,11 @@ public class JsonFunctions {
@ScalarFunction(nullableParameters = true)
public static Object[] jsonPathArrayDefaultEmpty(@Nullable Object object,
String jsonPath) {
+ if (!canExtractJsonPath(object)) {
+ return EMPTY;
+ }
try {
- Object[] result = object == null ? null : jsonPathArray(object,
jsonPath);
+ Object[] result = jsonPathArray(object, jsonPath);
return result == null ? EMPTY : result;
} catch (Exception e) {
return EMPTY;
@@ -172,6 +207,9 @@ public class JsonFunctions {
*/
@ScalarFunction(nullableParameters = true)
public static String jsonPathString(@Nullable Object object, String
jsonPath, String defaultValue) {
+ if (!canExtractJsonPath(object)) {
+ return defaultValue;
+ }
try {
Object jsonValue = jsonPath(object, jsonPath);
if (jsonValue instanceof String) {
@@ -196,6 +234,9 @@ public class JsonFunctions {
*/
@ScalarFunction(nullableParameters = true)
public static long jsonPathLong(@Nullable Object object, String jsonPath,
long defaultValue) {
+ if (!canExtractJsonPath(object)) {
+ return defaultValue;
+ }
try {
Object jsonValue = jsonPath(object, jsonPath);
if (jsonValue == null) {
@@ -223,6 +264,9 @@ public class JsonFunctions {
*/
@ScalarFunction(nullableParameters = true)
public static double jsonPathDouble(@Nullable Object object, String
jsonPath, double defaultValue) {
+ if (!canExtractJsonPath(object)) {
+ return defaultValue;
+ }
try {
Object jsonValue = jsonPath(object, jsonPath);
if (jsonValue == null) {
@@ -323,6 +367,33 @@ public class JsonFunctions {
return null;
}
+ /**
+ * Parse a JSON document into a reusable {@code Map}/{@code List}, or pass
through an already-parsed container,
+ * returning {@code null} for null, scalar, or non-JSON input <b>without
throwing</b>.
+ * <p>Intended for "parse-once" ingestion transforms: materialize an
intermediate object column once and point
+ * multiple {@code jsonPath*} extractions at it, instead of re-parsing the
same source document for every extracted
+ * field. {@link #jsonPath} navigates the returned object directly (no
re-parse), and because this never throws it
+ * is safe for columns that are sometimes structured JSON and sometimes
plain text (e.g. a log {@code message}
+ * field) - the plain-text rows simply yield {@code null} and fall through
to the default of the downstream
+ * extraction. Example transform configs:
+ * <pre>
+ * {"columnName": "message_obj", "transformFunction":
"jsonExtractObject(message)"}
+ * {"columnName": "level", "transformFunction":
"JSONPATHSTRING(message_obj, '$.level', null)"}
+ * </pre>
+ */
+ @Nullable
+ @ScalarFunction(nullableParameters = true)
+ public static Object jsonExtractObject(@Nullable Object object) {
+ if (object instanceof String) {
+ return jsonStringToListOrMap((String) object);
+ }
+ if (object instanceof Map || object instanceof List || object instanceof
Object[]) {
+ // Already parsed (e.g. a nested object surfaced by the decoder or an
upstream transform) - reuse as-is.
+ return object;
+ }
+ return null;
+ }
+
private static void setValuesToMap(String keyColumnName, String
valueColumnName, Object obj,
Map<String, String> result) {
if (obj instanceof Map) {
diff --git
a/pinot-common/src/test/java/org/apache/pinot/common/function/JsonFunctionsTest.java
b/pinot-common/src/test/java/org/apache/pinot/common/function/JsonFunctionsTest.java
index 39ea369c710..96d83b95119 100644
---
a/pinot-common/src/test/java/org/apache/pinot/common/function/JsonFunctionsTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/common/function/JsonFunctionsTest.java
@@ -34,6 +34,7 @@ import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
import static org.testng.Assert.assertTrue;
@@ -156,6 +157,92 @@ public class JsonFunctionsTest {
assertEquals(JsonFunctions.jsonPathString("{\"foo\": \"null\"}", "$.foo",
"default"), "null");
}
+ /**
+ * The default-value {@code jsonPath*} overloads return the caller's default
for input that cannot begin a JSON
+ * value (plain text, null, empty) without invoking the parser - this avoids
the {@code fillInStackTrace()} cost
+ * of a thrown-and-caught {@link InvalidJsonException} on the ingestion hot
path. Behavior is unchanged for valid
+ * JSON and for any input that could begin a JSON value (which is still
handed to the parser).
+ */
+ @Test
+ public void testJsonPathDefaultVariantsSkipNonJson() {
+ // Plain-text input (e.g. a raw log line) -> default, no exception thrown.
+ assertEquals(JsonFunctions.jsonPathString("INFO 2026-06-08 request done",
"$.level", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString("INFO 2026-06-08 request done",
"$", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathLong("ERROR boom code", "$.code", -1L),
-1L);
+ assertEquals(JsonFunctions.jsonPathDouble("WARN slow request",
"$.latency", -1.0), -1.0, 0.0);
+ assertEquals(JsonFunctions.jsonPathArrayDefaultEmpty("plain text line",
"$.items").length, 0);
+
+ // null input -> default.
+ assertEquals(JsonFunctions.jsonPathString(null, "$.level", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathLong(null, "$.code", -1L), -1L);
+ assertEquals(JsonFunctions.jsonPathArrayDefaultEmpty(null,
"$.items").length, 0);
+
+ // Empty / whitespace-only input -> default.
+ assertEquals(JsonFunctions.jsonPathString("", "$.x", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString(" ", "$.x", "def"), "def");
+
+ // Valid JSON object/array is still extracted, including with leading
whitespace.
+ assertEquals(JsonFunctions.jsonPathString("{\"level\":\"info\"}",
"$.level", "def"), "info");
+ assertEquals(JsonFunctions.jsonPathString(" \n {\"level\":\"info\"}",
"$.level", "def"), "info");
+ assertEquals(JsonFunctions.jsonPathString("[{\"k\":\"v\"}]", "$[0].k",
"def"), "v");
+
+ // Inputs that could begin a JSON value are still handed to the parser, so
behavior is unchanged:
+ // "not json" begins with 'n' (the `null` literal) -> parsed, fails,
default returned.
+ assertEquals(JsonFunctions.jsonPathString("not json", "$.x", "def"),
"def");
+ // Bare JSON scalars are parsed and returned for the whole-document path
(preserved behavior).
+ assertEquals(JsonFunctions.jsonPathString("12345", "$", "def"), "12345");
+ assertEquals(JsonFunctions.jsonPathString("\"hello\"", "$", "def"),
"hello");
+
+ // Inputs the fast-path skips that the parser also rejected before ->
default (equivalence preserved at the
+ // exact char boundary: these do not begin with {, [, ", -, a digit, or a
true/false/null literal).
+ assertEquals(JsonFunctions.jsonPathString("+5", "$", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString(".5", "$", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString("NaN", "$", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString("Infinity", "$", "def"), "def");
+ assertEquals(JsonFunctions.jsonPathString("'single quoted'", "$.x",
"def"), "def");
+ }
+
+ /**
+ * {@code jsonExtractObject} parses a JSON document once into a reusable
Map/List (or passes through an
+ * already-parsed container), returns null without throwing for
null/scalar/non-JSON input, and the returned object
+ * is navigable by {@code jsonPath*} with results identical to parsing the
raw string - enabling parse-once
+ * transform configs.
+ */
+ @Test
+ public void testJsonExtractObject()
+ throws Exception {
+ // JSON object -> reusable Map, navigable by jsonPath without re-parsing.
+ Object obj =
JsonFunctions.jsonExtractObject("{\"level\":\"info\",\"log\":{\"msg\":\"hi\"}}");
+ assertTrue(obj instanceof Map);
+ assertEquals(JsonFunctions.jsonPathString(obj, "$.log.msg", "def"), "hi");
+ assertEquals(JsonFunctions.jsonPathString(obj, "$.level", "def"), "info");
+
+ // JSON array -> reusable List.
+ Object arr = JsonFunctions.jsonExtractObject("[{\"k\":\"v\"}]");
+ assertTrue(arr instanceof List);
+ assertEquals(JsonFunctions.jsonPathString(arr, "$[0].k", "def"), "v");
+
+ // null / scalar / plain-text / empty -> null, no exception.
+ assertNull(JsonFunctions.jsonExtractObject(null));
+ assertNull(JsonFunctions.jsonExtractObject("INFO 2026-06-08 plain text log
line"));
+ assertNull(JsonFunctions.jsonExtractObject("12345"));
+ assertNull(JsonFunctions.jsonExtractObject(""));
+
+ // Already-parsed container -> passed through unchanged (Map and
Object[]), still navigable by jsonPath.
+ Map<String, Object> map = Map.of("a", "b");
+ assertEquals(JsonFunctions.jsonExtractObject(map), map);
+ Object[] preParsed = new Object[]{Map.of("k", "v")};
+ assertSame(JsonFunctions.jsonExtractObject(preParsed), preParsed);
+ assertEquals(JsonFunctions.jsonPathString(preParsed, "$[0].k", "def"),
"v");
+
+ // Parse-once equivalence: extracting through the parsed object matches
extracting from the raw string.
+ String json =
"{\"labels\":{\"service_name\":\"svc-1\",\"environment\":\"prod\"},\"id\":\"abc\"}";
+ Object parsed = JsonFunctions.jsonExtractObject(json);
+ assertEquals(JsonFunctions.jsonPathString(parsed, "$.labels.service_name",
"def"),
+ JsonFunctions.jsonPathString(json, "$.labels.service_name", "def"));
+ assertEquals(JsonFunctions.jsonPathString(parsed, "$.id", "def"), "abc");
+ }
+
@Test
public void testJsonFunctionExtractingArray()
throws JsonProcessingException {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
