imbajin commented on code in PR #3192: URL: https://github.com/apache/hugegraph/pull/3192#discussion_r4092035686
########## hugegraph-server/hugegraph-dist/docker/yamlscan.awk: ########## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# yamlscan.awk -- does the top-level `authentication` mapping of a Gremlin +# server YAML file name an authenticator? Prints exactly one of: +# +# none there is no top-level authentication mapping +# nameless the mapping exists but names no authenticator class +# named the mapping names an authenticator class +# +# The entrypoint asks this one question to decide whether +# rest-server.properties and gremlin-server.yaml configure authentication +# together. Getting it wrong toward "named" is how REST ends up enforcing +# StandardAuthenticator while Gremlin silently falls back to TinkerPop +# AllowAllAuthenticator, so the answer has to follow the same structure +# snakeyaml hands to the server, within the subset of YAML that shipped and +# mounted configs use: +# +# 1. the mapping must start at column 0 -- an `authentication:` nested under +# some other key belongs to that feature, not to the Gremlin server; +# 2. only a direct child `authenticator` counts -- a class reached through +# `authentication.config`, or through any other nested mapping, is not +# the server authenticator, because TinkerPop keeps `config` as its own +# map; +# 3. `#` outside quotes starts a comment: text behind one is not content, +# and a comment-only line is neither a child nor the end of the mapping; +# 4. in a flow mapping the key must sit at depth one between the braces, so +# `{authenticator: X}` names a class while `{config: {authenticator: X}}` +# does not; +# 5. a direct `authenticator` whose value is empty, `null` or `~` names no +# class -- the server reads the key, gets nothing and leaves +# authentication off, which is the nameless case that must be refused. +# +# Quote characters come from sprintf so this file holds no literal apostrophe: +# an awk program written into a single-quoted shell string breaks on one, and +# that has cost this repo twice already. + +function apos() { return sprintf("%c", 39) } +function dquo() { return sprintf("%c", 34) } + +function ltrim(s) { sub(/^[ \t]+/, "", s); return s } +function rtrim(s) { sub(/[ \t]+$/, "", s); return s } +function trim(s) { return rtrim(ltrim(s)) } + +function is_quote(c) { return c == apos() || c == dquo() } + +# Remove an unquoted trailing comment together with the whitespace that has to +# precede the `#` for it to be a comment rather than part of a scalar. +function strip_comment(s, i, n, c, q, prev) { + q = "" + prev = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + } else if (is_quote(c)) { + q = c + } else if (c == "#" && (prev == "" || prev == " " || prev == "\t")) { + return rtrim(substr(s, 1, i - 1)) + } + prev = c + } + return s +} + +# How many whitespace characters open the line, i.e. its block nesting level. +function indent_of(s, i, n, c) { + n = length(s) + i = 1 + while (i <= n) { + c = substr(s, i, 1) + if (c != " " && c != "\t") break + i++ + } + return i - 1 +} + +# One layer of matching quotes off a key or scalar. +function unquote(s, f) { + s = trim(s) + if (length(s) >= 2) { + f = substr(s, 1, 1) + if ((f == apos() || f == dquo()) && substr(s, length(s), 1) == f) + return substr(s, 2, length(s) - 2) + } + return s +} + +# Split `name: value` at the first colon outside quotes that is followed by end +# of line or a space, which is what makes a colon inside `http://host` part of +# the scalar. Results go to K_TXT / V_TXT because awk returns one value. +function split_pair(s, i, n, c, q) { + q = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + continue + } + if (is_quote(c)) { q = c; continue } + if (c != ":") continue + if (i == n || substr(s, i + 1, 1) ~ /^[ \t]/) { Review Comment: ⚠️ `split_pair()` only recognizes a colon followed by end of line, space, or tab. On CRLF input, the top-level line is `authentication:\r`, so this check misses it; its indented children are skipped and `yamlscan.awk` reports `none`. With Gremlin authentication present and REST authentication absent, `check_auth_sides()` accepts the config and leaves REST open. Strip the carriage return before parsing and add a CRLF parity case. ########## hugegraph-server/hugegraph-dist/docker/yamlscan.awk: ########## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# yamlscan.awk -- does the top-level `authentication` mapping of a Gremlin +# server YAML file name an authenticator? Prints exactly one of: +# +# none there is no top-level authentication mapping +# nameless the mapping exists but names no authenticator class +# named the mapping names an authenticator class +# +# The entrypoint asks this one question to decide whether +# rest-server.properties and gremlin-server.yaml configure authentication +# together. Getting it wrong toward "named" is how REST ends up enforcing +# StandardAuthenticator while Gremlin silently falls back to TinkerPop +# AllowAllAuthenticator, so the answer has to follow the same structure +# snakeyaml hands to the server, within the subset of YAML that shipped and +# mounted configs use: +# +# 1. the mapping must start at column 0 -- an `authentication:` nested under +# some other key belongs to that feature, not to the Gremlin server; +# 2. only a direct child `authenticator` counts -- a class reached through +# `authentication.config`, or through any other nested mapping, is not +# the server authenticator, because TinkerPop keeps `config` as its own +# map; +# 3. `#` outside quotes starts a comment: text behind one is not content, +# and a comment-only line is neither a child nor the end of the mapping; +# 4. in a flow mapping the key must sit at depth one between the braces, so +# `{authenticator: X}` names a class while `{config: {authenticator: X}}` +# does not; +# 5. a direct `authenticator` whose value is empty, `null` or `~` names no +# class -- the server reads the key, gets nothing and leaves +# authentication off, which is the nameless case that must be refused. +# +# Quote characters come from sprintf so this file holds no literal apostrophe: +# an awk program written into a single-quoted shell string breaks on one, and +# that has cost this repo twice already. + +function apos() { return sprintf("%c", 39) } +function dquo() { return sprintf("%c", 34) } + +function ltrim(s) { sub(/^[ \t]+/, "", s); return s } +function rtrim(s) { sub(/[ \t]+$/, "", s); return s } +function trim(s) { return rtrim(ltrim(s)) } + +function is_quote(c) { return c == apos() || c == dquo() } + +# Remove an unquoted trailing comment together with the whitespace that has to +# precede the `#` for it to be a comment rather than part of a scalar. +function strip_comment(s, i, n, c, q, prev) { + q = "" + prev = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + } else if (is_quote(c)) { + q = c + } else if (c == "#" && (prev == "" || prev == " " || prev == "\t")) { + return rtrim(substr(s, 1, i - 1)) + } + prev = c + } + return s +} + +# How many whitespace characters open the line, i.e. its block nesting level. +function indent_of(s, i, n, c) { + n = length(s) + i = 1 + while (i <= n) { + c = substr(s, i, 1) + if (c != " " && c != "\t") break + i++ + } + return i - 1 +} + +# One layer of matching quotes off a key or scalar. +function unquote(s, f) { + s = trim(s) + if (length(s) >= 2) { + f = substr(s, 1, 1) + if ((f == apos() || f == dquo()) && substr(s, length(s), 1) == f) + return substr(s, 2, length(s) - 2) + } + return s +} + +# Split `name: value` at the first colon outside quotes that is followed by end +# of line or a space, which is what makes a colon inside `http://host` part of +# the scalar. Results go to K_TXT / V_TXT because awk returns one value. +function split_pair(s, i, n, c, q) { + q = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + continue + } + if (is_quote(c)) { q = c; continue } + if (c != ":") continue + if (i == n || substr(s, i + 1, 1) ~ /^[ \t]/) { + K_TXT = rtrim(substr(s, 1, i - 1)) + V_TXT = ltrim(substr(s, i + 1)) + return 1 + } + } + return 0 +} + +function names_authenticator(k) { return unquote(k) == "authenticator" } + +# An authenticator entry only counts when it actually names a class, and the +# answer has to be what snakeyaml hands the server rather than what the bytes +# look like. The unsafe direction is `named` for a config that leaves Gremlin +# on AllowAllAuthenticator while REST enforces, so anything this scanner cannot +# resolve to a class is refused instead of guessed at: +# +# - a plain scalar that resolves to null in any spelling, and YAML resolves +# null case-insensitively (null, Null, NULL, nUll) as well as to ~, names +# no class; +# - a leading `!` makes the tag, not the text, decide the type: !!null is the +# explicit spelling of empty and every other tag is a type not resolvable +# here, so neither counts; +# - a quoted scalar is a string and never null, but `""` and the empty single +# quoted form are the empty string, and loadAuthenticator("") returns null, +# which is the same no-authenticator state; +# - an unterminated quote is not a scalar at all. +function names_class(v, first, last, body) { + v = trim(v) + if (v == "") return 0 + first = substr(v, 1, 1) + if (first == "!") return 0 + if (first == apos() || first == dquo()) { + if (length(v) < 2) return 0 + last = substr(v, length(v), 1) + if (last != first) return 0 + body = trim(substr(v, 2, length(v) - 2)) + return body != "" + } + if (v == "~") return 0 + if (tolower(v) == "null") return 0 + return 1 +} + +# Report and stop. Output happens in END only, because awk runs END after +# `exit` and a second print there would emit two states on one run. +function finish(r) { RESULT = r; exit } + +# Feed one line of a flow collection to the brace scanner. DEPTH counts open +# collections; keys and values are only read at depth one, which is what makes +# a nested mapping under `config` invisible to it. FSET records a direct +# authenticator that names a class. Returns 1 once the outermost collection +# has closed. +function scan_flow(s, i, n, c, q, esc) { + n = length(s) + q = "" + esc = 0 + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + # Every byte inside the quotes belongs to the scalar, delimiters + # included; unquote and names_class take the quotes off. Dropping + # the value here is what made {authenticator: "org.A"} read as + # nameless and refuse a valid mounted config. A backslash escapes + # the next byte in a double quoted scalar only -- in a single + # quoted one the way out is a doubled quote, which this loop + # already gets right because the first one closes and the next + # reopens, and the pair still counts as content. + if (FST == "key") CUR = CUR c + else if (FST == "val") CUR_VAL = CUR_VAL c + if (esc) esc = 0 + else if (q == dquo() && c == "\\") esc = 1 + else if (c == q) q = "" + continue + } + if (is_quote(c)) { + q = c + if (FST == "key") CUR = CUR c + else if (FST == "val") CUR_VAL = CUR_VAL c + continue + } + if (c == "{" || c == "[") { + DEPTH++ + CUR = "" + # Past depth one the whole entry is nested content and is skipped, + # including an authenticator key inside it. + FST = (DEPTH == 1 ? "key" : "skip") + continue + } + if (c == "}" || c == "]") { + if (DEPTH == 1 && FST == "val") commit_val() + DEPTH-- + CUR = "" + if (DEPTH == 0) { FST = "key"; return 1 } + FST = "skip" + continue + } + if (DEPTH != 1) continue + if (c == ":") { + if (FST == "key") { + CUR_KEY = CUR + CUR_VAL = "" + FST = "val" + } + CUR = "" + continue + } + if (c == ",") { + if (FST == "val") commit_val() + FST = "key" + CUR = "" + continue + } + if (c == " " || c == "\t") { + # A space ends an unquoted key but never carries a value byte. + continue + } + if (FST == "key") CUR = CUR c + else if (FST == "val") CUR_VAL = CUR_VAL c + } + return 0 +} + +# Close out the depth-one entry that was being read when a `,` or `}` arrived. +function commit_val( k) { + k = CUR_KEY + if (names_authenticator(k) && names_class(CUR_VAL)) FSET = 1 +} + +BEGIN { + DEPTH = 0 + FST = "key" + CUR = "" + CUR_KEY = "" + CUR_VAL = "" + FSET = 0 + found = 0 + child = -1 + flow = 0 + RESULT = "" +} + +{ + line = strip_comment($0) + + if (!found) { + if (line ~ /^[ \t]/) next + if (!split_pair(line)) next + if (unquote(K_TXT) != "authentication") next + found = 1 + if (substr(V_TXT, 1, 1) == "{") { + flow = 1 + if (scan_flow(V_TXT)) finish(FSET ? "named" : "nameless") + next + } + # Anything else on the key line -- a scalar, a sequence, nothing -- is + # not a mapping that names a class. Reading `authentication: some.Name` + # as named would accept a config the server cannot use. + next + } + + if (flow) { + if (scan_flow(line)) finish(FSET ? "named" : "nameless") + next + } + + if (trim(line) == "") next + # A column-0 line after the comment was stripped is a sibling key, so the + # mapping has ended. + if (indent_of(line) == 0) finish("nameless") + + if (!split_pair(line)) next + if (child < 0) child = indent_of(line) + if (indent_of(line) != child) next + if (names_authenticator(K_TXT) && names_class(V_TXT)) finish("named") Review Comment: ‼️ A later duplicate direct child can replace the value this scanner has already accepted. In block YAML, `authenticator: org.apache.hugegraph.auth.StandardAuthenticator` followed by `authenticator: null` hits `finish("named")` on the first row even though SnakeYAML keeps the last value; the parity check can pass while Gremlin has no authenticator. Scan the full mapping with the server's last-wins behavior or reject duplicate keys, and cover block and flow forms. ########## hugegraph-server/hugegraph-dist/src/assembly/static/bin/props.awk: ########## @@ -0,0 +1,374 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# props.awk — read and rewrite Java ".properties" files with the grammar +# HugeConfig (commons-configuration over JDK Properties) applies, so the +# entrypoint and the server agree on what a mounted file means. grep/sed +# rewrites do not: they see `\`-escaped keys, `:` separators, continuation +# lines and duplicate definitions differently, which is how a mounted +# config ends up with two definitions of one key. +# +# One invocation, selected with the `PROPS_MODE` environment variable: +# +# PROPS_MODE=get PROPS_KEY=K PROPS_FILE=F +# print the value of K's first logical definition, in the on-disk +# escaped form; with PROPS_DECODED=1 print it as java.util.Properties +# would hand it to the server. Always exits 0. +# PROPS_MODE=has PROPS_KEY=K PROPS_FILE=F +# print nothing; exit 0 when K has any definition at all, empty +# included, 1 when it has none, 2 on an error +# PROPS_MODE=set PROPS_KEY=K PROPS_FILE=F +# replace K's first definition in place, drop every other +# definition of K, append one when the file has none. The new +# value arrives pre-encoded in PROPS_VALUE_ENCODED (an environment +# variable, so secrets never appear in `ps` output or in awk's +# argv), and -v is not used for it so awk cannot mangle its +# backslash escapes. +# +# Grammar implemented (java.util.Properties line reader + the +# first-definition-wins rule Configuration.getString applies): +# - physical lines end at \r\n, \n or a bare \r, as in java.util.Properties +# - '#' / '!' comments and blank lines +# - '=' / ':' / whitespace separators, where the whitespace Java counts is +# space, tab and form feed, with whitespace then an optional single '=' or +# ':' accepted as one separator +# - continuations: a physical line ending in an odd number of +# backslashes joins the next line (its leading whitespace stripped) +# - backslash escapes in keys and values, including \uXXXX +# - duplicate logical keys resolve to the first definition +# +# Rewrites keep every untouched line byte-for-byte (comments, blank +# lines, unrelated entries), and replace the first definition where it +# stands, so mounted configs stay reviewable in git diffs. + +function die(msg) { + printf "props.awk: %s\n", msg > "/dev/stderr" + # 2 for an error, so a caller that reads exit status 1 as "the key is not + # there" (PROPS_MODE=has) cannot mistake an unreadable file for an absent + # property and append a definition on top of one it failed to read. + exit 2 +} + +function hex_digit(c) { + return index("0123456789abcdef", tolower(c)) - 1 +} + +# \uXXXX is a UTF-16 code unit in Java. Values here are effectively +# ISO-8859-1, so codes above 0xFF are kept as their literal escape text +# rather than being mangled through a single-byte sprintf. +function unescape(s, out, i, n, c, code, j, d, ok) { + out = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (c != "\\") { out = out c; continue } + if (i == n) break + i++ + c = substr(s, i, 1) + if (c == "u" && i + 4 <= n) { + code = 0 + ok = 1 + for (j = 1; j <= 4; j++) { + d = hex_digit(substr(s, i + j, 1)) + if (d < 0) { ok = 0; break } + code = code * 16 + d + } + if (ok) { + i += 4 + if (code <= 255) out = out sprintf("%c", code) + else out = out substr(s, i - 5, 6) + continue + } + } + if (c == "t") out = out "\t" + else if (c == "n") out = out "\n" + else if (c == "r") out = out "\r" + else if (c == "f") out = out "\f" + else out = out c + } + return out +} + +# A physical line is continued when it ends in an odd number of +# backslashes (an even count escapes itself). +function trailing_backslashes(s, n, k) { + n = length(s) + k = 0 + while (k < n && substr(s, n - k, 1) == "\\") k++ + return k +} + +function is_skipped(raw) { + return raw ~ /^[ \t\f]*([#!]|$)/ +} + +# Split a logical line into its raw (still-escaped) key and value parts. +# Results land in K_RAW / V_RAW because awk returns one value. +# Java treats form feed as whitespace on both sides of the separator, so +# `auth.authenticator<FF>=...` is one property here too; reading it as part of +# the key name made a valid mounted configuration invisible to the guards. +function split_kv(s, n, i, c, esc, sep_at, rest) { + n = length(s) + esc = 0 + sep_at = 0 + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (esc) { esc = 0; continue } + if (c == "\\") { esc = 1; continue } + if (c == "=" || c == ":" || c == " " || c == "\t" || c == "\f") { sep_at = i; break } + } + if (sep_at == 0) { + K_RAW = s + V_RAW = "" + return + } + K_RAW = substr(s, 1, sep_at - 1) + rest = substr(s, sep_at) + c = substr(rest, 1, 1) + if (c == "=" || c == ":") { + rest = substr(rest, 2) + } else { + sub(/^[ \t\f]+/, "", rest) + c = substr(rest, 1, 1) + if (c == "=" || c == ":") rest = substr(rest, 2) + } + sub(/^[ \t\f]+/, "", rest) + V_RAW = rest +} + +function shquote(s) { + gsub(/'/, "'\\''", s) + return "'" s "'" +} + +# java.util.Properties ends a physical line at \r\n, \n or a bare \r, but +# getline splits on \n alone. A properties file saved with CR-only endings -- +# which java.util.Properties writes for a lone `store()` on some platforms, and +# which a mounted config can arrive with -- therefore reached the parser as one +# enormous record: only its first key was ever seen, and rewriting that key +# replaced the whole record and dropped every later entry, including +# auth.authenticator. So the file is re-scanned for terminators here. +# +# RAW[] keeps the exact bytes of each line and RAWTERM[] its terminator, so a +# rewrite still replays untouched lines byte-for-byte. A file whose last line +# carries no terminator gets a \n, which is what the replay did before. +function scan_records(s, i, n, c, start, term, len, cnt) { + n = length(s) + cnt = 0 + start = 1 + i = 1 + while (i <= n) { + c = substr(s, i, 1) + if (c != "\r" && c != "\n") { i++; continue } + if (c == "\r" && substr(s, i + 1, 1) == "\n") { + term = "\r\n" + len = 2 + } else { + term = c + len = 1 + } + cnt++ + RAW[cnt] = substr(s, start, i - start) + RAWTERM[cnt] = term + start = i + len + i = start + } + if (start <= n) { + cnt++ + RAW[cnt] = substr(s, start) + RAWTERM[cnt] = "" + } + return cnt +} + +# Load `file` into per-block arrays: one block per comment/blank line or +# logical entry, spanning exactly the physical lines it occupies. +function props_load(file, raw, rc, content, nl, stripped, next_raw, start, logical) { + content = "" + while ((rc = (getline raw < file)) > 0) Review Comment: ‼️ `props_load()` reads only `PROPS_FILE`, while `HugeConfig` loads properties through Commons Configuration with includes enabled by default. If the main file has `include=rest-auth.properties` and the included file defines `auth.authenticator`, this helper reports REST authentication as absent; with no YAML auth mapping, the entrypoint passes the parity check and starts a REST-protected/Gremlin-open server. Resolve includes consistently or fail closed when an include is present. ########## hugegraph-server/hugegraph-dist/docker/yamlscan.awk: ########## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# yamlscan.awk -- does the top-level `authentication` mapping of a Gremlin +# server YAML file name an authenticator? Prints exactly one of: +# +# none there is no top-level authentication mapping +# nameless the mapping exists but names no authenticator class +# named the mapping names an authenticator class +# +# The entrypoint asks this one question to decide whether +# rest-server.properties and gremlin-server.yaml configure authentication +# together. Getting it wrong toward "named" is how REST ends up enforcing +# StandardAuthenticator while Gremlin silently falls back to TinkerPop +# AllowAllAuthenticator, so the answer has to follow the same structure +# snakeyaml hands to the server, within the subset of YAML that shipped and +# mounted configs use: +# +# 1. the mapping must start at column 0 -- an `authentication:` nested under +# some other key belongs to that feature, not to the Gremlin server; +# 2. only a direct child `authenticator` counts -- a class reached through +# `authentication.config`, or through any other nested mapping, is not +# the server authenticator, because TinkerPop keeps `config` as its own +# map; +# 3. `#` outside quotes starts a comment: text behind one is not content, +# and a comment-only line is neither a child nor the end of the mapping; +# 4. in a flow mapping the key must sit at depth one between the braces, so +# `{authenticator: X}` names a class while `{config: {authenticator: X}}` +# does not; +# 5. a direct `authenticator` whose value is empty, `null` or `~` names no +# class -- the server reads the key, gets nothing and leaves +# authentication off, which is the nameless case that must be refused. +# +# Quote characters come from sprintf so this file holds no literal apostrophe: +# an awk program written into a single-quoted shell string breaks on one, and +# that has cost this repo twice already. + +function apos() { return sprintf("%c", 39) } +function dquo() { return sprintf("%c", 34) } + +function ltrim(s) { sub(/^[ \t]+/, "", s); return s } +function rtrim(s) { sub(/[ \t]+$/, "", s); return s } +function trim(s) { return rtrim(ltrim(s)) } + +function is_quote(c) { return c == apos() || c == dquo() } + +# Remove an unquoted trailing comment together with the whitespace that has to +# precede the `#` for it to be a comment rather than part of a scalar. +function strip_comment(s, i, n, c, q, prev) { + q = "" + prev = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + } else if (is_quote(c)) { + q = c + } else if (c == "#" && (prev == "" || prev == " " || prev == "\t")) { + return rtrim(substr(s, 1, i - 1)) + } + prev = c + } + return s +} + +# How many whitespace characters open the line, i.e. its block nesting level. +function indent_of(s, i, n, c) { + n = length(s) + i = 1 + while (i <= n) { + c = substr(s, i, 1) + if (c != " " && c != "\t") break + i++ + } + return i - 1 +} + +# One layer of matching quotes off a key or scalar. +function unquote(s, f) { + s = trim(s) + if (length(s) >= 2) { + f = substr(s, 1, 1) + if ((f == apos() || f == dquo()) && substr(s, length(s), 1) == f) + return substr(s, 2, length(s) - 2) + } + return s +} + +# Split `name: value` at the first colon outside quotes that is followed by end +# of line or a space, which is what makes a colon inside `http://host` part of +# the scalar. Results go to K_TXT / V_TXT because awk returns one value. +function split_pair(s, i, n, c, q) { + q = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (q != "") { + if (c == q) q = "" + continue + } + if (is_quote(c)) { q = c; continue } + if (c != ":") continue + if (i == n || substr(s, i + 1, 1) ~ /^[ \t]/) { + K_TXT = rtrim(substr(s, 1, i - 1)) + V_TXT = ltrim(substr(s, i + 1)) + return 1 + } + } + return 0 +} + +function names_authenticator(k) { return unquote(k) == "authenticator" } + +# An authenticator entry only counts when it actually names a class, and the +# answer has to be what snakeyaml hands the server rather than what the bytes +# look like. The unsafe direction is `named` for a config that leaves Gremlin +# on AllowAllAuthenticator while REST enforces, so anything this scanner cannot +# resolve to a class is refused instead of guessed at: +# +# - a plain scalar that resolves to null in any spelling, and YAML resolves +# null case-insensitively (null, Null, NULL, nUll) as well as to ~, names +# no class; +# - a leading `!` makes the tag, not the text, decide the type: !!null is the +# explicit spelling of empty and every other tag is a type not resolvable +# here, so neither counts; +# - a quoted scalar is a string and never null, but `""` and the empty single +# quoted form are the empty string, and loadAuthenticator("") returns null, +# which is the same no-authenticator state; +# - an unterminated quote is not a scalar at all. +function names_class(v, first, last, body) { + v = trim(v) + if (v == "") return 0 + first = substr(v, 1, 1) + if (first == "!") return 0 + if (first == apos() || first == dquo()) { + if (length(v) < 2) return 0 + last = substr(v, length(v), 1) + if (last != first) return 0 + body = trim(substr(v, 2, length(v) - 2)) + return body != "" + } + if (v == "~") return 0 + if (tolower(v) == "null") return 0 + return 1 Review Comment: ‼️ `names_class()` treats YAML anchor/alias syntax as a non-empty class name. For valid YAML with `authentication: {authenticator: &noAuth null}`, SnakeYAML resolves the value to null but the scanner reports `named`; with REST authentication configured, `check_auth_sides()` accepts the mismatch and Gremlin can remain unauthenticated. Resolve aliases with the runtime YAML parser or reject anchors/aliases here, and cover null/empty aliases. ########## hugegraph-server/hugegraph-dist/src/assembly/static/bin/props.awk: ########## @@ -0,0 +1,374 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# props.awk — read and rewrite Java ".properties" files with the grammar +# HugeConfig (commons-configuration over JDK Properties) applies, so the +# entrypoint and the server agree on what a mounted file means. grep/sed +# rewrites do not: they see `\`-escaped keys, `:` separators, continuation +# lines and duplicate definitions differently, which is how a mounted +# config ends up with two definitions of one key. +# +# One invocation, selected with the `PROPS_MODE` environment variable: +# +# PROPS_MODE=get PROPS_KEY=K PROPS_FILE=F +# print the value of K's first logical definition, in the on-disk +# escaped form; with PROPS_DECODED=1 print it as java.util.Properties +# would hand it to the server. Always exits 0. +# PROPS_MODE=has PROPS_KEY=K PROPS_FILE=F +# print nothing; exit 0 when K has any definition at all, empty +# included, 1 when it has none, 2 on an error +# PROPS_MODE=set PROPS_KEY=K PROPS_FILE=F +# replace K's first definition in place, drop every other +# definition of K, append one when the file has none. The new +# value arrives pre-encoded in PROPS_VALUE_ENCODED (an environment +# variable, so secrets never appear in `ps` output or in awk's +# argv), and -v is not used for it so awk cannot mangle its +# backslash escapes. +# +# Grammar implemented (java.util.Properties line reader + the +# first-definition-wins rule Configuration.getString applies): +# - physical lines end at \r\n, \n or a bare \r, as in java.util.Properties +# - '#' / '!' comments and blank lines +# - '=' / ':' / whitespace separators, where the whitespace Java counts is +# space, tab and form feed, with whitespace then an optional single '=' or +# ':' accepted as one separator +# - continuations: a physical line ending in an odd number of +# backslashes joins the next line (its leading whitespace stripped) +# - backslash escapes in keys and values, including \uXXXX +# - duplicate logical keys resolve to the first definition +# +# Rewrites keep every untouched line byte-for-byte (comments, blank +# lines, unrelated entries), and replace the first definition where it +# stands, so mounted configs stay reviewable in git diffs. + +function die(msg) { + printf "props.awk: %s\n", msg > "/dev/stderr" + # 2 for an error, so a caller that reads exit status 1 as "the key is not + # there" (PROPS_MODE=has) cannot mistake an unreadable file for an absent + # property and append a definition on top of one it failed to read. + exit 2 +} + +function hex_digit(c) { + return index("0123456789abcdef", tolower(c)) - 1 +} + +# \uXXXX is a UTF-16 code unit in Java. Values here are effectively +# ISO-8859-1, so codes above 0xFF are kept as their literal escape text +# rather than being mangled through a single-byte sprintf. +function unescape(s, out, i, n, c, code, j, d, ok) { + out = "" + n = length(s) + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (c != "\\") { out = out c; continue } + if (i == n) break + i++ + c = substr(s, i, 1) + if (c == "u" && i + 4 <= n) { + code = 0 + ok = 1 + for (j = 1; j <= 4; j++) { + d = hex_digit(substr(s, i + j, 1)) + if (d < 0) { ok = 0; break } + code = code * 16 + d + } + if (ok) { + i += 4 + if (code <= 255) out = out sprintf("%c", code) + else out = out substr(s, i - 5, 6) + continue + } + } + if (c == "t") out = out "\t" + else if (c == "n") out = out "\n" + else if (c == "r") out = out "\r" + else if (c == "f") out = out "\f" + else out = out c + } + return out +} + +# A physical line is continued when it ends in an odd number of +# backslashes (an even count escapes itself). +function trailing_backslashes(s, n, k) { + n = length(s) + k = 0 + while (k < n && substr(s, n - k, 1) == "\\") k++ + return k +} + +function is_skipped(raw) { + return raw ~ /^[ \t\f]*([#!]|$)/ +} + +# Split a logical line into its raw (still-escaped) key and value parts. +# Results land in K_RAW / V_RAW because awk returns one value. +# Java treats form feed as whitespace on both sides of the separator, so +# `auth.authenticator<FF>=...` is one property here too; reading it as part of +# the key name made a valid mounted configuration invisible to the guards. +function split_kv(s, n, i, c, esc, sep_at, rest) { + n = length(s) + esc = 0 + sep_at = 0 + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (esc) { esc = 0; continue } + if (c == "\\") { esc = 1; continue } + if (c == "=" || c == ":" || c == " " || c == "\t" || c == "\f") { sep_at = i; break } + } + if (sep_at == 0) { + K_RAW = s + V_RAW = "" + return + } + K_RAW = substr(s, 1, sep_at - 1) + rest = substr(s, sep_at) + c = substr(rest, 1, 1) + if (c == "=" || c == ":") { + rest = substr(rest, 2) + } else { + sub(/^[ \t\f]+/, "", rest) + c = substr(rest, 1, 1) + if (c == "=" || c == ":") rest = substr(rest, 2) + } + sub(/^[ \t\f]+/, "", rest) + V_RAW = rest +} + +function shquote(s) { + gsub(/'/, "'\\''", s) + return "'" s "'" +} + +# java.util.Properties ends a physical line at \r\n, \n or a bare \r, but +# getline splits on \n alone. A properties file saved with CR-only endings -- +# which java.util.Properties writes for a lone `store()` on some platforms, and +# which a mounted config can arrive with -- therefore reached the parser as one +# enormous record: only its first key was ever seen, and rewriting that key +# replaced the whole record and dropped every later entry, including +# auth.authenticator. So the file is re-scanned for terminators here. +# +# RAW[] keeps the exact bytes of each line and RAWTERM[] its terminator, so a +# rewrite still replays untouched lines byte-for-byte. A file whose last line +# carries no terminator gets a \n, which is what the replay did before. +function scan_records(s, i, n, c, start, term, len, cnt) { + n = length(s) + cnt = 0 + start = 1 + i = 1 + while (i <= n) { + c = substr(s, i, 1) + if (c != "\r" && c != "\n") { i++; continue } + if (c == "\r" && substr(s, i + 1, 1) == "\n") { + term = "\r\n" + len = 2 + } else { + term = c + len = 1 + } + cnt++ + RAW[cnt] = substr(s, start, i - start) + RAWTERM[cnt] = term + start = i + len + i = start + } + if (start <= n) { + cnt++ + RAW[cnt] = substr(s, start) + RAWTERM[cnt] = "" + } + return cnt +} + +# Load `file` into per-block arrays: one block per comment/blank line or +# logical entry, spanning exactly the physical lines it occupies. +function props_load(file, raw, rc, content, nl, stripped, next_raw, start, logical) { + content = "" + while ((rc = (getline raw < file)) > 0) + content = content raw "\n" + if (rc == -1) + die("cannot read " file) + close(file) + + NLINES = scan_records(content) + + NBLOCK = 0 + for (nl = 1; nl <= NLINES; nl++) { + # RAW[] holds one java.util.Properties physical line with its terminator + # already removed, so no CR stripping is needed here. + stripped = RAW[nl] + if (is_skipped(stripped)) { + NBLOCK++ + BTYPE[NBLOCK] = "skip" + BFIRST[NBLOCK] = nl + BLAST[NBLOCK] = nl + continue + } + start = nl + logical = stripped + while (trailing_backslashes(logical) % 2 == 1 && nl < NLINES) { + logical = substr(logical, 1, length(logical) - 1) + nl++ + next_raw = RAW[nl] + sub(/^[ \t\f]+/, "", next_raw) + logical = logical next_raw + } + # java.util.Properties ignores whitespace before the key; strip it + # so split_kv's separator scan agrees (an indented key used to be + # read as a key whose name started with a space, and a set then + # appended a second definition of the real key). + sub(/^[ \t\f]+/, "", logical) + split_kv(logical) + NBLOCK++ + BTYPE[NBLOCK] = "entry" + BFIRST[NBLOCK] = start + BLAST[NBLOCK] = nl + BKEY[NBLOCK] = unescape(K_RAW) + # Values stay in their on-disk escaped form. get Prop callers feed + # the result straight back into set, which would corrupt a decoded + # value by re-writing its backslashes as literals; keys are + # unescaped because they are matched against plain names. + BVAL[NBLOCK] = V_RAW + } +} + +function props_set(file, key, enc_val, tmp, bak, cmd, b, first, ln, msg, nbs) { + props_load(file) + # A value whose encoded form ends in an odd number of backslashes would + # turn the line written after it into a continuation of that value. + # Measured against commons-configuration2 (what HugeConfig extends), the + # same input read back yields no property at all, so a secret written this + # way would never reach the server that is supposed to authenticate with + # it; the entrypoint has to refuse instead of guessing a target. + nbs = 0 + while (nbs < length(enc_val) && substr(enc_val, length(enc_val) - nbs, 1) == "\\") + nbs++ + if (nbs % 2 == 1) + die("refusing to write " key ": the encoded value ends in an odd number of backslashes") + first = 0 + for (b = 1; b <= NBLOCK; b++) { + if (BTYPE[b] == "entry" && BKEY[b] == key) { + if (first == 0) first = b + else BDROP[b] = 1 + } + } + # Staged rewrite: everything lands in a sibling temp file first, so a + # failure before the copy-back leaves the original untouched. The temp + # file can hold secrets, so it is pre-created 0600 before the first + # write: awk's `>` below would otherwise create it under the process + # umask (usually 0644), leaving auth.admin_pa or auth.token_secret + # briefly group- and world-readable. Truncating an existing file keeps + # its mode, and the chmod after close repairs a stale tmp left behind + # by a crashed run. + tmp = file ".tmp" + system("umask 077 && : > " shquote(tmp)) Review Comment: ⚠️ This predictable `.tmp` path is opened with truncating redirection without checking for a symlink; the later AWK redirection follows it too. If a writable mounted config directory contains `rest-server.properties.tmp` as a symlink, the default-root Docker entrypoint can overwrite a file elsewhere in the container while writing credentials. Use an exclusive 0600 temporary file with no-follow semantics, and secure the `.bak` destination the same way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
