vogievetsky opened a new pull request, #20507: URL: https://github.com/apache/druid/pull/20507
## Note from the human When the console was started I used [Ace](https://ace.c9.io/) because it was the only editor that could support all the features the console needed. Ace is really a pain to use, the documentation is basically non existent. Everything the console does with Ace is basically a hack. I have always dreamed of switching to one of the more popular editors but it always felt like a mountain of effort. Well Claude Opus 5.5 did it, and tidied up all the code to remove all the Ace-isms that were all over the place, and it fixed some bugs in highlighting and completions. Otherwise everything is pixel perfect (except for a few nothingburger edge cases). You can see a migration plan file that I checked in then deleted if you want to follow the reasoning. I reviewed the results after each commit. We are living in the future. This branch also migrates the console build to mise as volta is deprecated and bumps `http-cache-semantics` to avoid a reported npm issue --- ## AI Summary The web console's code editors moved from Ace (`ace-builds` + `react-ace`) to CodeMirror 6. All the editors now share one `CodeEditor` component: the workbench, the explore view's SQL inputs, JSON inputs, spec and value dialogs, history and explain. Two Lezer grammars, one for DruidSQL and one for Hjson, drive the highlighting and the completions. The goal was to keep the console looking and behaving the same. On top of that, the code was cleaned up until it reads as if the console had used CodeMirror from the start, with no Ace-shaped leftovers. Where the behavior changes, it's either a bug fix or listed below. ## What changed **Editor (`src/components/code-editor/`)** - **`CodeEditor`:** a controlled component (`value` / `onChange`). The props that change at runtime are applied through CodeMirror compartments: language, read-only, line numbers, padding, transparent background and placeholder. `ref` gives the CodeMirror `EditorView`. - **`stateCacheId` / `forgetEditorState`:** they keep the undo history and selection between mounts, using `EditorState.toJSON`. They replace `AceEditorStateCache`. - **Theme (`code-editor-theme.ts`):** reproduces the previous look (solarized_dark colors with the console's overrides). - **Tooltip host:** the autocomplete list and its doc panel render into a shared container at the start of `<body>`, so dialogs and popovers don't clip them. - **Find/replace panel:** built from Blueprint components (`search-panel.tsx`) instead of CodeMirror's default panel. - **Inline parse errors:** `showEditorError` (`error-mark.ts`) underlines a parse error in the text and shows the message as a `data-tooltip` on hover. `JsonInput` and the workbench (for JSON queries) use it. - **Docs:** a README for the component, and `CLEANUP.md`, which records what was cleaned up and which Ace behaviors were kept on purpose. **Languages (`src/editor-languages/`)** - **`dsql.grammar`:** DruidSQL tokens and nested parentheses. - An external specializer decides which identifiers are keywords, functions, constants or types. - `getDsqlLanguage(availableSqlFunctions)` makes a language that also knows the functions the cluster reports. - This replaces the global `initAceDsqlMode`. - **`hjson.grammar`:** a real Hjson grammar: objects, properties, arrays, the root object without braces, strings with escapes, and comments. An external tokenizer handles quoteless keys and quoteless strings. - **`dsql(options)` / `hjson({ jsonCompletions })`:** each returns a `LanguageSupport` that brings its own completions and bracket closing. - **Generated parsers:** `script/build-grammars.mjs` builds them at build time. They are gitignored, like `lib/sql-docs.ts`, and run as part of `script/build` and `npm run test-unit`. - **Keywords file:** `lib/keywords.ts` moved to `src/editor-languages/dsql-keywords.ts`. **Completions (`src/editor-completions/`)** - **Shape:** completions are CodeMirror `Completion`s with a structured `doc` (name, syntax, description), not `docHTML`. The SQL docs are now "doc markdown" rendered as DOM, with no HTML strings; `snarkdown` was removed. - **Context comes from the syntax tree,** not from scanning text: - comments and literals are detected from the tree; - the keyword before the cursor is found through the tree, including one on an earlier line or past a comment; - the SQL references and literals are collected from tree nodes; - the Hjson context (`hjson-context.ts`, which replaces the 400-line scanner in `utils/`) now sees the properties after the cursor too. - **Word characters per language:** DruidSQL treats `$` as a word character. Hjson treats `$` and `-` as word characters. **Positions** - **0-based → 1-based:** the 0-based `RowColumn` is replaced by the 1-based `LineColumn` everywhere, which matches how Druid, Hjson and CodeMirror number lines. - **Renamed helpers:** - `offsetToRowColumn` → `offsetToLineColumn` - `DruidError.startRowColumn` → `DruidError.startLineColumn`, and the same for `end` - `getRowColumnFromIssue` → `getLineColumnFromIssue` - `QuerySlice.startRowColumn` → `QuerySlice.startLineColumn`, and the same for `end` - `extractRowColumnFromHjsonError` → `extractLineColumnFromHjsonError` - **Moving the cursor:** `focusEditorAt(view, position)` replaces the `goToPosition` imperative handles. **Workbench** - **Run-query gutter buttons:** these are now a CodeMirror extension (`sub-query-markers.ts`). - A state field finds the queries in the text. - Each marker is a real DOM run button with a "Run this query" tooltip, replacing the CSS pseudo-element drawing. - Hovering a button highlights its query. ## User-visible changes - **Find panel:** a Blueprint find/replace bar at the bottom of the editor replaces Ace's floating search box. It has match case, regexp and by-word options, and a replace row unless the editor is read-only. - **Hjson highlighting is correct:** - Values containing colons (timestamps, `host:port`, URLs) are no longer partly shown as keys. That affected 97 of the 452 valid JSON examples in the Druid docs. - Quoteless values are highlighted as whole strings. - A word that is being typed inside an object is shown as a key. - An unclosed quoted string ends at the end of its line. - **SQL highlighting:** - An unclosed `'…` or `"…` is colored as a string or identifier up to the end of the line. - A sign is an operator: `a-1` is `a`, `-`, `1`. - **Parse errors:** they are underlined in the editor, with the message on hover. - **Completions:** - A function with several signatures is listed once, with every signature in the doc panel. - Docs like `ARRAY<STRING>` are no longer swallowed as HTML. - The word being typed is not suggested back. - `{` is not auto-closed in SQL. - In SQL, `-` no longer joins words, so `SELECT a-co` completes `co`. - Hjson completion rules can depend on properties written below the cursor. - **Undo:** CodeMirror groups typing into undo steps a little differently from Ace. ## Dependencies and licenses - **Added:** `@codemirror/{autocomplete,commands,language,search,state,view}` and `@lezer/{common,highlight,lr}`. `@lezer/generator` is a dev dependency, used only at build time. - **Removed:** `ace-builds`, `react-ace` and `snarkdown`. - **Licenses:** `licenses.yaml` and `licenses/bin` were updated. The CodeMirror packages and their dependencies (`style-mod`, `w3c-keyname`, `crelt`, `@marijn/find-cluster-break`) are all MIT. The `ace-builds`, `react-ace` and `fast-equals` entries were removed. ## Testing - **Unit tests:** `npm run test-unit` passes, with 850 Jest tests. - **New specs** cover: - the grammars (node names, unterminated tokens, Hjson edge cases, highlight styles); - the Hjson context; - SQL and Hjson completions, the completion source and its word characters; - doc rendering, the error mark, the sub-query markers; - `CodeEditor` itself (find/replace, state caching). - **Snapshots:** the snapshot serializer replaces each editor with a comment giving its language, value, placeholder and read-only state. In the updated snapshots only the editor wrapper markup changed. - **Corpus check:** the grammars were compared with the old Ace rules on the Druid docs examples. SQL matched in 447 of 448. Valid JSON is now always correct, where the old rules got 97 of 452 wrong. - **Manual check in a running console:** - workbench highlighting, completions and docs; - run-query markers; - undo history surviving tab switches; - find/replace; - JSON error underline and tooltip; - explore-view SQL inputs. --- ## Prompt for migrating a fork or feature branch that still uses Ace ``` You are updating a branch of the Apache Druid web console (web-console/) that was written against the old Ace editor (ace-builds + react-ace) so that it works on top of master, which now uses CodeMirror 6. Keep the branch's own features and behavior; only change how they talk to the editor. Before changing anything, read web-console/src/components/code-editor/README.md (the CodeEditor component, languages, completions, positions, styling, testing) and web-console/AGENTS.md (commands, generated files, conventions). Setup: Node is pinned in web-console/.node-version and installed with mise (do not change the global Node). After merging/rebasing, run `npm install` and `npm run compile` (it generates lib/sql-docs.ts and the Lezer parsers src/editor-languages/*.parser*.ts, which are gitignored; typecheck fails without them). Old -> new mapping: Editor component - `import AceEditor from 'react-ace'` -> `CodeEditor` from src/components (code-editor/code-editor.tsx). - mode="dsql" / mode="hjson" -> language={dsql(...)} / language={hjson(...)} from src/editor-languages/. - showGutter -> showLineNumbers; focus -> autoFocus; width/height -> style or a className with CSS; theme, name, setOptions, editorProps -> remove (the theme is built in). Extra behavior -> CodeMirror extensions via the `extensions` prop (see src/views/workbench-view/flexible-query-input/sub-query-markers.ts for an example). - `onLoad`/editor instance access -> `ref` (gives the CodeMirror EditorView). - Imports of src/bootstrap/ace(.scss) -> remove. `.ace_*` CSS -> `.cm-*` classes or code-editor-theme.ts. - AceEditorStateCache -> the `stateCacheId` prop, and `forgetEditorState(id)` when the state is no longer needed. Languages and completions - src/ace-modes/* -> src/editor-languages/ (dsql.ts, hjson.ts, Lezer grammars). `initAceDsqlMode(fns)` is gone: pass `availableSqlFunctions` to `dsql({...})` (get them with `useAvailableSqlFunctions()`). - src/ace-completions/* -> src/editor-completions/. Completions come with the language: `dsql({ columnMetadata, columns, availableSqlFunctions, skipAggregates })`, `hjson({ jsonCompletions })`. Remove any custom Ace completers / getCompletions props. - Ace completion fields -> EditorCompletion (CodeMirror's Completion): value -> label, caption -> displayLabel, meta -> detail, score -> boost (range -99..99), docHTML/makeDocHtml -> doc: { name, syntax?, description?, descriptionMarkdown? } (plain data, no HTML). - getSqlCompletions/getHjsonCompletions are now builders `(context: CompletionContext, word: CompletionWord, options)`; getSqlLiterals/getPossibleSqlReferences take an EditorState (with dsql()) instead of text. - getHjsonContext moved from src/utils to src/editor-completions/hjson-context.ts: `getHjsonContext(state, pos)`, no `isEditingComment`, `currentKey` only when editing a value. - lib/keywords.ts -> src/editor-languages/dsql-keywords.ts. Positions (0-based -> 1-based!) - RowColumn { row, column } (0-based) -> LineColumn { line, column } (1-based). Convert any arithmetic: drop the `- 1`s that converted Druid/Hjson positions to Ace rows, and use `line - 1` where a 0-based line count is needed (e.g. `changePrefixLines(slice.startLineColumn.line - 1)`). - offsetToRowColumn -> offsetToLineColumn; extractRowColumnFromHjsonError -> extractLineColumnFromHjsonError; DruidError.startRowColumn/endRowColumn (and extractStart/EndRowColumn) -> startLineColumn/endLineColumn; WorkbenchQuery.getRowColumnFromIssue -> getLineColumnFromIssue; QuerySlice.startRowColumn/endRowColumn -> startLineColumn/endLineColumn; QueryErrorPane's moveCursorTo takes a LineColumn. - goToPosition(rowColumn) on FlexibleQueryInputHandle / SqlInputHandle -> the ref is the EditorView: `focusEditorAt(view, lineColumn)`. To mark an error in the text: `showEditorError(view, { position, message })`. Component props - SqlInput: editorHeight -> height, showGutter -> showLineNumbers, ref -> EditorView. - FlexibleQueryInput: showGutter -> showLineNumbers, editorStateId -> stateCacheId, leaveBackground -> transparentBackground (inverted: leaveBackground={true} becomes transparentBackground={false}; the default is still transparent), ref -> EditorView, new readOnly. - JsonInput: focus -> autoFocus, width removed (size it with CSS), new readOnly. Tests - Ace API in specs (editor.getSession(), setValue, etc.) -> `EditorView.findFromDOM(el.querySelector('.cm-editor'))` and `view.dispatch({ changes })` (wrap in act() when React state changes). - Completion specs: `completionContextAt(dsql(), 'SELECT * FROM t WHERE |')` from src/test-utils/completion-context.ts. - DOM snapshots show each editor as a comment like `Code editor, language: dsql, value: "..."`. After updating snapshots (`npx jest -u <files>`), check that only editor markup changed. Process 1. Merge master, resolve conflicts using the mapping above. 2. Grep the branch for leftovers until none remain: `ace`, `Ace`, `react-ace`, `ace-builds`, `RowColumn`, `rowColumn`, `.row`, `showGutter`, `editorHeight`, `editorStateId`, `leaveBackground`, `docHTML`, `caption`, `meta:`, `score:`, `goToPosition`, `initAceDsqlMode`, `ace-modes`, `ace-completions`, `lib/keywords`. 3. Run `npm run test-unit` (generators, typecheck, eslint, stylelint, prettier, jest) and fix everything. 4. Start the dev server (check whether port 18081 is already in use first) and check the branch's features in the browser, especially anything that positions the cursor or reads positions (off-by-one errors are the most likely mistake). 5. Summarize what was migrated, anything that could not be mapped one to one, and any behavior differences. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
