[GitHub] [druid] vogievetsky opened a new pull request, #12919: Web console: Multi-stage query support

GitBox Wed, 17 Aug 2022 23:40:28 -0700


vogievetsky opened a new pull request, #12919:
URL: https://github.com/apache/druid/pull/12919

Do you ever use the web console? Well, be prepared to have your experience
elevated!

This PR adds functionality to the web console primarily driven by the desire
to support the [Multi-stage distributed
queries](https://github.com/apache/druid/issues/12262) effort with appropriate
UI/UX. This PR needs https://github.com/apache/druid/pull/12918 to work.

Here are the high level bits.

> Please note: The example data is hosted on
[static.imply.io](http://static.imply.io/). Storage space and bandwidth on this
server is being donated by Imply as a service to the community. I thought about
using [druid.apache.org](http://druid.apache.org/) instead, but that didn't
seem like a good place to host larger data files, since the server is designed
for website content.

## New query view (WorkbenchView) with tabs and long running query support

The new query lets you execute [multi-stage, task based,
queries](https://github.com/apache/druid/issues/12262) via the
`/druid/v2/sql/task` and `/druid/indexer/v1/task/*` APIs as well as native and
sql-native queries just like the old Query view. A key point of the
sql-msq-task based queries is that they may run for a long time. This inspired
/ necessitated many UX changes including (but not limited to):

### Tabs

You can now have many queries stored and running at the same time. In my
opinion tabs are a UX game changer

You can, open many tabs, duplicate them, and copy them as text to be pasted
into any console and reopened there.

### Progress reports (counter reports)

sql-msq-task queries have detailed progress reports and these can be
visualized using the summary progress bar, or the in detail execution table
that provides summaries of the counters for every step.

### Error and warning reports

sql-msq-task queries go out of their way to present user friendly warnings
and errors should anything go wrong.
The new query view has components to visualize these with their full detail
including a stack-trace.

### Recent query panel

Since sql-msq-task queries are tasks is it possible to show queries that are
executing currently and that have executed in the recent past.

For any query in this panel you can view the execution details for it and
you can also attach it as a new tab and continue iterating on the query. It is
also possible to download the "query detail archive" which is a JSON file
containing all the important details for a given query that can be easily
shared for troubleshooting.

### Connect external data flow

Running SQL-based ingestion queries is all well and good once you have the
query, but where do queries come from?

The `Connect external data` flow lets you use the sampler to sample your
data, determine its schema, and generate a fully formed SQL query that you can
edit to your hearts content before clicking `Run`.

This point-and-click flow will save you a bunch of typing.

### Preview button

The `Preview` button appears when you type in an INSERT or REPLACE SQL query.

When clicked it removes the INSERT or REPLACE clause and executes your query
as an "inline" query (with a limit). This lets you get a sense of how your data
will be shaped after all the transformations that you coded into the SQL query
have been applied.

### Results table

The query results table has been improved in style and function.

It now shows you type icons for the column types and supports the ability to
manipulate nested columns with ease.

### Helper queries

Do you like notebooks? Do you like [CTEs]()? Well now they have some UI
affordances in the console.
Helper queries are collapsable elements that hold a query that can be
referenced from the main query just like they were defined with a WITH
statement.

The idea is that if you are composing a complicated query it is nice to be
able to break it down into queries and be able to preview each part on its own.

### Misc tools

Along with these features there are also miscellaneous tools available from
the `...` menu.

Along these you will find:

- Explain query - show the query plan for a sql-native and sql-msq-task
query.
- Convert ingestion spec to SQL - Helps you migrate your native batch and
hadoop based specs to the SQL-based format.
- Open query detail archive - lets you open a query detail archive
downloaded earlier.
- Load demo queries - lets you load a set of pre-made queries to play around
with sql-msq-task functionality.

## New SQL-based data loader

The data loader exists as a GUI wizard to help users craft a JSON ingestion
spec using point and click and quick previews.
The SQL data loader is the SQL-based ingestion analog of that.

Just like the native based data loader that serves as its inspiration, all
the state is stored in the actual SQL query and at any moment the user can opt
to manipulate the query directly.

What is the difference between the SQL-based data loader and the "Connect
external data" flow? They start off exactly the same, they figure out the
source and format to use and generate the external. After the external has been
defined the "Connect external data" flow generates a seed query that you can
then edit as you wish. The key point is that (unless you just want to ingest
with the seed query) you must edit a rather complex SQL statement and preview
it using the "Preview" functionality that is not sub-second (it reached out to
the data for every preview). On the other hand the data loader let's you
manipulate the (SQL) ingestion spec completely with point-and-click it also
samples the data and lets you preview it using sql-native making the previews
sub-second. The SQL data loader limits you to what kind of query you can write,
for example you can not currently use it to express a query involving a JOIN,
but at any point in time it lets you take your query to the regular que
ry view so there is no need to worry about being restricted.

Since the SQL-based data loader is batch only and the existing data loader
is for streaming (supervisors) and batch data (tasks). The existing data loader
has been split up into its streaming and batch components to make the
comparison clearer.

## Misc other changes

Since the MSQ part is so transformative it drove other parts of the console
development as well.

### Header rearrangment

Now that the query view has tabs and can do ingestion and data manipulation
it is about time it took the helm as the most important view in the header.

There is now also a more `...` button that can host all the views not
important enough to be top level.

### Data preview for both the Datasource and Segment detail modals

You can now click on a datasource or segment to see a preview of the data
within.

### Task cancel status

The task table now explicitly shows if a task has been canceled in a
different color than a failed task.

### Other improvements

- Expose the tuning config property `maxColumnsToMerge` for batch ingestion
and compaction in the form UI
- Fixed docs link URL
- Better filtering on all data tables (Datasources, Ingestion, Services, etc)
- Made filtering on modules work better in the Status dialog
- Added the file `filter` property to all inputSources where it applies
- Mixed some confusing labels in the UI
- Future proof the `useQueryManager` hook to work with React18

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[GitHub] [druid] vogievetsky opened a new pull request, #12919: Web console: Multi-stage query support

Reply via email to