[GitHub] [incubator-druid] vogievetsky opened a new pull request #8729: Docs: fix doc headers
vogievetsky opened a new pull request #8729: Docs: fix doc headers URL: https://github.com/apache/incubator-druid/pull/8729 Fixes #8726. Some pages did not start their headers with h2 which breaks the right nav generation (see ticket above) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] natsumehu commented on issue #8714: Theta Sketch upfront uniform sampling probability setting
natsumehu commented on issue #8714: Theta Sketch upfront uniform sampling probability setting URL: https://github.com/apache/incubator-druid/issues/8714#issuecomment-545749823 Sure, I will do that This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8724: Web console: Button to pretty print Druid JSON query
vogievetsky commented on issue #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724#issuecomment-545725669 Thank you for the contrib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky merged pull request #8724: Web console: Button to pretty print Druid JSON query
vogievetsky merged pull request #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (137c2a6 -> fdbc4ae)
This is an automated email from the ASF dual-hosted git repository. vogievetsky pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 137c2a6 Web console: disable data loader Submit button when submitting so as not to submit multiple times (#8725) add fdbc4ae Web console: Button to pretty print Druid JSON query (#8724) No new revisions were added by this update. Summary of changes: .../src/views/query-view/__snapshots__/query-view.spec.tsx.snap| 1 + web-console/src/views/query-view/query-view.tsx| 7 +++ web-console/src/views/query-view/run-button/run-button.spec.tsx| 1 + web-console/src/views/query-view/run-button/run-button.tsx | 5 + 4 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times
fjy merged pull request #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times URL: https://github.com/apache/incubator-druid/pull/8725 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (094936c -> 137c2a6)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 094936c Remove commit() method Firehose (#8688) add 137c2a6 Web console: disable data loader Submit button when submitting so as not to submit multiple times (#8725) No new revisions were added by this update. Summary of changes: web-console/src/utils/ingestion-spec.tsx | 8 ++ .../src/views/load-data-view/load-data-view.tsx| 115 - 2 files changed, 74 insertions(+), 49 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8728: Initial join support
vogievetsky commented on issue #8728: Initial join support URL: https://github.com/apache/incubator-druid/issues/8728#issuecomment-545704190 I know everyone will be very excited about JOINs but the proposal to have lookups queryable via SQL will enable amazing stuff in the web console! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8728: Initial join support
gianm commented on issue #8728: Initial join support URL: https://github.com/apache/incubator-druid/issues/8728#issuecomment-545702698 I am planning to do work towards implementing this proposal over the next few months. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new issue #8728: Initial join support
gianm opened a new issue #8728: Initial join support
URL: https://github.com/apache/incubator-druid/issues/8728
## Motivation
Druid aims to be a more powerful analytical database, and implementing joins
is a very common ask from the user community. Druid does support some related
features today:
- [Lookups](https://druid.apache.org/docs/latest/querying/lookups.html)
enable certain star-schema use cases, and are similar to joins in that regard.
But they are limited in that they can only support LEFT joins of
fact-to-dimension (not right/inner/outer) and that they cannot support more
than one value per key.
- Druid SQL supports
[semijoins](https://druid.apache.org/docs/latest/querying/sql#query-execution)
through both the `WHERE x IN (SELECT ...)` syntax and traditional JOIN syntax.
But it only supports one subquery per SQL query, and does not support negation
("NOT IN").
Real JOIN support would be more powerful, enabling even more star-schema and
subquery-based use cases:
- More flexibility for join types (LEFT/RIGHT/INNER/FULL OUTER).
- More flexibility for join conditions.
- Joining against more than one subquery at once.
- Easier integration with third-party applications that generate JOIN
queries for star schemas.
- Open the door for joining two datasources together.
## Proposed changes
The idea is to add a "join" datasource, expose it through SQL, and add
machinery to brokers and data servers to allow hash-based equijoins of **zero
or one "table" datasource** and **any number of "lookup", "inline", or "query"
datasources**. As a side effect, these proposed changes will add the ability to
query lookups directly, using a datasource of type "lookup".
I think this subset of functionality is a good start, because it adds
meaningful new capabilities and helps unify existing ones. Things that are not
in scope for this initial proposal include: joining two "table" datasources to
each other, non-equijoins, non-hash-based joins. I think these should all be
implemented at some point, as future work.
There are four main areas to this proposal,
1. SQL: add rules to plan joins, adding "lookup" schema.
2. Native queries: add "join", "lookup", and "inline" datasources.
3. Broker: must be able to translate anything that data servers can’t do
into something they can do (e.g. evaluate a subquery and replace it with an
"inline" datasource). Must also be able to fully evaluate queries that only use
local datasource like "lookup" and "inline".
4. Data servers (historical, etc): must be able to evaluate joins of _one
table_ onto _any number of lookup or inline datasources, or queries on top of
those_.
The next few sections expand on these areas.
### SQL
1. Add lookups to SQL in a "lookup" schema.
2. Allow any equi-joins that involve zero or one normal datasource and any
number of lookups or subqueries, through a new join-planning system.
3. Remove semi-join specific planning code, since that should now be handled
by the generic join planning code in (2) above.
An example SQL query might be:
```
SELECT products.value AS product_name, SUM(sales.revenue)
FROM sales
LEFT JOIN lookup.products ON sales.product_id = products.key
GROUP BY products.value
```
This query takes advantage of the fact that unqualified tables like sales
are assumed to be normal datasources. Lookups are referenced as part of the
lookup schema, like lookup.products.
Multiple join queries can be specified per SQL query. We will need to guide
Calcite’s cost-based optimizer towards reordering them optimally. This may
require more statistics than we currently possess, so adding these and
improving join plans may end up being an area of future work.
### Native queries
1. Add a "join" datasource type that represents a join of two other
datasources. It’d include a left datasource, right datasource, condition (which
I am thinking will be restricted to equality at first), and type (left, right,
full outer, inner).
2. Add "lookup" and "inline" datasources to provide things to join onto.
These can be specified as inputs to a join, or they can be directly queried (a
new capability for lookups!)’
3. Allow joining on to "query" datasources as well. To make this work, we’ll
need to add a sense of a ‘standard translation’ of results from certain query
types into flat schemas that we can offer column selectors on top of. There may
be more than one way to do this, since certain query types (notably, topN and
scan) return nested results in some cases. We could do this by adding a new
QueryToolChest method.
The rows coming out of a join datasource would be the result of the join.
Any query type could use a join datasource without being aware of the fact that
joins exist.
Join datasources can be nested within each other. Unlike SQL, native
[GitHub] [incubator-druid] vogievetsky commented on issue #8726: Docusaurus: Right sidebar is sometime missing
vogievetsky commented on issue #8726: Docusaurus: Right sidebar is sometime missing URL: https://github.com/apache/incubator-druid/issues/8726#issuecomment-545698105 Hmm good catch it looks like https://druid.apache.org/docs/latest/querying/groupbyquery.html starts with an h3 (instead of h2). I bet that is the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8713: Improper result-level cache ETag handling for union datasources
gianm commented on issue #8713: Improper result-level cache ETag handling for union datasources URL: https://github.com/apache/incubator-druid/issues/8713#issuecomment-545697482 > I think `UnionDataSource` should handle this special case which would solve the problem for both `ResultLevelCache` and for users with their own result level cache outside of Druid that are using same protocol. I am worried about a situation where UnionQueryRunner needs to know about ETags. What if there's some _other_ response context field now (or in the future) that has the same problem? It is too bug-prone for people to need to remember to update UnionQueryRunner every time they add a response context field. Also, if there are things other than UnionQueryRunner that split up queries into subqueries, they would all need to know about ETags too, which seems weird. What do you think about making ResponseContext have a concurrency-safe "compute" method similar to the one on `java.util.Map`? We could even get rid of the `put` method and only offer `compute`, to ensure we're always handling this case properly. Or keep `put` but make it throw an error if there is an existing value. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource
vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545696851 FYI this view is powered by a `cardinality(byRow)` query. There should be some tweaks around making the slow state work nice and also around setting good priorities and timeouts - so more UI work. But the idea is solid. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource
vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545696181 @fjy what do you mean? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource
fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545692520 > @fjy I think if there is a lot of data for the selected interval then the tool will be slow. It should probably mention it in the blurb Uh... it seems like we should think about this more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource
vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545689888 @fjy I think if there is a lot of data for the selected interval then the tool will be slow. It should probably mention it in the blurb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource
vogievetsky commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545689629 I think you are missing a snapshot test here This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource
fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545689334 > @fjy No it will calculate the ratio for the entire datasource based on the interval, the first 20 rows is just to serve as a preview for the user to look at. What happens if the interval covers a lot of data? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 commented on issue #8727: Web console: Estimate rollup ratio for a datasource
renevan10 commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545689181 @fjy No it will calculate the ratio for the entire datasource based on the interval, the first 20 rows is just to serve as a preview for the user to look at. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource
fjy commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545687337 @renevan10 @vogievetsky does the algorithm only look at the first 20 rolls? I don't think that is going to accurately estimate the rollup ratio This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8727: Web console: Estimate rollup ratio for a datasource
lgtm-com[bot] commented on issue #8727: Web console: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545685034 This pull request **introduces 2 alerts** when merging 8298a79b8ec8a4983625302225618f767cd5b68a into a8b674e00d9ccad334b8ed6ecb3712dc7e3259b8 - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-0277552da8595e16722481bd482ce9d514711571) **new alerts:** * 2 for Unused or undefined state property This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times
vogievetsky commented on issue #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times URL: https://github.com/apache/incubator-druid/pull/8725#issuecomment-545684180 I have not been adding any unit tests for fine grained interactions like "a button becomes disabled during this call". I do test it all manually (hence the screenshot). The `isTask` function could have a unit test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug edited a comment on issue #8713: Improper result-level cache ETag handling for union datasources
himanshug edited a comment on issue #8713: Improper result-level cache ETag handling for union datasources URL: https://github.com/apache/incubator-druid/issues/8713#issuecomment-545676670 Thanks for the detailed information. I think `UnionDataSource` should handle this special case which would solve the problem for both `ResultLevelCache` and for users with their own result level cache outside of Druid that are using same protocol. `UnionDataSource` should have different `ResponseContext` objects for different queries and merge them into a single [maybe nested] `ResponseContext` object in the end. The merge process creates the `top level Etag = serialize-to-string(array of ETag objects from each query response)` Also, If `UnionDataSource` receives `If-None-Match` key in the incoming `ResponseObject` , it should assume that would be a serialized string from array, break it into elements and add to one per datasource query `ResponseObject` it creates. If `If-None-Match` value is not deserializable into string array of size N (number of datasources) then dont use it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (d9c9aef -> 2518478)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from d9c9aef Druid Doctor (#8672) add 2518478 Remove deprecated parameter for Checkpoint request (#8707) No new revisions were added by this update. Summary of changes: .../MaterializedViewSupervisor.java| 7 +- .../druid/indexing/kafka/KafkaIndexTaskTest.java | 3 +- .../kafka/supervisor/KafkaSupervisorTest.java | 100 --- .../indexing/kinesis/KinesisIndexTaskTest.java | 3 +- .../kinesis/supervisor/KinesisSupervisorTest.java | 107 - .../CheckPointDataSourceMetadataAction.java| 20 +--- .../overlord/supervisor/SupervisorManager.java | 5 +- .../SeekableStreamIndexTaskRunner.java | 1 - .../supervisor/SeekableStreamSupervisor.java | 52 +- .../overlord/supervisor/NoopSupervisorSpec.java| 6 +- .../indexing/overlord/supervisor/Supervisor.java | 7 +- 11 files changed, 15 insertions(+), 296 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (2518478 -> 094936c)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 2518478 Remove deprecated parameter for Checkpoint request (#8707) add 094936c Remove commit() method Firehose (#8688) No new revisions were added by this update. Summary of changes: .../org/apache/druid/data/input/Committer.java | 3 ++ .../java/org/apache/druid/data/input/Firehose.java | 38 ++-- .../apache/druid/data/input/FirehoseFactory.java | 4 +- .../data/input/impl/FileIteratingFirehose.java | 7 --- .../task/AppenderatorDriverRealtimeIndexTask.java | 3 +- .../indexing/common/task/RealtimeIndexTask.java| 3 +- .../indexing/overlord/sampler/SamplerCache.java| 7 --- .../seekablestream/SeekableStreamSamplerSpec.java | 7 --- .../apache/druid/indexing/common/TestFirehose.java | 7 --- .../AppenderatorDriverRealtimeIndexTaskTest.java | 7 --- .../druid/indexing/overlord/TaskLifecycleTest.java | 26 --- .../firehose/CombiningFirehoseFactory.java | 6 --- .../firehose/EventReceiverFirehoseFactory.java | 7 --- .../firehose/FixedCountFirehoseFactory.java| 6 --- .../realtime/firehose/IngestSegmentFirehose.java | 7 --- .../segment/realtime/firehose/InlineFirehose.java | 7 --- .../realtime/firehose/PredicateFirehose.java | 6 --- .../segment/realtime/firehose/SqlFirehose.java | 7 --- .../firehose/TimedShutoffFirehoseFactory.java | 6 --- .../druid/segment/realtime/plumber/Committers.java | 53 -- .../firehose/CombiningFirehoseFactoryTest.java | 7 --- .../realtime/firehose/InlineFirehoseTest.java | 9 .../plumber/RealtimePlumberSchoolTest.java | 32 + 23 files changed, 43 insertions(+), 222 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 commented on issue #8727: Estimate rollup ratio for a datasource
renevan10 commented on issue #8727: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545683499 @fjy yes! You can select multiple columns. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on a change in pull request #8688: Remove commit() method Firehose
fjy commented on a change in pull request #8688: Remove commit() method Firehose URL: https://github.com/apache/incubator-druid/pull/8688#discussion_r338329259 ## File path: server/src/main/java/org/apache/druid/segment/realtime/plumber/Committers.java ## @@ -40,54 +42,15 @@ public void run() // Do nothing } }; + private static final Supplier NIL_SUPPLIER = Suppliers.ofInstance(NIL); Review comment: Agree that we should remove the nil supplier. A follow up PR is fine as long as we remember to do it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #8688: Remove commit() method Firehose
fjy merged pull request #8688: Remove commit() method Firehose URL: https://github.com/apache/incubator-druid/pull/8688 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #8707: Remove deprecated parameter for Checkpoint request
fjy merged pull request #8707: Remove deprecated parameter for Checkpoint request URL: https://github.com/apache/incubator-druid/pull/8707 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (a8b674e -> d9c9aef)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.
from a8b674e fix indexer when used with http inventory view (#8723)
add d9c9aef Druid Doctor (#8672)
No new revisions were added by this update.
Summary of changes:
web-console/package-lock.json | 107 ++
web-console/package.json | 4 +-
.../__snapshots__/header-bar.spec.tsx.snap | 9 +
.../src/components/header-bar/header-bar.tsx | 8 +
.../__snapshots__/about-dialog.spec.tsx.snap | 2 +-
.../src/dialogs/about-dialog/about-dialog.tsx | 9 +-
.../__snapshots__/doctor-dialog.spec.tsx.snap} | 61 +--
.../src/dialogs/doctor-dialog/doctor-checks.tsx| 423 +
.../doctor-dialog.scss}| 13 +-
.../doctor-dialog.spec.tsx}| 9 +-
.../src/dialogs/doctor-dialog/doctor-dialog.tsx| 200 ++
web-console/src/utils/general.tsx | 6 +
web-console/src/utils/ingestion-spec.tsx | 2 +-
web-console/src/utils/sampler.ts | 5 +-
14 files changed, 729 insertions(+), 129 deletions(-)
copy
web-console/src/dialogs/{table-action-dialog/__snapshots__/table-action-dialog.spec.tsx.snap
=> doctor-dialog/__snapshots__/doctor-dialog.spec.tsx.snap} (63%)
create mode 100644 web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
copy web-console/src/dialogs/{compaction-dialog/compaction-dialog.scss =>
doctor-dialog/doctor-dialog.scss} (87%)
copy web-console/src/dialogs/{about-dialog/about-dialog.spec.tsx =>
doctor-dialog/doctor-dialog.spec.tsx} (85%)
create mode 100644 web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #8672: Druid Doctor
fjy merged pull request #8672: Druid Doctor URL: https://github.com/apache/incubator-druid/pull/8672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on issue #8727: Estimate rollup ratio for a datasource
fjy commented on issue #8727: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727#issuecomment-545682441 @renevan10 this supporting arbitrarily selecting multiple columns to exclude right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8583: Regex parser should have the option to 'skip' some number of header rows like the CSV parser can
vogievetsky commented on issue #8583: Regex parser should have the option to 'skip' some number of header rows like the CSV parser can URL: https://github.com/apache/incubator-druid/issues/8583#issuecomment-545682171 @tan31989 are you still interested in working on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 opened a new pull request #8727: Estimate rollup ratio for a datasource
renevan10 opened a new pull request #8727: Estimate rollup ratio for a datasource URL: https://github.com/apache/incubator-druid/pull/8727 This PR adds a dialog to preview and estimate rollup of a datasource under the datasources section. The top 20 rows of the datasource will be previewed. The user may get the estimated rollup ratio by: - changing the interval of the data, - selecting columns to exclude for the rollup(which will highlight the entire column in grey) - changing the granularity If the ingested data was previously rolled up, the original rollup ratio will also be displayed.  Note that at least one column must be de-selected for the calculation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8713: Improper result-level cache ETag handling for union datasources
himanshug commented on issue #8713: Improper result-level cache ETag handling for union datasources URL: https://github.com/apache/incubator-druid/issues/8713#issuecomment-545676670 Thanks for the detailed information. I think `UnionDataSource` should handle this special case which would solve the problem for both `ResultLevelCache` and for users with their own result level cache outside of Druid that are using same protocol. `UnionDataSource` should have different `ResponseContext` objects for different queries and merge them into a single [maybe nested] `ResponseContext` object. The merge process creates the `top level Etag = serialize-to-string(array of ETag objects from each query)` Also, If `UnionDataSource` receives `If-None-Match` key in the incoming `ResponseObject` , it should assume that would be a serialized string from array of size N (equal to number of DataSources in the query), break it into elements and add to one per datasource query `ResponseObject` it creates. If `If-None-Match` value is not deserializable into string array of size N then fail with `IAE` exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug edited a comment on issue #8713: Improper result-level cache ETag handling for union datasources
himanshug edited a comment on issue #8713: Improper result-level cache ETag handling for union datasources URL: https://github.com/apache/incubator-druid/issues/8713#issuecomment-545676670 Thanks for the detailed information. I think `UnionDataSource` should handle this special case which would solve the problem for both `ResultLevelCache` and for users with their own result level cache outside of Druid that are using same protocol. `UnionDataSource` should have different `ResponseContext` objects for different queries and merge them into a single [maybe nested] `ResponseContext` object in the end. The merge process creates the `top level Etag = serialize-to-string(array of ETag objects from each query response)` Also, If `UnionDataSource` receives `If-None-Match` key in the incoming `ResponseObject` , it should assume that would be a serialized string from array of size N (equal to number of DataSources in the query), break it into elements and add to one per datasource query `ResponseObject` it creates. If `If-None-Match` value is not deserializable into string array of size N then fail with `IAE` exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8716: Handle coordinator leader changes for segments view
himanshug commented on issue #8716: Handle coordinator leader changes for segments view URL: https://github.com/apache/incubator-druid/issues/8716#issuecomment-545669761 haven't looked at the code, but relevant part should just use `DruidLeaderClient` class that is meant to be used for talking to coordinator/overlord and handles all that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug closed issue #8710: druid indexer running Kafka indexing not seen as dataSource
himanshug closed issue #8710: druid indexer running Kafka indexing not seen as dataSource URL: https://github.com/apache/incubator-druid/issues/8710 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (3abd0c4 -> a8b674e)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 3abd0c4 Web console: start transition to React hooks and functional components (#8722) add a8b674e fix indexer when used with http inventory view (#8723) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/druid/cli/CliIndexer.java | 24 +- 1 file changed, 23 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #8723: fix indexer segment announcement when used with http inventory view
himanshug merged pull request #8723: fix indexer segment announcement when used with http inventory view URL: https://github.com/apache/incubator-druid/pull/8723 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8656: Message rejection absolute date
himanshug commented on a change in pull request #8656: Message rejection
absolute date
URL: https://github.com/apache/incubator-druid/pull/8656#discussion_r338309318
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisorIOConfig.java
##
@@ -62,9 +67,19 @@ public SeekableStreamSupervisorIOConfig(
this.lateMessageRejectionPeriod = lateMessageRejectionPeriod == null
? Optional.absent()
:
Optional.of(lateMessageRejectionPeriod.toStandardDuration());
+this.lateMessageRejectionStartDateTime = lateMessageRejectionStartDateTime
== null
+ ? Optional.absent()
+ :
Optional.of(lateMessageRejectionStartDateTime);
this.earlyMessageRejectionPeriod = earlyMessageRejectionPeriod == null
? Optional.absent()
:
Optional.of(earlyMessageRejectionPeriod.toStandardDuration());
+
+if (this.lateMessageRejectionPeriod.isPresent()
+&& this.lateMessageRejectionStartDateTime.isPresent()) {
+ throw new JsonMappingException("SeekableStreamSupervisorIOConfig does
not support "
Review comment:
for validation errors, we don't use checked exceptions but .. `IAE` would be
more appropriate. please remove the `throws Exception` etc from constructor of
other classes where it was added due to this.
```suggestion
throw new IAE("SeekableStreamSupervisorIOConfig does not support "
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8656: Message rejection absolute date
himanshug commented on a change in pull request #8656: Message rejection
absolute date
URL: https://github.com/apache/incubator-druid/pull/8656#discussion_r338309318
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisorIOConfig.java
##
@@ -62,9 +67,19 @@ public SeekableStreamSupervisorIOConfig(
this.lateMessageRejectionPeriod = lateMessageRejectionPeriod == null
? Optional.absent()
:
Optional.of(lateMessageRejectionPeriod.toStandardDuration());
+this.lateMessageRejectionStartDateTime = lateMessageRejectionStartDateTime
== null
+ ? Optional.absent()
+ :
Optional.of(lateMessageRejectionStartDateTime);
this.earlyMessageRejectionPeriod = earlyMessageRejectionPeriod == null
? Optional.absent()
:
Optional.of(earlyMessageRejectionPeriod.toStandardDuration());
+
+if (this.lateMessageRejectionPeriod.isPresent()
+&& this.lateMessageRejectionStartDateTime.isPresent()) {
+ throw new JsonMappingException("SeekableStreamSupervisorIOConfig does
not support "
Review comment:
for validation errors, we don't use checked exceptions but .. `IAE` would be
more appropriate. please remove the `throws Exception` etc from constructor of
other classes where it was added due to this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8656: Message rejection absolute date
himanshug commented on a change in pull request #8656: Message rejection absolute date URL: https://github.com/apache/incubator-druid/pull/8656#discussion_r338308311 ## File path: docs/development/extensions-core/kafka-ingestion.md ## @@ -206,7 +206,8 @@ For Roaring bitmaps: |`period`|ISO8601 Period|How often the supervisor will execute its management logic. Note that the supervisor will also run in response to certain events (such as tasks succeeding, failing, and reaching their taskDuration) so this value specifies the maximum time between iterations.|no (default == PT30S)| |`useEarliestOffset`|Boolean|If a supervisor is managing a dataSource for the first time, it will obtain a set of starting offsets from Kafka. This flag determines whether it retrieves the earliest or latest offsets in Kafka. Under normal circumstances, subsequent tasks will start from where the previous segments ended so this flag will only be used on first run.|no (default == false)| |`completionTimeout`|ISO8601 Period|The length of time to wait before declaring a publishing task as failed and terminating it. If this is set too low, your tasks may never publish. The publishing clock for a task begins roughly after `taskDuration` elapses.|no (default == PT30M)| -|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| +|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to reject messages with timestamps earlier than this date time; for example if this is set to `2016-01-01T11:00Z` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). This option exclude `lateMessageRejectionPeriod`.|no (default == none)| Review comment: ```suggestion |`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to reject messages with timestamps earlier than this date time; for example if this is set to `2016-01-01T11:00Z` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8656: Message rejection absolute date
himanshug commented on a change in pull request #8656: Message rejection absolute date URL: https://github.com/apache/incubator-druid/pull/8656#discussion_r338307976 ## File path: docs/development/extensions-core/kafka-ingestion.md ## @@ -206,7 +206,8 @@ For Roaring bitmaps: |`period`|ISO8601 Period|How often the supervisor will execute its management logic. Note that the supervisor will also run in response to certain events (such as tasks succeeding, failing, and reaching their taskDuration) so this value specifies the maximum time between iterations.|no (default == PT30S)| |`useEarliestOffset`|Boolean|If a supervisor is managing a dataSource for the first time, it will obtain a set of starting offsets from Kafka. This flag determines whether it retrieves the earliest or latest offsets in Kafka. Under normal circumstances, subsequent tasks will start from where the previous segments ended so this flag will only be used on first run.|no (default == false)| |`completionTimeout`|ISO8601 Period|The length of time to wait before declaring a publishing task as failed and terminating it. If this is set too low, your tasks may never publish. The publishing clock for a task begins roughly after `taskDuration` elapses.|no (default == PT30M)| -|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| +|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to reject messages with timestamps earlier than this date time; for example if this is set to `2016-01-01T11:00Z` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). This option exclude `lateMessageRejectionPeriod`.|no (default == none)| +|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). Please note that `lateMessageRejectionPeriod` is ignored if `lateMessageRejectionStartDateTime` is specified.|no (default == none)| Review comment: ```suggestion |`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). Please note that only one of `lateMessageRejectionPeriod` or `lateMessageRejectionStartDateTime` can be specified.|no (default == none)| ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8656: Message rejection absolute date
himanshug commented on a change in pull request #8656: Message rejection absolute date URL: https://github.com/apache/incubator-druid/pull/8656#discussion_r338307976 ## File path: docs/development/extensions-core/kafka-ingestion.md ## @@ -206,7 +206,8 @@ For Roaring bitmaps: |`period`|ISO8601 Period|How often the supervisor will execute its management logic. Note that the supervisor will also run in response to certain events (such as tasks succeeding, failing, and reaching their taskDuration) so this value specifies the maximum time between iterations.|no (default == PT30S)| |`useEarliestOffset`|Boolean|If a supervisor is managing a dataSource for the first time, it will obtain a set of starting offsets from Kafka. This flag determines whether it retrieves the earliest or latest offsets in Kafka. Under normal circumstances, subsequent tasks will start from where the previous segments ended so this flag will only be used on first run.|no (default == false)| |`completionTimeout`|ISO8601 Period|The length of time to wait before declaring a publishing task as failed and terminating it. If this is set too low, your tasks may never publish. The publishing clock for a task begins roughly after `taskDuration` elapses.|no (default == PT30M)| -|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no (default == none)| +|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to reject messages with timestamps earlier than this date time; for example if this is set to `2016-01-01T11:00Z` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). This option exclude `lateMessageRejectionPeriod`.|no (default == none)| +|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject messages with timestamps earlier than this period before the task was created; for example if this is set to `PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* will be dropped. This may help prevent concurrency issues if your data stream has late messages and you have multiple pipelines that need to operate on the same segments (e.g. a realtime and a nightly batch ingestion pipeline). Please note that `lateMessageRejectionPeriod` is ignored if `lateMessageRejectionStartDateTime` is specified.|no (default == none)| Review comment: `Please note that ... ignored if ...` is not accurate anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8688: Remove commit() method Firehose
clintropolis commented on a change in pull request #8688: Remove commit() method Firehose URL: https://github.com/apache/incubator-druid/pull/8688#discussion_r338299004 ## File path: server/src/main/java/org/apache/druid/segment/realtime/plumber/Committers.java ## @@ -40,54 +42,15 @@ public void run() // Do nothing } }; + private static final Supplier NIL_SUPPLIER = Suppliers.ofInstance(NIL); Review comment: It seems like a smell that this class still exists and only has this nil supplier, and maybe can be removed entirely. But I'm ok if we continue clean-up in the future since this looks a bit more invasive to dump than just removing `commit` from the `Firehose`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] woshifudayun removed a comment on issue #8666: resolve resolveip command not found issue in integration test
woshifudayun removed a comment on issue #8666: resolve resolveip command not found issue in integration test URL: https://github.com/apache/incubator-druid/pull/8666#issuecomment-545644685 @leventov Do we have the correct aws access key and secret key for the s3 static.druid.io bucket? I do get below error when I'm trying to list the object in that bucket with the keys provided in the integration test. ``` Yun-Fus-MBP15:.aws yun$ aws s3 ls s3://static.druid.io An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] woshifudayun edited a comment on issue #8666: resolve resolveip command not found issue in integration test
woshifudayun edited a comment on issue #8666: resolve resolveip command not found issue in integration test URL: https://github.com/apache/incubator-druid/pull/8666#issuecomment-545644685 @leventov Do we have the correct aws access key and secret key for the s3 static.druid.io bucket? I do get below error when I'm trying to list the object in that bucket with the keys provided in the integration test. ``` Yun-Fus-MBP15:.aws yun$ aws s3 ls s3://static.druid.io An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] woshifudayun edited a comment on issue #8666: resolve resolveip command not found issue in integration test
woshifudayun edited a comment on issue #8666: resolve resolveip command not found issue in integration test URL: https://github.com/apache/incubator-druid/pull/8666#issuecomment-545644685 @leventov Do we have the correct aws access key and secret key for the s3 static.druid.io bucket? I do get below error when I'm trying to list the object in that bucket with the keys provided in the integration test. ```Yun-Fus-MBP15:.aws yun$ aws s3 ls s3://static.druid.io An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] woshifudayun commented on issue #8666: resolve resolveip command not found issue in integration test
woshifudayun commented on issue #8666: resolve resolveip command not found issue in integration test URL: https://github.com/apache/incubator-druid/pull/8666#issuecomment-545644685 @leventov Do we have the correct aws access key and secret key for the s3 static.druid.io bucket? I do get below error when I'm trying to list the object in that bucket with the keys provided in the integration test. ```Yun-Fus-MBP15:.aws yun$ aws s3 ls s3://static.druid.io An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #6573: Druid tasks fail occasionally on Azure Storage
clintropolis commented on issue #6573: Druid tasks fail occasionally on Azure Storage URL: https://github.com/apache/incubator-druid/issues/6573#issuecomment-545641708 Has anyone had the chance to try Druid 0.16 with Azure deep storage to determine if this issue is still present after the changes made in #8296? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6876: Only single-lined json objects inside input files are parseable
stale[bot] commented on issue #6876: Only single-lined json objects inside input files are parseable URL: https://github.com/apache/incubator-druid/issues/6876#issuecomment-545635427 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] yurmix opened a new issue #8726: Docusaurus: Right sidebar is sometime missing
yurmix opened a new issue #8726: Docusaurus: Right sidebar is sometime missing URL: https://github.com/apache/incubator-druid/issues/8726 ### Affected Version 0.16.0 Documentation ### Description The right sidebar doesn't appear for all pages. It also has other side effects (see below). This page has it: https://druid.apache.org/docs/latest/querying/sql.html This doesn't: https://druid.apache.org/docs/latest/querying/groupbyquery.html In pages that don't have the sidebar, the page width is shorter than usual and ends abruptly with a white area, without a scrollbar to suggest further text exists. Example: https://druid.apache.org/docs/latest/querying/query-context.html I suggest to fix it in a way that all pages will have the right sidebar and the same width. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jnaous commented on issue #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times
jnaous commented on issue #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times URL: https://github.com/apache/incubator-druid/pull/8725#issuecomment-545610929 Tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8724: Web console: Button to pretty print Druid JSON query
lgtm-com[bot] commented on issue #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724#issuecomment-545600721 This pull request **introduces 1 alert** when merging aa7f21f5e2b3bc87970df325054cfd2d550f67b6 into 3abd0c4a3324f71615b936ad65a6a67658e0a964 - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-75c65593d26cbc2c293936d049ec2d757faffa22) **new alerts:** * 1 for Potentially inconsistent state update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy commented on issue #8672: Druid Doctor
fjy commented on issue #8672: Druid Doctor URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-545596983 @vogievetsky can you resolve the merge conflicts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky opened a new pull request #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times
vogievetsky opened a new pull request #8725: Web console: disable data loader Submit button when submitting so as not to submit multiple times URL: https://github.com/apache/incubator-druid/pull/8725 Fixes a small issue where it was easy and tempting to accidentally submit a spec multiple times. https://user-images.githubusercontent.com/177816/67426341-bdccfd80-f58e-11e9-9ebf-493eabc2f240.png;> Also makes task checking more robust and allows sending in a context This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #8724: Web console: Button to pretty print Druid JSON query
vogievetsky commented on a change in pull request #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724#discussion_r338198466 ## File path: web-console/src/views/query-view/query-view.tsx ## @@ -354,6 +354,11 @@ export class QueryView extends React.PureComponent
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8724: Web console: Button to pretty print Druid JSON query
lgtm-com[bot] commented on issue #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724#issuecomment-545549202 This pull request **introduces 1 alert** when merging f20951d57c9630179f874353d4b931eb659b5ed9 into 3abd0c4a3324f71615b936ad65a6a67658e0a964 - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-0a2bf386a2fb0d2c2dc0af527fc244ccce062c89) **new alerts:** * 1 for Potentially inconsistent state update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 opened a new pull request #8724: Web console: Button to pretty print Druid JSON query
renevan10 opened a new pull request #8724: Web console: Button to pretty print Druid JSON query URL: https://github.com/apache/incubator-druid/pull/8724 This PR adds a button in the web console to allow a user to prettier-format their Druid query in JSON format, which is useful for viewing. Before button is clicked:  After button is clicked:  Note: This button will only appear if it is a Druid query. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6866: Reverse dependencies from /main/ to /test/
stale[bot] commented on issue #6866: Reverse dependencies from /main/ to /test/ URL: https://github.com/apache/incubator-druid/issues/6866#issuecomment-545541237 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8703: Add docs for docker; fix compose with latest
vogievetsky commented on issue #8703: Add docs for docker; fix compose with latest URL: https://github.com/apache/incubator-druid/pull/8703#issuecomment-545535774 I see the docker compose starts every service individually. Would it be better to not use docker compose and instead use a single docker running the `bin/start-quickstart` script? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #8722: Web console: start transition to React hooks and functional components
fjy merged pull request #8722: Web console: start transition to React hooks and functional components URL: https://github.com/apache/incubator-druid/pull/8722 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (f5b9bf5 -> 3abd0c4)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from f5b9bf5 Cluster-wide configuration for query vectorization (#8657) add 3abd0c4 Web console: start transition to React hooks and functional components (#8722) No new revisions were added by this update. Summary of changes: .../src/components/action-cell/action-cell.tsx | 36 +- .../src/components/action-icon/action-icon.tsx | 8 +- web-console/src/components/auto-form/auto-form.tsx | 14 +- .../components/center-message/center-message.tsx | 22 +- .../clearable-input/clearable-input.spec.tsx | 4 +- .../components/clearable-input/clearable-input.tsx | 28 +- web-console/src/components/deferred/deferred.tsx | 11 +- .../src/components/external-link/external-link.tsx | 19 +- .../__snapshots__/header-bar.spec.tsx.snap | 78 + .../src/components/header-bar/header-bar.tsx | 364 ++--- .../json-collapse/json-collapse.spec.tsx | 4 +- .../src/components/json-collapse/json-collapse.tsx | 6 +- .../src/components/json-input/json-input.spec.tsx | 4 +- .../src/components/json-input/json-input.tsx | 10 +- web-console/src/components/loader/loader.tsx | 58 ++-- .../src/components/menu-checkbox/menu-checkbox.tsx | 14 +- .../components/refresh-button/refresh-button.tsx | 44 ++- .../src/components/rule-editor/rule-editor.tsx | 282 ++-- .../segment-timeline/segment-timeline.tsx | 5 +- .../src/components/show-value/show-value.tsx | 56 ++-- .../suggestible-input/suggestible-input.tsx| 2 +- .../__snapshots__/table-cell.spec.tsx.snap | 16 +- .../src/components/table-cell/table-cell.tsx | 106 +++--- .../table-column-selector.tsx | 48 ++- .../view-control-bar/view-control-bar.tsx | 25 +- .../src/dialogs/about-dialog/about-dialog.tsx | 68 ++-- .../async-action-dialog/async-action-dialog.tsx| 154 - .../datasource-table-action-dialog.tsx | 74 ++--- .../edit-context-dialog/edit-context-dialog.tsx| 92 +++--- .../src/dialogs/history-dialog/history-dialog.tsx | 25 +- .../lookup-edit-dialog/lookup-edit-dialog.tsx | 151 + .../query-history-dialog/query-history-dialog.tsx | 105 ++ .../query-plan-dialog/query-plan-dialog.tsx| 196 ++- .../segment-table-action-dialog.tsx| 74 ++--- .../show-value-dialog/show-value-dialog.tsx| 31 +- .../src/dialogs/spec-dialog/spec-dialog.tsx| 101 +++--- .../supervisor-table-action-dialog.tsx | 143 .../table-action-dialog/table-action-dialog.tsx| 69 ++-- .../task-table-action-dialog.tsx | 152 - web-console/src/utils/load-rule.ts | 99 ++ web-console/src/utils/query-history.ts | 57 .../src/views/datasource-view/datasource-view.tsx | 21 +- .../home-view/home-view-card/home-view-card.tsx| 41 ++- web-console/src/views/home-view/home-view.tsx | 28 +- .../example-picker/example-picker.tsx | 81 ++--- .../__snapshots__/filter-table.spec.tsx.snap | 6 +- .../load-data-view/filter-table/filter-table.tsx | 106 +++--- .../src/views/load-data-view/load-data-view.tsx| 10 +- .../__snapshots__/parse-data-table.spec.tsx.snap | 6 +- .../parse-data-table/parse-data-table.tsx | 138 .../__snapshots__/parse-time-table.spec.tsx.snap | 6 +- .../parse-time-table/parse-time-table.tsx | 158 + .../__snapshots__/schema-table.spec.tsx.snap | 6 +- .../load-data-view/schema-table/schema-table.tsx | 196 ++- .../__snapshots__/transform-table.spec.tsx.snap| 6 +- .../transform-table/transform-table.tsx| 120 --- .../src/views/lookups-view/lookups-view.tsx| 20 +- .../number-menu-items/number-menu-items.tsx| 42 ++- .../string-menu-items/string-menu-items.tsx| 42 ++- .../time-menu-items/time-menu-items.tsx| 209 +--- .../query-extra-info/query-extra-info.tsx | 118 --- .../views/query-view/query-output/query-output.tsx | 222 ++--- web-console/src/views/query-view/query-view.tsx| 8 +- .../src/views/segments-view/segments-view.tsx | 20 +- .../src/views/servers-view/servers-view.tsx| 20 +- web-console/src/views/task-view/tasks-view.tsx | 30 +- web-console/src/visualization/bar-unit.tsx | 32 +- web-console/src/visualization/chart-axis.tsx | 20 +- .../src/visualization/stacked-bar-chart.tsx| 83 ++--- 69 files changed, 2158 insertions(+), 2492 deletions(-) create mode 100644 web-console/src/utils/load-rule.ts create mode 100644 web-console/src/utils/query-history.ts
[incubator-druid] branch master updated (b453fda -> f5b9bf5)
This is an automated email from the ASF dual-hosted git repository. asdf2014 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from b453fda docs: clarify native batch ingestion w/ overlapping segments (#8720) add f5b9bf5 Cluster-wide configuration for query vectorization (#8657) No new revisions were added by this update. Summary of changes: .../benchmark/GroupByTypeInterfaceBenchmark.java | 2 + .../query/CachingClusteredClientBenchmark.java | 2 + .../druid/benchmark/query/GroupByBenchmark.java| 2 + docs/configuration/index.md| 9 +++ docs/querying/query-context.md | 4 +- .../druid/segment/MapVirtualColumnGroupByTest.java | 2 + .../java/org/apache/druid/query/QueryConfig.java | 62 + .../java/org/apache/druid/query/QueryContexts.java | 26 +-- .../epinephelinae/GroupByQueryEngineV2.java| 10 +-- .../epinephelinae/vector/VectorGroupByEngine.java | 7 +- .../query/groupby/strategy/GroupByStrategyV2.java | 12 +++- .../query/timeseries/TimeseriesQueryEngine.java| 20 -- .../java/org/apache/druid/query/TestQuery.java | 2 +- ...GroupByLimitPushDownInsufficientBufferTest.java | 7 +- .../GroupByLimitPushDownMultiNodeMergeTest.java| 6 ++ .../query/groupby/GroupByMultiSegmentTest.java | 2 + .../query/groupby/GroupByQueryMergeBufferTest.java | 2 + .../groupby/GroupByQueryRunnerFailureTest.java | 2 + .../query/groupby/GroupByQueryRunnerTest.java | 2 + .../query/groupby/NestedQueryPushDownTest.java | 6 ++ .../apache/druid/query/search/QueryConfigTest.java | 80 ++ .../apache/druid/guice/QueryToolChestModule.java | 2 + 22 files changed, 247 insertions(+), 22 deletions(-) create mode 100644 processing/src/main/java/org/apache/druid/query/QueryConfig.java create mode 100644 processing/src/test/java/org/apache/druid/query/search/QueryConfigTest.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] asdf2014 merged pull request #8657: Cluster-wide configuration for query vectorization
asdf2014 merged pull request #8657: Cluster-wide configuration for query vectorization URL: https://github.com/apache/incubator-druid/pull/8657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #8723: fix indexer segment announcement when used with http inventory view
clintropolis opened a new pull request #8723: fix indexer segment announcement when used with http inventory view URL: https://github.com/apache/incubator-druid/pull/8723 Fixes #8710. ### Description This PR modifies `CliIndexer` to announce itself as a `dataNodeService` in addition to `lookupNodeService` and `workerNodeService`, so that it functions correctly with `druid.serverview.type=http`. This involved both adding the service class to the `DiscoverySideEffectsProvider` announcer binding, as well as adding a dummy `null` provider for `SegmentLoadDropHandler` to make `SegmentListerResource` (which actually provides the segment "announcement" functionality in http inventory management) happy instead of explodey. MiddleManager nodes do not have this problem because tasks that are not running on an Indexer [perform this announcement themselves](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java#L394), since it is per task. Indexers rightfully suppress this per task announcement which would be super redundant, but were lacking this part of the announcement happening anywhere at an outer level. I could not determine good way to add automated tests for this at this time. I believe integration tests would probably be the most appropriate way, but adding additional containers does not seem like the correct or a scalable approach. Ideally I think the integration tests would probably run in some sort of configuration matrix so we could try out the variations to make sure everything is cool, but we also maybe have too much variation to make this worth the time. In this specific case, maybe continuing to push to phase out reliance on Zookeeper would result in test coverage, assuming this configuration became the default, though I think we still have a lot of qualification to do before that change can be made. I did at least ensure things worked correctly on a tiny laptop cluster once the fix was in place. This PR has: - [x] been self-reviewed. - [x] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website-src] asdf2014 commented on issue #46: Remove broken urls
asdf2014 commented on issue #46: Remove broken urls URL: https://github.com/apache/incubator-druid-website-src/pull/46#issuecomment-545298686 @gianm You are welcome. @Fokko Thanks for your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
