[GitHub] [druid] Awezome opened a new issue #9504: is there any way to get version 0.10.1 documents ?
Awezome opened a new issue #9504: is there any way to get version 0.10.1 documents ? URL: https://github.com/apache/druid/issues/9504 hi, is there any way to get version 0.10.1 documents ? thx ! there is no 0.10.1 documents http://druid.apache.org/docs/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm opened a new pull request #9503: Link up row-based datasources to serving layer.
gianm opened a new pull request #9503: Link up row-based datasources to serving layer. URL: https://github.com/apache/druid/pull/9503 - Add SegmentWrangler interface that allows linking of DataSources to Segments. - Add LocalQuerySegmentWalker that uses SegmentWranglers to compute queries on data that is available locally. - Modify ClientQuerySegmentWalker to use LocalQuerySegmentWalker when the base datasource is concrete and not a table. - Add SegmentWranglerModule to the Broker so it has them available and can properly instantiate . LocalQuerySegmentWalkers. - Set InlineDataSource and LookupDataSource to concrete, since they can be directly queried now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm merged pull request #9502: Ability to directly query row-based datasources.
gianm merged pull request #9502: Ability to directly query row-based datasources. URL: https://github.com/apache/druid/pull/9502 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (c74749f -> 4f08589)
This is an automated email from the ASF dual-hosted git repository. gian pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from c74749f Don't exclude null dimension values from the map based query response (#9438) add 4f08589 Ability to directly query row-based datasources. (#9502) No new revisions were added by this update. Summary of changes: .../apache/druid/query/lookup/LookupSegment.java | 82 +++ .../segment/RowBasedColumnSelectorFactory.java | 17 +- .../org/apache/druid/segment/RowBasedCursor.java | 137 .../org/apache/druid/segment/RowBasedSegment.java | 104 +++ .../druid/segment/RowBasedStorageAdapter.java | 209 ++ .../java/org/apache/druid/segment/RowWalker.java | 88 +++ .../druid/query/lookup/LookupSegmentTest.java | 217 ++ .../org/apache/druid/segment/IndexBuilder.java | 61 +- .../druid/segment/RowBasedStorageAdapterTest.java | 794 + .../druid/segment/filter/BaseFilterTest.java | 55 +- 10 files changed, 1724 insertions(+), 40 deletions(-) create mode 100644 processing/src/main/java/org/apache/druid/query/lookup/LookupSegment.java create mode 100644 processing/src/main/java/org/apache/druid/segment/RowBasedCursor.java create mode 100644 processing/src/main/java/org/apache/druid/segment/RowBasedSegment.java create mode 100644 processing/src/main/java/org/apache/druid/segment/RowBasedStorageAdapter.java create mode 100644 processing/src/main/java/org/apache/druid/segment/RowWalker.java create mode 100644 processing/src/test/java/org/apache/druid/query/lookup/LookupSegmentTest.java create mode 100644 processing/src/test/java/org/apache/druid/segment/RowBasedStorageAdapterTest.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] sbespalov edited a comment on issue #8818: Eclipse Java Compiler issues fixed
sbespalov edited a comment on issue #8818: Eclipse Java Compiler issues fixed URL: https://github.com/apache/druid/pull/8818#issuecomment-597411381 relevant, waiting for review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #8818: Eclipse Java Compiler issues fixed
stale[bot] commented on issue #8818: Eclipse Java Compiler issues fixed URL: https://github.com/apache/druid/pull/8818#issuecomment-597411394 This issue is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] sbespalov commented on issue #8818: Eclipse Java Compiler issues fixed
sbespalov commented on issue #8818: Eclipse Java Compiler issues fixed URL: https://github.com/apache/druid/pull/8818#issuecomment-597411381 relevant This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6996: [Proposal] Adding a Troubleshooting section for developers/contributors
stale[bot] commented on issue #6996: [Proposal] Adding a Troubleshooting section for developers/contributors URL: https://github.com/apache/druid/issues/6996#issuecomment-597411089 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #7608: Does Druid have a plan to provide SQL TASK
stale[bot] closed issue #7608: Does Druid have a plan to provide SQL TASK URL: https://github.com/apache/druid/issues/7608 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #7443: druid ingestion data show java.util.concurrent.TimeoutException
stale[bot] commented on issue #7443: druid ingestion data show java.util.concurrent.TimeoutException URL: https://github.com/apache/druid/issues/7443#issuecomment-597411077 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6507: Indexer should report an ip address along with an instance id during provisioning or termination.
stale[bot] commented on issue #6507: Indexer should report an ip address along with an instance id during provisioning or termination. URL: https://github.com/apache/druid/issues/6507#issuecomment-597411101 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #7586: Support for Datasource and Column comments
stale[bot] closed issue #7586: Support for Datasource and Column comments URL: https://github.com/apache/druid/issues/7586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #6507: Indexer should report an ip address along with an instance id during provisioning or termination.
stale[bot] closed issue #6507: Indexer should report an ip address along with an instance id during provisioning or termination. URL: https://github.com/apache/druid/issues/6507 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #6996: [Proposal] Adding a Troubleshooting section for developers/contributors
stale[bot] closed issue #6996: [Proposal] Adding a Troubleshooting section for developers/contributors URL: https://github.com/apache/druid/issues/6996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #7443: druid ingestion data show java.util.concurrent.TimeoutException
stale[bot] closed issue #7443: druid ingestion data show java.util.concurrent.TimeoutException URL: https://github.com/apache/druid/issues/7443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #7608: Does Druid have a plan to provide SQL TASK
stale[bot] commented on issue #7608: Does Druid have a plan to provide SQL TASK URL: https://github.com/apache/druid/issues/7608#issuecomment-597411093 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #8818: Eclipse Java Compiler issues fixed
stale[bot] commented on issue #8818: Eclipse Java Compiler issues fixed URL: https://github.com/apache/druid/pull/8818#issuecomment-597411037 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #7586: Support for Datasource and Column comments
stale[bot] commented on issue #7586: Support for Datasource and Column comments URL: https://github.com/apache/druid/issues/7586#issuecomment-597411069 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountSta
stale[bot] commented on issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountStatsMonitor" URL: https://github.com/apache/druid/issues/5855#issuecomment-597411055 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9477: Web console: standardize the spec format
jihoonson commented on issue #9477: Web console: standardize the spec format URL: https://github.com/apache/druid/pull/9477#issuecomment-597350361 Other changes look good to me.. At least it works properly in my laptop. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9477: Web console: standardize the spec format
jihoonson commented on a change in pull request #9477: Web console: standardize
the spec format
URL: https://github.com/apache/druid/pull/9477#discussion_r390633016
##
File path: web-console/src/utils/__snapshots__/ingestion-spec.spec.ts.snap
##
@@ -28,9 +21,31 @@ Object {
"type": "longSum",
},
],
-"timestampSpec": Object {
- "column": "timestamp",
- "format": "iso",
+"parser": Object {
Review comment:
Hmm, this change seems like a rollback to the old spec. Is this intended?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm opened a new pull request #9502: Ability to directly query row-based datasources.
gianm opened a new pull request #9502: Ability to directly query row-based datasources. URL: https://github.com/apache/druid/pull/9502 Includes: - Foundational classes RowBasedSegment, RowBasedStorageAdapter, RowBasedCursor provide a queryable interface on top of a RowBasedColumnSelectorFactory. - Add LookupSegment: A RowBasedSegment that is built on lookup data. - Improve capability reporting in RowBasedColumnSelectorFactory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (7401bb3 -> c74749f)
This is an automated email from the ASF dual-hosted git repository. gian pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 7401bb3 Improve OvershadowableManager performance (#9441) add c74749f Don't exclude null dimension values from the map based query response (#9438) No new revisions were added by this update. Summary of changes: .../org/apache/druid/query/groupby/ResultRow.java | 5 +-- .../apache/druid/query/groupby/ResultRowTest.java | 44 ++ 2 files changed, 45 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm closed issue #8631: Null values not returned for group by JSON response
gianm closed issue #8631: Null values not returned for group by JSON response URL: https://github.com/apache/druid/issues/8631 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm merged pull request #9438: Don't exclude null dimension values from the map based query response
gianm merged pull request #9438: Don't exclude null dimension values from the map based query response URL: https://github.com/apache/druid/pull/9438 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #9500: Update druid-query-toolkit to version 4.0
vogievetsky commented on a change in pull request #9500: Update
druid-query-toolkit to version 4.0
URL: https://github.com/apache/druid/pull/9500#discussion_r390620414
##
File path: web-console/src/views/query-view/query-output/query-output.tsx
##
@@ -239,12 +239,14 @@ export const QueryOutput = React.memo(function
QueryOutput(props: QueryOutputPro
const sorted = parsedQuery.getSorted();
if (sorted) {
className.push(
- sorted.map(sorted => {
-if (sorted.id === header) {
- return sorted.desc ? '-sort-desc' : '-sort-asc';
-}
-return '';
- })[0],
+ sorted
+.map(sorted => {
+ if (sorted.id === header) {
+return sorted.desc ? '-sort-desc' : '-sort-asc';
+ }
+ return '';
+})
+.join(''),
Review comment:
This definitely does not make sense this could create classes like
`-sort-desc-sort-asc`
The existing code probably does not make sense either as it will add the
`undefined` class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9501: [WIP] Adding s3, gcs, azure integration tests
maytasm3 commented on a change in pull request #9501: [WIP] Adding s3, gcs,
azure integration tests
URL: https://github.com/apache/druid/pull/9501#discussion_r390618863
##
File path: integration-tests/docker/druid.sh
##
@@ -73,4 +76,20 @@ setupConfig() {
var=$(echo "$evar" | sed -e 's?^\([^=]*\)=.*?\1?g' -e 's?_?.?g')
setKey $DRUID_SERVICE "$var" "$val"
done
-}
\ No newline at end of file
+}
+
+setupData()
Review comment:
This will work for how our travis CI is setup but if user try running
manually locally with...
1) -DexcludedGroups=not-query
2) without -Dgroups and without DexcludedGroups
then the data won't be setup
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh commented on a change in pull request #9490: Fix NPE when using IndexedTable and all left rows are filtered out
ccaominh commented on a change in pull request #9490: Fix NPE when using
IndexedTable and all left rows are filtered out
URL: https://github.com/apache/druid/pull/9490#discussion_r390601557
##
File path:
processing/src/main/java/org/apache/druid/segment/join/table/IndexedTableJoinMatcher.java
##
@@ -81,6 +81,7 @@
)
{
this.table = table;
+this.currentRow = -1;
Review comment:
Perhaps create a named constant for this that's also used by the `reset()`
method in this class
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (7e0e767 -> 7401bb3)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.
from 7e0e767 Ability to Delete task logs and segments from S3 (#9459)
add 7401bb3 Improve OvershadowableManager performance (#9441)
No new revisions were added by this update.
Summary of changes:
.../VersionedIntervalTimelineBenchmark.java| 2 +-
.../druid/timeline/VersionedIntervalTimeline.java | 38 ++-
.../timeline/partition/AtomicUpdateGroup.java | 10 +
.../partition/ImmutablePartitionHolder.java| 4 +-
.../timeline/partition/OvershadowableManager.java | 348 -
.../druid/timeline/partition/PartitionHolder.java | 36 ++-
.../VersionedIntervalTimelineSpecificDataTest.java | 9 +-
.../timeline/VersionedIntervalTimelineTest.java| 82 -
.../VersionedIntervalTimelineTestBase.java | 26 +-
...ardSpecTest.java => AtomicUpdateGroupTest.java} | 55 ++--
.../partition/OvershadowableManagerTest.java | 61 +++-
.../task/batch/parallel/SinglePhaseSubTask.java| 11 +-
.../druid/indexing/input/DruidInputSource.java | 21 +-
.../parallel/SinglePhaseParallelIndexingTest.java | 5 +-
.../tests/indexer/AbstractITBatchIndexTest.java| 13 +-
.../apache/druid/client/DataSourcesSnapshot.java | 1 -
.../IndexerSQLMetadataStorageCoordinator.java | 26 +-
.../duty/NewestSegmentFirstIterator.java | 42 +--
.../coordinator/duty/CompactSegmentsTest.java | 17 +-
19 files changed, 526 insertions(+), 281 deletions(-)
copy
core/src/test/java/org/apache/druid/timeline/partition/{SingleDimensionPartialShardSpecTest.java
=> AtomicUpdateGroupTest.java} (52%)
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9441: Improve OvershadowableManager performance
jihoonson merged pull request #9441: Improve OvershadowableManager performance URL: https://github.com/apache/druid/pull/9441 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson closed issue #9383: Performance regression in OvershadowableManager
jihoonson closed issue #9383: Performance regression in OvershadowableManager URL: https://github.com/apache/druid/issues/9383 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (75a5591 -> 7e0e767)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 75a5591 remove old unused zookeeper dependent lookups code (#9480) add 7e0e767 Ability to Delete task logs and segments from S3 (#9459) No new revisions were added by this update. Summary of changes: .../common/utils/CurrentTimeMillisSupplier.java| 13 +- .../druid/storage/s3/S3DataSegmentKiller.java | 32 ++- .../druid/storage/s3/S3StorageDruidModule.java | 1 + .../org/apache/druid/storage/s3/S3TaskLogs.java| 41 ++- .../apache/druid/storage/s3/S3TaskLogsConfig.java | 6 + .../java/org/apache/druid/storage/s3/S3Utils.java | 52 .../storage/s3/ServerSideEncryptingAmazonS3.java | 6 + .../druid/storage/s3/S3DataSegmentKillerTest.java | 190 + .../apache/druid/storage/s3/S3TaskLogsTest.java| 307 +++-- .../org/apache/druid/storage/s3/S3TestUtils.java | 180 10 files changed, 794 insertions(+), 34 deletions(-) copy server/src/main/java/org/apache/druid/server/ZkPathsModule.java => core/src/main/java/org/apache/druid/common/utils/CurrentTimeMillisSupplier.java (80%) create mode 100644 extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3DataSegmentKillerTest.java create mode 100644 extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3TestUtils.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9459: Ability to Delete task logs and segments from S3
clintropolis merged pull request #9459: Ability to Delete task logs and segments from S3 URL: https://github.com/apache/druid/pull/9459 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 opened a new pull request #9501: Adding s3, gcs, azure integration tests
maytasm3 opened a new pull request #9501: Adding s3, gcs, azure integration tests URL: https://github.com/apache/druid/pull/9501 Adding s3, gcs, azure integration tests ### Description Adding s3, gcs, azure integration tests which can be run BYO cloud style. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. # Key changed/added classes in this PR * `MyFoo` * `OurBar` * `TheirBaz` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] mcbrewster opened a new pull request #9500: Update druid-query-toolkit to version 4.0
mcbrewster opened a new pull request #9500: Update druid-query-toolkit to version 4.0 URL: https://github.com/apache/druid/pull/9500 https://user-images.githubusercontent.com/37322608/76350585-42fc4900-62c9-11ea-89ce-7af41478c8f6.png";> Updates the druid query toolkit to the latest version. The latest version is more throughly tested and can parse a wider range of queries including those with comments. Minor adjustments have been made to the api. Additionally the class to denote the direction of a query was not applying the correct class to all ordered columns, this is fixed now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #9480: remove old unused zookeeper dependent lookups code
himanshug merged pull request #9480: remove old unused zookeeper dependent lookups code URL: https://github.com/apache/druid/pull/9480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (559c7b6 -> 75a5591)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 559c7b6 Suppress CVEs for htrace-core4 and openstack-swift (#9489) add 75a5591 remove old unused zookeeper dependent lookups code (#9480) No new revisions were added by this update. Summary of changes: docs/querying/lookups.md | 4 +- .../lookup/LookupListeningAnnouncerConfig.java | 11 +- .../apache/druid/query/lookup/LookupModule.java| 10 - .../lookup/LookupResourceListenerAnnouncer.java| 46 -- .../announcer/ListenerResourceAnnouncer.java | 111 - .../announcer/ListeningAnnouncerConfig.java| 101 - .../listener/resource/AbstractListenerHandler.java | 4 +- .../server/listener/resource/ListenerResource.java | 3 - .../announcer/ListenerResourceAnnouncerTest.java | 134 -- .../resource/AbstractListenerHandlerTest.java | 277 .../listener/resource/ListenerResourceTest.java| 491 - 11 files changed, 4 insertions(+), 1188 deletions(-) delete mode 100644 server/src/main/java/org/apache/druid/query/lookup/LookupResourceListenerAnnouncer.java delete mode 100644 server/src/main/java/org/apache/druid/server/listener/announcer/ListenerResourceAnnouncer.java delete mode 100644 server/src/main/java/org/apache/druid/server/listener/announcer/ListeningAnnouncerConfig.java delete mode 100644 server/src/test/java/org/apache/druid/server/listener/announcer/ListenerResourceAnnouncerTest.java delete mode 100644 server/src/test/java/org/apache/druid/server/listener/resource/AbstractListenerHandlerTest.java delete mode 100644 server/src/test/java/org/apache/druid/server/listener/resource/ListenerResourceTest.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9480: remove old unused zookeeper dependent lookups code
himanshug commented on issue #9480: remove old unused zookeeper dependent lookups code URL: https://github.com/apache/druid/pull/9480#issuecomment-597263582 @clintropolis thanks for quick review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9407: query laning and load shedding
himanshug commented on a change in pull request #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407#discussion_r390549892 ## File path: docs/configuration/index.md ## @@ -1481,9 +1481,35 @@ These Broker configurations can be defined in the `broker/runtime.properties` fi |`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`| |`druid.broker.select.tier.custom.priorities`|`An array of integer priorities.`|Select servers in tiers with a custom priority list.|None| +# Query laning + +*Laning strategies* allow you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane. Requests in excess of the capacity are discarded with an HTTP 429 status code. + +|Property|Description|Default| +||---|---| +|`druid.query.scheduler.numThreads`|Maximum number of HTTP threads to dedicate to query processing. To save HTTP thread capacity, this should be lower than `druid.server.http.numThreads`.|Unbounded| Review comment: thanks, I understand the reasoning now. lookup end points already have a QoS filter to never consume more than two threads from jetty, I wonder if in this world it makes sense to setup QoS filter for non-query endpoints(say hardcoded to 2) so that we can ensure that they don't end up consuming more jetty threads than intended. then default `druid.query.scheduler.numThreads = druid.server.http.numThreads - numReservedForOthers=4` and users would likely never be expected to touch `druid.query.scheduler.numThreads` . Major behavior change with lane usage is really losing the queuing of requests to handle spikes and instead sending 429s immediately. In future, we could introduce mechanism to maintain statically/dynamically sized [per lane] waiting queue ourselves as well along with concurrency limits in lane strategy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9459: Ability to Delete task logs and segments from S3
zachjsh commented on a change in pull request #9459: Ability to Delete task
logs and segments from S3
URL: https://github.com/apache/druid/pull/9459#discussion_r390538553
##
File path:
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java
##
@@ -200,6 +204,54 @@ public static S3ObjectSummary
getSingleObjectSummary(ServerSideEncryptingAmazonS
return objectSummary;
}
+ public static void deleteObjectsInPath(
+ ServerSideEncryptingAmazonS3 s3Client,
+ S3InputDataConfig config,
+ String bucket,
+ String prefix,
+ Predicate filter
+ )
+ throws Exception
+ {
+final List keysToDelete = new
ArrayList<>(config.getMaxListingLength());
+final ObjectSummaryIterator iterator = new ObjectSummaryIterator(
+s3Client,
+ImmutableList.of(new CloudObjectLocation(bucket, prefix).toUri("s3")),
+config.getMaxListingLength()
+);
+
+while (iterator.hasNext()) {
+ final S3ObjectSummary nextObject = iterator.next();
+ if (filter.apply(nextObject)) {
+keysToDelete.add(new
DeleteObjectsRequest.KeyVersion(nextObject.getKey()));
+if (keysToDelete.size() == config.getMaxListingLength()) {
+ deleteBucketKeys(s3Client, bucket, keysToDelete);
+ log.info("Deleted %d files", keysToDelete.size());
+ keysToDelete.clear();
+}
+ }
+}
+
+if (keysToDelete.size() > 0) {
+ deleteBucketKeys(s3Client, bucket, keysToDelete);
+ log.info("Deleted %d files", keysToDelete.size());
+}
+ }
+
+ public static void deleteBucketKeys(
Review comment:
I'll get this in the next change which should be coming shortly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9459: Ability to Delete task logs and segments from S3
zachjsh commented on a change in pull request #9459: Ability to Delete task logs and segments from S3 URL: https://github.com/apache/druid/pull/9459#discussion_r390538564 ## File path: extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java ## @@ -200,6 +204,54 @@ public static S3ObjectSummary getSingleObjectSummary(ServerSideEncryptingAmazonS return objectSummary; } + public static void deleteObjectsInPath( Review comment: I'll get this in the next change which should be coming shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9443: Hdfs inputSource not supported by native batch ingestion v0.17
jihoonson commented on issue #9443: Hdfs inputSource not supported by native batch ingestion v0.17 URL: https://github.com/apache/druid/issues/9443#issuecomment-597248502 If you have loaded it, it should be logged at the beginning of the task logs. Would you please check the task logs? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky opened a new pull request #9499: Web console: add config control for the query context
vogievetsky opened a new pull request #9499: Web console: add config control for the query context URL: https://github.com/apache/druid/pull/9499 Allow the console config to pass in default / mandatory query contexts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jalbs commented on issue #9443: Hdfs inputSource not supported by native batch ingestion v0.17
jalbs commented on issue #9443: Hdfs inputSource not supported by native batch ingestion v0.17 URL: https://github.com/apache/druid/issues/9443#issuecomment-597234575 Yes, I have that extension loaded along with parquet, avro and mysql-metadata This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh commented on a change in pull request #9489: Suppress CVEs for htrace-core4 and openstack-swift
ccaominh commented on a change in pull request #9489: Suppress CVEs for htrace-core4 and openstack-swift URL: https://github.com/apache/druid/pull/9489#discussion_r390508531 ## File path: owasp-dependency-check-suppressions.xml ## @@ -169,27 +170,18 @@ -^pkg:maven/com\.fasterxml\.jackson\.core/jackson\-databind@.*$ -CVE-2017-7525 -CVE-2017-15095 -CVE-2017-17485 -CVE-2018-5968 -CVE-2018-7489 -CVE-2018-11307 -CVE-2018-14718 -CVE-2018-14719 -CVE-2018-14720 -CVE-2018-14721 -CVE-2018-19360 -CVE-2018-19361 -CVE-2018-19362 -CVE-2019-14540 -CVE-2019-16335 -CVE-2019-16942 -CVE-2019-16943 -CVE-2019-17267 -CVE-2019-17531 -CVE-2019-20330 -CVE-2020-8840 +^pkg:maven/com\.fasterxml\.jackson\.core/jackson\-databind@2.4.0$ +CVE-2018-14721 +10 + + +
[GitHub] [druid] ccaominh opened a new issue #9498: Update parquet version in parquet-extensions to address security vulnerabilities
ccaominh opened a new issue #9498: Update parquet version in parquet-extensions to address security vulnerabilities URL: https://github.com/apache/druid/issues/9498 ### Description The latest version (1.1.0) of parquet includes a shaded jackson-databind:2.9.10, which has security vulnerabilities with CVSS 7 or higher. When a newer version of parquet is available with an updated version of jackson (2.9.10+), update the version used by parquet-extensions. ### Motivation Address security vulnerabilities. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9489: Suppress CVEs for htrace-core4 and openstack-swift
jihoonson merged pull request #9489: Suppress CVEs for htrace-core4 and openstack-swift URL: https://github.com/apache/druid/pull/9489 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (c6c2282 -> 559c7b6)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from c6c2282 Harmonization and bug-fixing for selector and filter behavior on unknown types. (#9484) add 559c7b6 Suppress CVEs for htrace-core4 and openstack-swift (#9489) No new revisions were added by this update. Summary of changes: owasp-dependency-check-suppressions.xml | 44 ++--- 1 file changed, 18 insertions(+), 26 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh commented on issue #8432: Druid Vulnerability Analysis
ccaominh commented on issue #8432: Druid Vulnerability Analysis URL: https://github.com/apache/druid/issues/8432#issuecomment-597221572 Fixed by #8878 and #8980 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh closed issue #8432: Druid Vulnerability Analysis
ccaominh closed issue #8432: Druid Vulnerability Analysis URL: https://github.com/apache/druid/issues/8432 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] Observe-secretly opened a new issue #9497: The Druid I build occasionally shows the error "Rename cannot overwrite non empty destination directory / tmp / hadoop-root / mapred / local
Observe-secretly opened a new issue #9497: The Druid I build occasionally shows
the error "Rename cannot overwrite non empty destination directory / tmp /
hadoop-root / mapred / local / 1583859663629". I have no way to solve this
problem now. please help me
URL: https://github.com/apache/druid/issues/9497
### Affected Version
0.17.0
### Description
Our Druid cluster occasionally shows the error "Rename cannot overwrite non
empty destination directory / tmp / hadoop-root / mapred / local /
1583859663629". It did not have this problem in the early stage of operation.
So he is a problem that has only recently appeared. We did not clean up any
files under "/ tmp / hadoop-root / mapred". We haven't found any information
stored in this directory. The following is the exception stack, hope to help me
locate the problem. thank
### Exception stack
```
2020-03-10 17:00:53,654 task-runner-0-priority-0 WARN JNDI lookup class is
not available because this JRE does not support JNDI. JNDI string lookups will
not be available, continuing configuration. Ignoring
java.lang.ClassCastException: Cannot cast
org.apache.logging.log4j.core.lookup.JndiLookup to
org.apache.logging.log4j.core.lookup.StrLookup
2020-03-10 17:00:53,667 task-runner-0-priority-0 WARN JMX runtime input
lookup class is not available because this JRE does not support JMX. JMX
lookups will not be available, continuing configuration. Ignoring
java.lang.ClassCastException: Cannot cast
org.apache.logging.log4j.core.lookup.JmxRuntimeInputArgumentsLookup to
org.apache.logging.log4j.core.lookup.StrLookup
2020-03-10 17:00:53,725 task-runner-0-priority-0 WARN JNDI lookup class is
not available because this JRE does not support JNDI. JNDI string lookups will
not be available, continuing configuration. Ignoring
java.lang.ClassCastException: Cannot cast
org.apache.logging.log4j.core.lookup.JndiLookup to
org.apache.logging.log4j.core.lookup.StrLookup
2020-03-10 17:00:53,726 task-runner-0-priority-0 WARN JMX runtime input
lookup class is not available because this JRE does not support JMX. JMX
lookups will not be available, continuing configuration. Ignoring
java.lang.ClassCastException: Cannot cast
org.apache.logging.log4j.core.lookup.JmxRuntimeInputArgumentsLookup to
org.apache.logging.log4j.core.lookup.StrLookup
2020-03-10T17:00:54,888 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@68df8c6{/,null,AVAILABLE}
2020-03-10T17:00:54,965 INFO [main]
org.eclipse.jetty.server.AbstractConnector - Started
ServerConnector@37f41a81{HTTP/1.1,[http/1.1]}{0.0.0.0:8101}
2020-03-10T17:00:54,965 INFO [main] org.eclipse.jetty.server.Server -
Started @20088ms
2020-03-10T17:00:54,966 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle
[module] stage [ANNOUNCEMENTS]
2020-03-10T17:00:55,053 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started
lifecycle [module]
2020-03-10T17:00:55,166 INFO [task-runner-0-priority-0]
org.hibernate.validator.internal.util.Version - HV01: Hibernate Validator
5.2.5.Final
2020-03-10T17:00:56,849 INFO [task-runner-0-priority-0]
org.apache.druid.initialization.Initialization - Loading extension
[druid-hdfs-storage], jars: hadoop-mapreduce-client-core-2.8.5.jar,
hadoop-yarn-api-2.8.5.jar, commons-configuration-1.6.jar,
apacheds-i18n-2.0.0-M15.jar, hadoop-common-2.8.5.jar,
jetty-sslengine-6.1.26.jar, hadoop-client-2.8.5.jar,
curator-framework-4.1.0.jar, htrace-core4-4.0.1-incubating.jar,
commons-digester-1.8.jar, jcip-annotations-1.0-1.jar, xmlenc-0.52.jar,
hadoop-mapreduce-client-app-2.8.5.jar, json-smart-2.3.jar,
hadoop-auth-2.8.5.jar, asm-7.1.jar, jackson-core-asl-1.9.13.jar,
jsp-api-2.1.jar, hadoop-yarn-client-2.8.5.jar, api-util-1.0.3.jar,
commons-collections-3.2.2.jar, api-asn1-api-1.0.0-M20.jar,
apacheds-kerberos-codec-2.0.0-M15.jar, hadoop-yarn-server-common-2.8.5.jar,
hadoop-annotations-2.8.5.jar, hadoop-mapreduce-client-jobclient-2.8.5.jar,
hadoop-hdfs-client-2.8.5.jar, curator-recipes-4.1.0.jar,
accessors-smart-1.2.jar, gson-2.2.4.jar, leveldbjni-all-1.8.jar,
commons-net-3.6.jar, jackson-mapper-asl-1.9.13.jar,
hadoop-mapreduce-client-common-2.8.5.jar,
hadoop-mapreduce-client-shuffle-2.8.5.jar, nimbus-jose-jwt-4.41.1.jar,
druid-hdfs-storage-0.17.0.jar
2020-03-10T17:00:56,864 INFO [task-runner-0-priority-0]
org.apache.druid.initialization.Initialization - Loading extension
[druid-kafka-indexing-service], jars: druid-kafka-indexing-service-0.17.0.jar,
snappy-java-1.1.7.2.jar, zstd-jni-1.3.3-1.jar, lz4-java-1.6.0.jar,
kafka-clients-2.2.1.jar
2020-03-10T17:00:56,907 INFO [task-runner-0-priority-0]
org.apache.druid.initialization.Initialization - Loading extension
[druid-datasketches], jars: druid-datasketches-0.17.0.jar,
commons-math3-3.6.1.jar
2020-03-10T17:00:56,909 INFO [ta
[GitHub] [druid] ceastman-ibm opened a new issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
ceastman-ibm opened a new issue #9496: druid management ui in kubernetes fails
if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496
### Affected Version
0.17.0
### Description
individual deployment/stateful set with seven replicas each of:
historical/data
coordinator/overlord
broker/query
ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "name" . }}
{{- with .Values.druid.ingress.annotations }}
annotations:
{{- . | toYaml | nindent 4 }}
{{- end }}
spec:
tls:
- hosts:
- {{ .Values.druid.ingress.hostnameRegional | quote }}
- {{ .Values.druid.ingress.hostnameGlobal | quote }}
secretName: {{ template "name" . }}-global-ssl-secret
rules:
- host: {{ .Values.druid.ingress.hostnameRegional }}
http:
paths:
- path: /
backend:
serviceName: druid-router
servicePort:
- path: /druid/v2/sql
backend:
serviceName: druid-broker
servicePort: 8082
- path: /druid/coordinator
backend:
serviceName: druid-coordinator
servicePort: 8081
- path: /druid/indexer
backend:
serviceName: druid-overlord
servicePort: 8081
- host: {{ .Values.druid.ingress.hostnameGlobal }}
http:
paths:
- path: /
backend:
serviceName: druid-router
servicePort:
- path: /druid/v2/sql
backend:
serviceName: druid-broker
servicePort: 8082
- path: /druid/coordinator
backend:
serviceName: druid-coordinator
servicePort: 8081
- path: /druid/indexer
backend:
serviceName: druid-overlord
servicePort: 8081
https://user-images.githubusercontent.com/25593207/76340220-3a0b7780-62c9-11ea-8763-96ff6ce586d5.png";>
when looking at the browser diagnostics i see these two urls are returning
404s:
9:57
https://<>/druid/coordinator/v1/isLeader
https://<>/druid/indexer/v1/isLeader
when i scale the deployment of the coordinator/overlord to one replica the
issue goes away.
when i go directly to those two urls listed above it returns a json object:
isLeader: false
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9407: query laning and load shedding
himanshug commented on a change in pull request #9407: query laning and load
shedding
URL: https://github.com/apache/druid/pull/9407#discussion_r390469100
##
File path:
server/src/main/java/org/apache/druid/server/QueryLaningStrategy.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.fasterxml.jackson.annotation.JsonTypeInfo;
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import org.apache.druid.client.SegmentServerSelector;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.server.scheduling.HiLoQueryLaningStrategy;
+import org.apache.druid.server.scheduling.NoQueryLaningStrategy;
+
+import java.util.Optional;
+import java.util.Set;
+
+
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "strategy", defaultImpl =
NoQueryLaningStrategy.class)
+@JsonSubTypes(value = {
+@JsonSubTypes.Type(name = "none", value = NoQueryLaningStrategy.class),
+@JsonSubTypes.Type(name = "hilo", value = HiLoQueryLaningStrategy.class)
+})
+public interface QueryLaningStrategy
+{
+ /**
+ * Provide a map of lane names to the limit on the number of concurrent
queries for that lane
+ * @param totalLimit
+ */
+ Object2IntMap getLaneLimits(int totalLimit);
Review comment:
Hehe, that happens with multiple reviewers. Yeah, having a utility method is
equally good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] mkapoor17 commented on issue #9495: Switch Docker base to CentOS and include kerberos tools
mkapoor17 commented on issue #9495: Switch Docker base to CentOS and include kerberos tools URL: https://github.com/apache/druid/issues/9495#issuecomment-597191500 I too would advocate for this change. To utilize Kerberos and the kafka extension, I ended creating my own custom image that is now a lot different than the base image and went through a lot of headache trying to get ktutil to work - changing to CentOS did the trick. I do want to go back to the official image but the significant differences between the OS distros and ktutil output makes it unlikely that I will. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] sascha-coenen edited a comment on issue #9493: threshold based automatic query prioritization
sascha-coenen edited a comment on issue #9493: threshold based automatic query prioritization URL: https://github.com/apache/druid/pull/9493#issuecomment-597170972 AWESOME! I love it. This will be so useful. If a query surpasses the thresholds several times, for instance it would be several times the segment threshold or the duration threshold would fit into the query time range several times, would the "adjustment" be decremented several times too? With the laning being implemented, will this become an alternative to the 'adjustment" property, to specify different lanes based on the query weight? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] sascha-coenen commented on issue #9493: threshold based automatic query prioritization
sascha-coenen commented on issue #9493: threshold based automatic query prioritization URL: https://github.com/apache/druid/pull/9493#issuecomment-597170972 AWESOME! I love it. This will be so useful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xrl opened a new issue #9495: Change Docker base to be CentOS and also include kerberos tools
xrl opened a new issue #9495: Change Docker base to be CentOS and also include kerberos tools URL: https://github.com/apache/druid/issues/9495 ### Description We are using Druid with a SASL (GSSAPI/Kerberos) Kafka cluster. In order to regenerate tokens we provide the jaas.conf with a script for running ktutil. The ktutil which comes in Debian does not allow us to feed in the password from the script but the CentOS ktutil does work. ### Motivation It would be helpful to all Druid users talking to a kerberized kafka if docker image was all ready to go. I think Kafka is sufficiently common that it makes sense to provide this authentication tooling out of the box. This would also keep Druid users on the official docker image, so future bug fixes and security patches are easy to distribute with a docker tag update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] rbkumar88 opened a new issue #9494: Support for wildcard in dimensionExclusions
rbkumar88 opened a new issue #9494: Support for wildcard in dimensionExclusions URL: https://github.com/apache/druid/issues/9494 We have a set of dynamic fields which share the same prefix (dynafield-123,dynafield-234), would be nice if we can have dimensionExclusions include the support of excluding such fields say like dynafield-* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm merged pull request #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types.
gianm merged pull request #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types. URL: https://github.com/apache/druid/pull/9484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm edited a comment on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types.
gianm edited a comment on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types. URL: https://github.com/apache/druid/pull/9484#issuecomment-597108855 > Tagged release notes because this PR changes the behavior of complex metric aggregation at ingestion time when SQL compatible null handling is _disabled_ (the default mode) to now aggregate the default 0 values for rows instead of skipping them. This change is for the better imo since it makes things symmetrical to as if you ingested the raw data and built the sketch at query time, but it is different so worth calling out, and you can see the effects in some of the test changes in this PR. Thanks for pointing that out. Yes, I agree, it is for the better since it makes the ingest-time behavior and query-time behavior the same. This is part of the promise of Druid rollup in the first place (you can move aggregations to ingest time if you want). Btw, this patch also ends up making ingest-time transforms and filters behave more consistently with query-time ones. The reason is that all this ingest-time stuff runs in unknown-type mode, which til now had various inconsistencies with known-type mode (which is used at query time). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: Harmonization and bug-fixing for selector and filter behavior on unknown types. (#9484)
This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new c6c2282 Harmonization and bug-fixing for selector and filter behavior
on unknown types. (#9484)
c6c2282 is described below
commit c6c2282b59cda107089a9b3944477fd630bc0657
Author: Gian Merlino
AuthorDate: Tue Mar 10 07:15:57 2020 -0700
Harmonization and bug-fixing for selector and filter behavior on unknown
types. (#9484)
* Harmonization and bug-fixing for selector and filter behavior on unknown
types.
- Migrate ValueMatcherColumnSelectorStrategy to newer ColumnProcessorFactory
system, and set defaultType COMPLEX so unknown types can be dynamically
matched.
- Remove ValueGetters in favor of ColumnComparisonFilter doing its own
thing.
- Switch various methods to use convertObjectToX when casting to numbers,
rather
than ad-hoc and inconsistent logic.
- Fix bug in RowBasedExpressionColumnValueSelector: isBindingArray should
return
true even for 0- or 1- element arrays.
- Adjust various javadocs.
* Add throwParseExceptions option to Rows.objectToNumber, switch back to
that.
* Update tests.
* Adjust moment sketch tests.
---
.../org/apache/druid/data/input/MapBasedRow.java | 2 +-
.../java/org/apache/druid/data/input/Rows.java | 40 ++-
.../aggregator/MomentsSketchAggregatorTest.java| 18 +-
.../tuple/ArrayOfDoublesSketchAggregationTest.java | 16 +-
.../ApproximateHistogramFoldingSerde.java | 4 +-
.../histogram/FixedBucketsHistogramSerde.java | 2 +-
.../apache/druid/indexer/InputRowSerdeTest.java| 2 +-
.../DoubleValueMatcherColumnSelectorStrategy.java | 95 --
.../FloatValueMatcherColumnSelectorStrategy.java | 94 --
.../org/apache/druid/query/filter/InDimFilter.java | 31 +-
.../LongValueMatcherColumnSelectorStrategy.java| 93 --
.../druid/query/filter/SelectorDimFilter.java | 120 +--
.../query/filter/SelectorPredicateFactory.java | 150 +
.../StringValueMatcherColumnSelectorStrategy.java | 131
.../org/apache/druid/query/filter/ValueGetter.java | 37 ---
.../apache/druid/query/filter/ValueMatcher.java| 26 --
.../filter/ValueMatcherColumnSelectorStrategy.java | 51 ---
.../ValueMatcherColumnSelectorStrategyFactory.java | 63
.../SingleValueStringVectorValueMatcher.java | 4 +-
.../epinephelinae/RowBasedGrouperHelper.java | 7 +-
.../timeseries/TimeseriesQueryQueryToolChest.java | 16 +-
.../druid/segment/ColumnProcessorFactory.java | 28 +-
.../org/apache/druid/segment/ColumnProcessors.java | 73 -
.../druid/segment/DimensionHandlerUtils.java | 5 -
.../java/org/apache/druid/segment/RowAdapters.java | 60
.../segment/RowBasedColumnSelectorFactory.java | 96 +++---
.../druid/segment/column/ColumnCapabilities.java | 7 +-
.../segment/filter/ColumnComparisonFilter.java | 117 +--
.../filter/ConstantValueMatcherFactory.java| 83 +
.../org/apache/druid/segment/filter/Filters.java | 51 ++-
.../filter/PredicateValueMatcherFactory.java | 205
.../apache/druid/segment/filter/ValueMatchers.java | 365 +
.../segment/incremental/IncrementalIndex.java | 8 +-
.../segment/join/lookup/LookupJoinMatcher.java | 2 +-
.../join/table/IndexedTableJoinMatcher.java| 2 +-
.../druid/segment/transform/Transformer.java | 9 +-
.../RowBasedExpressionColumnValueSelector.java | 2 +-
.../query/groupby/GroupByQueryRunnerTest.java | 4 +-
.../druid/segment/filter/BaseFilterTest.java | 13 +-
.../druid/segment/filter/ExpressionFilterTest.java | 5 +-
.../druid/segment/filter/SelectorFilterTest.java | 3 +-
.../apache/druid/segment/join/JoinTestHelper.java | 2 +-
.../join/table/IndexedTableJoinMatcherTest.java| 2 +-
.../virtual/ExpressionVirtualColumnTest.java | 37 ++-
44 files changed, 1280 insertions(+), 901 deletions(-)
diff --git a/core/src/main/java/org/apache/druid/data/input/MapBasedRow.java
b/core/src/main/java/org/apache/druid/data/input/MapBasedRow.java
index ae564e9..229a385 100644
--- a/core/src/main/java/org/apache/druid/data/input/MapBasedRow.java
+++ b/core/src/main/java/org/apache/druid/data/input/MapBasedRow.java
@@ -90,7 +90,7 @@ public class MapBasedRow implements Row
@Override
public Number getMetric(String metric)
{
-return Rows.objectToNumber(metric, event.get(metric));
+return Rows.objectToNumber(metric, event.get(metric), true);
}
@Override
diff --git a/core/src/main/java/org/apache/druid/data/input/Rows.java
b/core/src/main/java/org/apache/druid/data/input/Rows.java
index 545d98f..4f6d71b 100644
--- a/core/src/main/java/org/apache/druid/data/input/Rows.jav
[GitHub] [druid] gianm commented on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types.
gianm commented on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types. URL: https://github.com/apache/druid/pull/9484#issuecomment-597108855 > Tagged release notes because this PR changes the behavior of complex metric aggregation at ingestion time when SQL compatible null handling is _disabled_ (the default mode) to now aggregate the default 0 values for rows instead of skipping them. This change is for the better imo since it makes things symmetrical to as if you ingested the raw data and built the sketch at query time, but it is different so worth calling out, and you can see the effects in some of the test changes in this PR. Thanks for pointing that out. Yes, I agree, it is for the better since it makes the ingest-time behavior and query-time behavior the same. This is part of the promise of Druid rollup in the first place (you can move aggregations to ingest time if you want). Btw, this patch also ends up making ingest-time transforms and filters behave more consistently with query-time ones. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis edited a comment on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types.
clintropolis edited a comment on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types. URL: https://github.com/apache/druid/pull/9484#issuecomment-597053167 Tagged release notes because this PR changes the behavior of complex metric aggregation at ingestion time when SQL compatible null handling is _disabled_ (the default mode) to now aggregate the default 0 values for rows instead of skipping them. This change is for the better imo since it makes things symmetrical to as if you ingested the raw data and built the sketch at query time, but it is different so worth calling out, and you can see the effects in some of the test changes in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types.
clintropolis commented on issue #9484: Harmonization and bug-fixing for selector and filter behavior on unknown types. URL: https://github.com/apache/druid/pull/9484#issuecomment-597053167 Tagged release notes because this PR changes the behavior of complex metric aggregation at ingestion time when SQL compatible null handling is _disabled_ (the default mode) to now aggregate the default 0 values for rows instead of skipping them. This change is for the better imo since it makes things symmetrical to as if you ingested the raw data and built the sketch at query time, but it is different so worth calling out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9459: Ability to Delete task logs and segments from S3
clintropolis commented on a change in pull request #9459: Ability to Delete
task logs and segments from S3
URL: https://github.com/apache/druid/pull/9459#discussion_r390066428
##
File path:
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java
##
@@ -200,6 +204,54 @@ public static S3ObjectSummary
getSingleObjectSummary(ServerSideEncryptingAmazonS
return objectSummary;
}
+ public static void deleteObjectsInPath(
+ ServerSideEncryptingAmazonS3 s3Client,
+ S3InputDataConfig config,
+ String bucket,
+ String prefix,
+ Predicate filter
+ )
+ throws Exception
+ {
+final List keysToDelete = new
ArrayList<>(config.getMaxListingLength());
+final ObjectSummaryIterator iterator = new ObjectSummaryIterator(
+s3Client,
+ImmutableList.of(new CloudObjectLocation(bucket, prefix).toUri("s3")),
+config.getMaxListingLength()
+);
+
+while (iterator.hasNext()) {
+ final S3ObjectSummary nextObject = iterator.next();
+ if (filter.apply(nextObject)) {
+keysToDelete.add(new
DeleteObjectsRequest.KeyVersion(nextObject.getKey()));
+if (keysToDelete.size() == config.getMaxListingLength()) {
+ deleteBucketKeys(s3Client, bucket, keysToDelete);
+ log.info("Deleted %d files", keysToDelete.size());
+ keysToDelete.clear();
+}
+ }
+}
+
+if (keysToDelete.size() > 0) {
+ deleteBucketKeys(s3Client, bucket, keysToDelete);
+ log.info("Deleted %d files", keysToDelete.size());
+}
+ }
+
+ public static void deleteBucketKeys(
Review comment:
nit: this can be private actually (my bad)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9459: Ability to Delete task logs and segments from S3
clintropolis commented on a change in pull request #9459: Ability to Delete task logs and segments from S3 URL: https://github.com/apache/druid/pull/9459#discussion_r390067077 ## File path: extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java ## @@ -200,6 +204,54 @@ public static S3ObjectSummary getSingleObjectSummary(ServerSideEncryptingAmazonS return objectSummary; } + public static void deleteObjectsInPath( Review comment: nit: javadocs describing this method would be nice This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #9493: threshold based automatic query prioritization
clintropolis opened a new pull request #9493: threshold based automatic query prioritization URL: https://github.com/apache/druid/pull/9493 ### Description This PR is a follow-up to #9407 that adds a new interface `QueryPrioritizationStrategy` intended to enable implementations to automatically prioritize queries based on some criteria. As a proof of concept implementation of this functionality, it provides `ThresholdBasedQueryDeprioritizationStrategy`, which offers the 3 thresholds of: period from the current time of the query, duration of the interval of the query, and number of segments taking part in the query, described in #6993. This strategy can be enabled by setting `druid.query.scheduler.prioritization.strategy` to `threshold`. |Property|Description|Default| ||---|---| |`druid.query.scheduler.prioritization.periodThreshold`|ISO duration threshold for how old data can be queried before automatically adjusting query priority.|None| |`druid.query.scheduler.prioritization.durationThreshold`|ISO duration threshold for maximum duration a queries interval can span before the priority is automatically adjusted.|None| |`druid.query.scheduler.prioritization.segmentCountThreshold`|Number threshold for maximum number of segments that can take part in a query before its priority is automatically adjusted.|None| |`druid.query.scheduler.prioritization.adjustment`|Amount to reduce the priority of queries which cross any threshold.|None| This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #9492: add manual laning strategy, integration test
clintropolis opened a new pull request #9492: add manual laning strategy, integration test URL: https://github.com/apache/druid/pull/9492 ### Description This PR is a follow-up to #9407 which adds a `ManualLaningStrategy` that in addition to being useful to make it easy to add an integration test for the query laning functionality, is also well suited for cases where one or more external applications which query Druid are able to manually decide which lane a query belongs to by adding a `lane` parameter to the query context. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [x] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths. - [x] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query
stale[bot] commented on issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query URL: https://github.com/apache/druid/issues/5180#issuecomment-597006769 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (75e2051 -> 8b9fe6f)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.
from 75e2051 Convert array_contains() and array_overlaps() into native
filters if possible (#9487)
add 8b9fe6f query laning and load shedding (#9407)
No new revisions were added by this update.
Summary of changes:
.../query/CachingClusteredClientBenchmark.java | 6 +-
distribution/bin/check-licenses.py | 1 +
docs/configuration/index.md| 28 +-
docs/querying/query-context.md | 1 +
docs/querying/querying.md | 2 +
.../movingaverage/MovingAverageQueryTest.java | 5 +-
.../common/task/RealtimeIndexTaskTest.java | 2 +-
licenses.yaml | 22 +
pom.xml| 6 +
.../druid/query/ChainedExecutionQueryRunner.java | 2 +-
.../druid/query/GroupByMergedQueryRunner.java | 2 +-
.../main/java/org/apache/druid/query/Query.java| 7 +
.../java/org/apache/druid/query/QueryContexts.java | 6 +
.../org/apache/druid/query/QueryException.java | 68 +--
.../druid/query/QueryInterruptedException.java | 48 +-
.../java/org/apache/druid/query/QueryWatcher.java | 2 +-
.../epinephelinae/GroupByMergingQueryRunnerV2.java | 2 +-
.../SegmentMetadataQueryRunnerFactory.java | 2 +-
.../query/ChainedExecutionQueryRunnerTest.java | 4 +-
...GroupByLimitPushDownInsufficientBufferTest.java | 2 +-
.../query/groupby/GroupByMultiSegmentTest.java | 2 +-
server/pom.xml | 4 +
.../druid/client/CachingClusteredClient.java | 94 ++--
.../org/apache/druid/client/DirectDruidClient.java | 2 +-
...ventoryView.java => SegmentServerSelector.java} | 33 +-
.../druid/guice/QueryRunnerFactoryModule.java | 27 +-
.../apache/druid/server/BrokerQueryResource.java | 4 +-
.../server/QueryCapacityExceededException.java | 60 ++
.../apache/druid/server/QueryLaningStrategy.java | 53 ++
.../java/org/apache/druid/server/QueryManager.java | 86 ---
.../org/apache/druid/server/QueryResource.java | 24 +-
.../org/apache/druid/server/QueryScheduler.java| 258 +
.../apache/druid/server/QuerySchedulerConfig.java | 24 +-
.../QuerySchedulerProvider.java} | 25 +-
.../server/scheduling/HiLoQueryLaningStrategy.java | 79 +++
.../server/scheduling/NoQueryLaningStrategy.java | 52 ++
.../CachingClusteredClientFunctionalityTest.java | 6 +-
.../druid/client/CachingClusteredClientTest.java | 6 +-
.../org/apache/druid/server/QueryResourceTest.java | 290 --
.../apache/druid/server/QuerySchedulerTest.java| 610 +
.../scheduling/HiLoQueryLaningStrategyTest.java| 180 ++
.../scheduling/NoQueryLaningStrategyTest.java | 72 +++
.../org/apache/druid/sql/http/SqlResource.java | 5 +
.../druid/sql/calcite/util/CalciteTests.java | 13 +-
.../util/SpecificSegmentsQuerySegmentWalker.java | 54 +-
.../org/apache/druid/sql/http/SqlResourceTest.java | 139 -
website/.spelling | 1 +
47 files changed, 2076 insertions(+), 345 deletions(-)
copy server/src/main/java/org/apache/druid/client/indexing/IndexingWorker.java
=> processing/src/main/java/org/apache/druid/query/QueryException.java (50%)
copy
server/src/main/java/org/apache/druid/client/{FilteredServerInventoryView.java
=> SegmentServerSelector.java} (57%)
create mode 100644
server/src/main/java/org/apache/druid/server/QueryCapacityExceededException.java
create mode 100644
server/src/main/java/org/apache/druid/server/QueryLaningStrategy.java
delete mode 100644
server/src/main/java/org/apache/druid/server/QueryManager.java
create mode 100644
server/src/main/java/org/apache/druid/server/QueryScheduler.java
copy
indexing-service/src/main/java/org/apache/druid/indexing/common/config/FileTaskLogsConfig.java
=> server/src/main/java/org/apache/druid/server/QuerySchedulerConfig.java (69%)
copy
server/src/main/java/org/apache/druid/{client/selector/LowestPriorityTierSelectorStrategy.java
=> server/QuerySchedulerProvider.java} (56%)
create mode 100644
server/src/main/java/org/apache/druid/server/scheduling/HiLoQueryLaningStrategy.java
create mode 100644
server/src/main/java/org/apache/druid/server/scheduling/NoQueryLaningStrategy.java
create mode 100644
server/src/test/java/org/apache/druid/server/QuerySchedulerTest.java
create mode 100644
server/src/test/java/org/apache/druid/server/scheduling/HiLoQueryLaningStrategyTest.java
create mode 100644
server/src/test/java/org/apache/druid/server/scheduling/NoQueryLaningStrategyTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For addition
[GitHub] [druid] clintropolis merged pull request #9407: query laning and load shedding
clintropolis merged pull request #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #9407: query laning and load shedding
clintropolis commented on issue #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407#issuecomment-596999295 Thanks for taking a look @himanshug! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9407: query laning and load shedding
clintropolis commented on a change in pull request #9407: query laning and load
shedding
URL: https://github.com/apache/druid/pull/9407#discussion_r390201171
##
File path:
server/src/main/java/org/apache/druid/server/QueryLaningStrategy.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.fasterxml.jackson.annotation.JsonTypeInfo;
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import org.apache.druid.client.SegmentServerSelector;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.server.scheduling.HiLoQueryLaningStrategy;
+import org.apache.druid.server.scheduling.NoQueryLaningStrategy;
+
+import java.util.Optional;
+import java.util.Set;
+
+
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "strategy", defaultImpl =
NoQueryLaningStrategy.class)
+@JsonSubTypes(value = {
+@JsonSubTypes.Type(name = "none", value = NoQueryLaningStrategy.class),
+@JsonSubTypes.Type(name = "hilo", value = HiLoQueryLaningStrategy.class)
+})
+public interface QueryLaningStrategy
+{
+ /**
+ * Provide a map of lane names to the limit on the number of concurrent
queries for that lane
+ * @param totalLimit
+ */
+ Object2IntMap getLaneLimits(int totalLimit);
Review comment:
>is it expected that sum(returned_map.values) < totalLimit ?
Right now I am not requiring this needs to be true since the limits are not
guaranteed capacity, but rather maximums. It seemed more flexible to leave it
to individual `QueryLaningStrategy` implementations to enforce that if they
wish.
>nit: Also wonder why limits returned here were not percentages but absolute
counts given that each implementation would probably end up doing that
calculation.
Hmm, I think that is definitely worth considering, though @jihoonson was
asking for absolute limits in this comment
https://github.com/apache/druid/pull/9407#discussion_r389173425, so in the very
least I will add a utility method to the `QueryLaningStrategy` interface for
doing this conversion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9407: query laning and load shedding
clintropolis commented on a change in pull request #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407#discussion_r390198573 ## File path: docs/configuration/index.md ## @@ -1481,9 +1481,35 @@ These Broker configurations can be defined in the `broker/runtime.properties` fi |`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`| |`druid.broker.select.tier.custom.priorities`|`An array of integer priorities.`|Select servers in tiers with a custom priority list.|None| +# Query laning + +*Laning strategies* allow you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane. Requests in excess of the capacity are discarded with an HTTP 429 status code. + +|Property|Description|Default| +||---|---| +|`druid.query.scheduler.numThreads`|Maximum number of HTTP threads to dedicate to query processing. To save HTTP thread capacity, this should be lower than `druid.server.http.numThreads`.|Unbounded| Review comment: >in what use case would I ever want to set it something other than druid.server.http.numThreads ? I actually think we might always want to set it lower than `druid.server.http.numThreads`, but I was too nervous to make this the default and made it opt in behavior instead (since it grabs and releases locks for each query if there is some bug in releasing locks a broker would eventually stop accepting queries entirely). The primary reason I think we want it lower than `druid.server.http.numThreads` is to save some 'slack' space for non-query http connections, like accepting health checks, lookup management, and other such things that can be starved when long running queries start to pile up. >If the recommendation is to set it lower than `druid.server.http.numThreads` then why the default value is not set to `druid.server.http.numThreads - 1` ? See my above nervousness, but I think `druid.server.http.numThreads - 1` would probably be a good default. This might want to be adjusted to be even lower depending on how much other non query http traffic the server is receiving (e.g. frequently polled/updated lookups, etc). >I guess, as a user I don't quite understand the importance of setting this higher/same/lower compared to druid.server.http.numThreads and when I should choose one vs the other. I would agree the current documentation doesn't quite adequately describe how this stuff might be utilized, in a future PR i want to add a section to [cluster tuning docs](https://github.com/apache/druid/blob/master/docs/operations/basic-cluster-tuning.md) to more properly advise on when and how to set this stuff up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9407: query laning and load shedding
clintropolis commented on a change in pull request #9407: query laning and load
shedding
URL: https://github.com/apache/druid/pull/9407#discussion_r390194216
##
File path:
server/src/main/java/org/apache/druid/server/QuerySchedulerConfig.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.server.scheduling.NoQueryLaningStrategy;
+
+public class QuerySchedulerConfig
+{
+ @JsonProperty
+ private Integer numThreads = 0;
Review comment:
Hmm, that is a good point, will change in one of the follow-up PRs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #8987: Adding support for autoscaling in GCE
clintropolis commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-596992428 >@clintropolis — let me know if you need some more input from me :) I don't as far as I know, I just haven't had a chance to get back to this yet. I will try my best to finish up soon 😅 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xuejiaqing closed issue #9466: org.apache.druid.indexing.overlord.TaskLockbox: Cannot create a new taskLockPosse for request
xuejiaqing closed issue #9466: org.apache.druid.indexing.overlord.TaskLockbox: Cannot create a new taskLockPosse for request URL: https://github.com/apache/druid/issues/9466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xuejiaqing commented on issue #9466: org.apache.druid.indexing.overlord.TaskLockbox: Cannot create a new taskLockPosse for request
xuejiaqing commented on issue #9466: org.apache.druid.indexing.overlord.TaskLockbox: Cannot create a new taskLockPosse for request URL: https://github.com/apache/druid/issues/9466#issuecomment-596977095 sorry, my fault. I hava a task is not finished. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on issue #8987: Adding support for autoscaling in GCE
frnidito commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-596953711 @clintropolis — let me know if you need some more input from me :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing
stale[bot] commented on issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing URL: https://github.com/apache/druid/issues/6208#issuecomment-596944533 This issue is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson opened a new issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing
jihoonson opened a new issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing URL: https://github.com/apache/druid/issues/6208 Checkpoint was introduced for incremental publishing in Kafka indexing service. Each kafka task can publish their segments if the segment size is large enough in the middle of indexing. Checkpointing is a very important feature and should be tested in various situations. The main purpose of this proposal is for easy testing of rolling update while checkpointing. If we have an API for manual checkpointing, it would be much easier to test it by simply calling the checkpoint API and do the rolling update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9407: query laning and load shedding
himanshug commented on a change in pull request #9407: query laning and load
shedding
URL: https://github.com/apache/druid/pull/9407#discussion_r390132201
##
File path:
server/src/main/java/org/apache/druid/server/QueryLaningStrategy.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.fasterxml.jackson.annotation.JsonTypeInfo;
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import org.apache.druid.client.SegmentServerSelector;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.server.scheduling.HiLoQueryLaningStrategy;
+import org.apache.druid.server.scheduling.NoQueryLaningStrategy;
+
+import java.util.Optional;
+import java.util.Set;
+
+
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "strategy", defaultImpl =
NoQueryLaningStrategy.class)
+@JsonSubTypes(value = {
+@JsonSubTypes.Type(name = "none", value = NoQueryLaningStrategy.class),
+@JsonSubTypes.Type(name = "hilo", value = HiLoQueryLaningStrategy.class)
+})
+public interface QueryLaningStrategy
+{
+ /**
+ * Provide a map of lane names to the limit on the number of concurrent
queries for that lane
+ * @param totalLimit
+ */
+ Object2IntMap getLaneLimits(int totalLimit);
Review comment:
is it expected that `sum(returned_map.values) < totalLimit` ?
nit: Also wonder why limits returned here were not percentages but
absolute counts given that each implementation would probably end up doing that
calculation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9407: query laning and load shedding
himanshug commented on a change in pull request #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407#discussion_r390124045 ## File path: docs/configuration/index.md ## @@ -1481,9 +1481,35 @@ These Broker configurations can be defined in the `broker/runtime.properties` fi |`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`| |`druid.broker.select.tier.custom.priorities`|`An array of integer priorities.`|Select servers in tiers with a custom priority list.|None| +# Query laning + +*Laning strategies* allow you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane. Requests in excess of the capacity are discarded with an HTTP 429 status code. + +|Property|Description|Default| +||---|---| +|`druid.query.scheduler.numThreads`|Maximum number of HTTP threads to dedicate to query processing. To save HTTP thread capacity, this should be lower than `druid.server.http.numThreads`.|Unbounded| Review comment: in what use case would I ever want to set it something other than `druid.server.http.numThreads` ? If the recommendation is to set it lower than `druid.server.http.numThreads` then why the default value is not set to `druid.server.http.numThreads - 1` ? I guess, as a user I don't quite understand the importance of setting this higher/same/lower compared to `druid.server.http.numThreads` and when I should choose one vs the other. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9407: query laning and load shedding
himanshug commented on a change in pull request #9407: query laning and load
shedding
URL: https://github.com/apache/druid/pull/9407#discussion_r390098011
##
File path:
server/src/main/java/org/apache/druid/server/QuerySchedulerConfig.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.server.scheduling.NoQueryLaningStrategy;
+
+public class QuerySchedulerConfig
+{
+ @JsonProperty
+ private Integer numThreads = 0;
Review comment:
why is this Integer and not int considering it has a default non-null value?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6625: [Proposal] Allow ingesting timestamp from Kafka's Metadata envelope.
stale[bot] commented on issue #6625: [Proposal] Allow ingesting timestamp from Kafka's Metadata envelope. URL: https://github.com/apache/druid/issues/6625#issuecomment-596938371 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing
stale[bot] commented on issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing URL: https://github.com/apache/druid/issues/6208#issuecomment-596938383 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #5727: Lookup module not registered with overlord
stale[bot] commented on issue #5727: Lookup module not registered with overlord URL: https://github.com/apache/druid/issues/5727#issuecomment-596938377 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #6542: KIS - wrong values in multi-value dimension
stale[bot] closed issue #6542: KIS - wrong values in multi-value dimension URL: https://github.com/apache/druid/issues/6542 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6542: KIS - wrong values in multi-value dimension
stale[bot] commented on issue #6542: KIS - wrong values in multi-value dimension URL: https://github.com/apache/druid/issues/6542#issuecomment-596938369 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #5727: Lookup module not registered with overlord
stale[bot] closed issue #5727: Lookup module not registered with overlord URL: https://github.com/apache/druid/issues/5727 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #6625: [Proposal] Allow ingesting timestamp from Kafka's Metadata envelope.
stale[bot] closed issue #6625: [Proposal] Allow ingesting timestamp from Kafka's Metadata envelope. URL: https://github.com/apache/druid/issues/6625 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing
stale[bot] closed issue #6208: [Proposal] Add 'checkpoint' API to supervisor for manual checkpointing URL: https://github.com/apache/druid/issues/6208 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
