[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888120
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
Review comment:
This line, and a few other places, say "MiddleManager" when they should say
"Indexer".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888497
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
+|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can
accept.|Number of available processors - 1|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
+|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
+|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
+|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
+|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager
restart for restorable tasks to gracefully exit.|PT5M|
+|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
+|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will
attempt to stop tasks gracefully on shutdown and restore them on restart.|false|
+|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
+|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887454 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. Review comment: "and better enable resource sharing between tasks"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8245: Add docs for CliIndexer as an experimental feature
gianm commented on issue #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#issuecomment-518511216 I am so amped about the Indexer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310885372
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
+|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can
accept.|Number of available processors - 1|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
+|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
+|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
+|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
+|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager
restart for restorable tasks to gracefully exit.|PT5M|
+|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
+|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will
attempt to stop tasks gracefully on shutdown and restore them on restart.|false|
+|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
+|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888041
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
Review comment:
How about adding these to `configuration/index.md`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887504 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: Review comment: `.` would work better than `:` since the following sections are subheads (not a list). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310885038
##
File path:
processing/src/main/java/org/apache/druid/segment/filter/Filters.java
##
@@ -651,4 +652,17 @@ private static void generateAllCombinations(
generateAllCombinations(result, andList.subList(1, andList.size()),
nonAndList);
}
}
+
+ public static boolean shouldUseIndex(Filter filter, BitmapIndexSelector
indexSelector, FilterTuning filterTuning)
Review comment:
Could use a bit of javadocs as to its purpose ("standard" implementation of
Filter.shouldUseIndex) and how it works (returns true if the filter supports
indexes and if all underlying columns match the thresholds in `filterTuning`).
Also that `filterTuning` is nullable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310864811
##
File path: processing/src/main/java/org/apache/druid/query/filter/Filter.java
##
@@ -129,4 +130,19 @@ default boolean canVectorizeMatcher()
{
return false;
}
+
+ /**
+ * Set of columns used by a filter
+ */
+ Set getRequiredColumns();
+
+ /**
+ * Determine if a filter *should* use a bitmap index based on information
collected from the supplied
+ * {@link BitmapIndexSelector}. This method differs from {@link
#supportsBitmapIndex(BitmapIndexSelector)} in that
+ * the former only indicates if a bitmap index is available and {@link
#getBitmapIndex(BitmapIndexSelector)} may be
+ * used. This method, by default, will consider a {@link FilterTuning} to
make decisions about when to use an
+ * available index. Override this method in a {@link Filter} implementation
when {@link FilterTuning} alone is not
+ * adequate for making this decision.
+ */
Review comment:
I'd reword slightly. This method doesn't do anything by default anymore (it
did in an earlier version). Maybe cut the last two sentences, and replace with:
> Implementations of this methods typically consider a {@link FilterTuning}
to make decisions about when to use an available index.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310861548
##
File path:
extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java
##
@@ -168,20 +183,40 @@ public BloomKFilter getBloomKFilter()
return bloomKFilter;
}
+ @Nullable
@JsonProperty
public ExtractionFn getExtractionFn()
{
return extractionFn;
}
+ @Nullable
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ @JsonProperty
+ public FilterTuning getFilterTuning()
+ {
+return filterTuning;
+ }
+
+ @Override
+ public RangeSet getDimensionRangeSet(String dimension)
+ {
+return null;
+ }
+
+ @Override
+ public Set getRequiredColumns()
+ {
+return ImmutableSet.of(dimension);
+ }
+
@Override
public String toString()
{
-if (extractionFn != null) {
- return StringUtils.format("%s(%s) = %s", extractionFn, dimension,
hash.toString());
-} else {
- return StringUtils.format("%s = %s", dimension, hash.toString());
-}
+return new DimFilterToStringBuilder().appendDimension(dimension,
extractionFn)
Review comment:
Whoa. This is cool.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8232: Web-Console: add dimensions to datasources icon
vogievetsky commented on issue #8232: Web-Console: add dimensions to datasources icon URL: https://github.com/apache/incubator-druid/pull/8232#issuecomment-518491012 Looks good. Thank you for fixing. g2g once travis passes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310870445 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java ## @@ -146,10 +146,13 @@ private final RetryPolicyFactory retryPolicyFactory; @JsonIgnore - private List indexTaskSpecs; + private final AppenderatorsManager appenderatorsManager; @JsonIgnore - private AppenderatorsManager appenderatorsManager; + private List indexTaskSpecs; + + @Nullable + private volatile IndexTask currentRunningTaskSpec = null; Review comment: Added javadoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner;
fix a couple of concurrency bugs in batch tasks
URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310870428
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java
##
@@ -935,7 +900,7 @@ private TaskStatus generateAndPublishSegments(
final BatchAppenderatorDriver driver = newDriver(appenderator,
toolbox, segmentAllocator);
final Firehose firehose =
firehoseFactory.connect(dataSchema.getParser(), firehoseTempDir)
) {
- this.appenderator = appenderator;
+ registerResourceCloserOnAbnormalExit(config -> appenderator.closeNow());
Review comment:
Took `Closeable` out of `Appenderator`. Callers of `Appenderator` should
explicitly call `close()` or `closeNow()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 0235b33 Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231) 0235b33 is described below commit 0235b338fc82cc5475011d7928006a493b8966b9 Author: Vadim Ogievetsky AuthorDate: Mon Aug 5 18:54:39 2019 -0700 Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231) * teach table about resultAsArray * use query result decoder * fix snapshot --- licenses.yaml | 349 + licenses/bin/commander.MIT | 22 ++ licenses/bin/d3-axis.BSD3 | 27 ++ licenses/bin/d3-brush.BSD3 | 27 ++ licenses/bin/d3-chord.BSD3 | 27 ++ licenses/bin/d3-collection.BSD3| 27 ++ licenses/bin/d3-color.BSD3 | 27 ++ licenses/bin/d3-contour.BSD3 | 27 ++ licenses/bin/d3-dispatch.BSD3 | 27 ++ licenses/bin/d3-drag.BSD3 | 27 ++ licenses/bin/d3-dsv.BSD3 | 27 ++ licenses/bin/d3-ease.BSD3 | 28 ++ licenses/bin/d3-fetch.BSD3 | 27 ++ licenses/bin/d3-force.BSD3 | 27 ++ licenses/bin/d3-format.BSD3| 27 ++ licenses/bin/d3-geo.BSD3 | 48 +++ licenses/bin/d3-hierarchy.BSD3 | 27 ++ licenses/bin/d3-interpolate.BSD3 | 27 ++ licenses/bin/d3-path.BSD3 | 27 ++ licenses/bin/d3-polygon.BSD3 | 27 ++ licenses/bin/d3-quadtree.BSD3 | 27 ++ licenses/bin/d3-random.BSD3| 27 ++ licenses/bin/d3-scale-chromatic.BSD3 | 44 +++ licenses/bin/d3-scale.BSD3 | 27 ++ licenses/bin/d3-selection.BSD3 | 26 ++ licenses/bin/d3-shape.BSD3 | 27 ++ licenses/bin/d3-time-format.BSD3 | 27 ++ licenses/bin/d3-time.BSD3 | 27 ++ licenses/bin/d3-timer.BSD3 | 27 ++ licenses/bin/d3-transition.BSD3| 58 licenses/bin/d3-voronoi.BSD3 | 50 +++ licenses/bin/d3-zoom.BSD3 | 27 ++ licenses/bin/d3.BSD3 | 27 ++ licenses/bin/rw.BSD3 | 26 ++ web-console/package-lock.json | 8 + web-console/package.json | 1 + web-console/src/setup-tests.ts | 1 + web-console/src/utils/general.tsx | 15 +- web-console/src/utils/index.tsx| 1 - web-console/src/utils/rune-decoder.tsx | 141 - .../__snapshots__/query-view.spec.tsx.snap | 2 +- .../views/query-view/query-output/query-output.tsx | 2 +- web-console/src/views/query-view/query-view.tsx| 82 ++--- .../query-view/run-button/run-button.spec.tsx | 2 +- .../src/views/query-view/run-button/run-button.tsx | 18 +- web-console/src/views/task-view/tasks-view.tsx | 6 +- 46 files changed, 1392 insertions(+), 213 deletions(-) diff --git a/licenses.yaml b/licenses.yaml index eb1d4a4..f6f8337 100644 --- a/licenses.yaml +++ b/licenses.yaml @@ -2378,6 +2378,16 @@ license_file_path: licenses/bin/classnames.MIT --- +name: "commander" +license_category: binary +module: web-console +license_name: MIT License +copyright: TJ Holowaychuk +version: 2.20.0 +license_file_path: licenses/bin/commander.MIT + +--- + name: "copy-to-clipboard" license_category: binary module: web-console @@ -2433,11 +2443,331 @@ license_category: binary module: web-console license_name: BSD-3-Clause License copyright: Mike Bostock +version: 1.2.4 +license_file_path: licenses/bin/d3-array.BSD3 + +--- + +name: "d3-array" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock version: 2.2.0 license_file_path: licenses/bin/d3-array.BSD3 --- +name: "d3-axis" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock +version: 1.0.12 +license_file_path: licenses/bin/d3-axis.BSD3 + +--- + +name: "d3-brush" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock +version: 1.0.6 +license_file_path: licenses/bin/d3-brush.BSD3 + +--- + +name: "d3-chord" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock
[GitHub] [incubator-druid] clintropolis merged pull request #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console
clintropolis merged pull request #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console URL: https://github.com/apache/incubator-druid/pull/8231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei opened a new pull request #8245: Add docs for CliIndexer as an experimental feature
jon-wei opened a new pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245 Adds a doc page for the new CliIndexer process (see #7900) as an experimental feature. This PR also adds a configuration option for the number of concurrent merge/persists allowed on the Indexer, `druid.worker.numConcurrentMerges` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
gianm commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242#issuecomment-518450278 I’m a bit nervous about removing something so widely used. I wonder if we can introduce a compat shim that translates these kinds of filters? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8244: Update docker build
himanshug commented on issue #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244#issuecomment-518438602 > @himanshug I am really sorry for opening this PR again. My forked master branch was messed up. Its fine, everyone messes up including me. +1 for proactively fixing the issues. +1 after build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to
control filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310830104
##
File path:
processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java
##
@@ -263,43 +290,30 @@ public boolean equals(Object o)
if (o == null || getClass() != o.getClass()) {
return false;
}
-
BoundDimFilter that = (BoundDimFilter) o;
-
-if (isLowerStrict() != that.isLowerStrict()) {
- return false;
-}
-if (isUpperStrict() != that.isUpperStrict()) {
- return false;
-}
-if (!getDimension().equals(that.getDimension())) {
- return false;
-}
-if (getUpper() != null ? !getUpper().equals(that.getUpper()) :
that.getUpper() != null) {
- return false;
-}
-if (getLower() != null ? !getLower().equals(that.getLower()) :
that.getLower() != null) {
- return false;
-}
-if (getExtractionFn() != null
-? !getExtractionFn().equals(that.getExtractionFn())
-: that.getExtractionFn() != null) {
- return false;
-}
-return getOrdering().equals(that.getOrdering());
+return lowerStrict == that.lowerStrict &&
+ upperStrict == that.upperStrict &&
+ dimension.equals(that.dimension) &&
+ Objects.equals(upper, that.upper) &&
+ Objects.equals(lower, that.lower) &&
+ Objects.equals(extractionFn, that.extractionFn) &&
+ Objects.equals(ordering, that.ordering) &&
+ Objects.equals(filterTuning, that.filterTuning);
}
@Override
public int hashCode()
{
-int result = getDimension().hashCode();
-result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0);
-result = 31 * result + (getLower() != null ? getLower().hashCode() : 0);
-result = 31 * result + (isLowerStrict() ? 1 : 0);
-result = 31 * result + (isUpperStrict() ? 1 : 0);
-result = 31 * result + (getExtractionFn() != null ?
getExtractionFn().hashCode() : 0);
-result = 31 * result + getOrdering().hashCode();
-return result;
+return Objects.hash(
+dimension,
+upper,
+lower,
+lowerStrict,
+upperStrict,
+extractionFn,
+ordering,
+filterTuning
+);
}
@Override
Review comment:
Refactored custom `DimFilter.toString` implementations, extracting common
string builder stuff into a `DimFilterToStringBuilder` to clean this up a bit
and help keep these custom `toString` implementations consistent with each
other at least, and added `filterTuning` to them when it is not null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console
vogievetsky commented on issue #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console URL: https://github.com/apache/incubator-druid/pull/8231#issuecomment-518435704 Ok updated to address @fjy 's comments. Also updated the licenses that went out of date after the segment timeline PR was merged in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310812077 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -42,12 +45,14 @@ private final BloomKFilter bloomKFilter; private final HashCode hash; private final ExtractionFn extractionFn; + private final FilterTuning filterTuning; Review comment: I think I've updated all `DimFilter` implementations to have the appropriate things annotated with `@Nullable` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310811872 ## File path: processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java ## @@ -114,10 +141,4 @@ public boolean equals(Object o) .collect(Collectors.toSet()) Review comment: I've changed the `DimFilter` interface to use `Set` instead of `HashSet` and adjusted this to no longer dupe the set, along with changing single dimension filters to use `ImmutableSet.of` instead of `Sets.newHashSet` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to
control filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310807921
##
File path:
processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java
##
@@ -263,43 +290,30 @@ public boolean equals(Object o)
if (o == null || getClass() != o.getClass()) {
return false;
}
-
BoundDimFilter that = (BoundDimFilter) o;
-
-if (isLowerStrict() != that.isLowerStrict()) {
- return false;
-}
-if (isUpperStrict() != that.isUpperStrict()) {
- return false;
-}
-if (!getDimension().equals(that.getDimension())) {
- return false;
-}
-if (getUpper() != null ? !getUpper().equals(that.getUpper()) :
that.getUpper() != null) {
- return false;
-}
-if (getLower() != null ? !getLower().equals(that.getLower()) :
that.getLower() != null) {
- return false;
-}
-if (getExtractionFn() != null
-? !getExtractionFn().equals(that.getExtractionFn())
-: that.getExtractionFn() != null) {
- return false;
-}
-return getOrdering().equals(that.getOrdering());
+return lowerStrict == that.lowerStrict &&
+ upperStrict == that.upperStrict &&
+ dimension.equals(that.dimension) &&
+ Objects.equals(upper, that.upper) &&
+ Objects.equals(lower, that.lower) &&
+ Objects.equals(extractionFn, that.extractionFn) &&
+ Objects.equals(ordering, that.ordering) &&
+ Objects.equals(filterTuning, that.filterTuning);
}
@Override
public int hashCode()
{
-int result = getDimension().hashCode();
-result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0);
-result = 31 * result + (getLower() != null ? getLower().hashCode() : 0);
-result = 31 * result + (isLowerStrict() ? 1 : 0);
-result = 31 * result + (isUpperStrict() ? 1 : 0);
-result = 31 * result + (getExtractionFn() != null ?
getExtractionFn().hashCode() : 0);
-result = 31 * result + getOrdering().hashCode();
-return result;
+return Objects.hash(
+dimension,
+upper,
+lower,
+lowerStrict,
+upperStrict,
+extractionFn,
+ordering,
+filterTuning
+);
}
@Override
Review comment:
I'm not exactly sure the best way to add `filterTuning` to the `toString`
output of these `DimFilter` with non-standard implementations, I guess a
parenthesis maybe? e.g. `(filterTuning=...`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7648: Handle exceptions while fetching offsets
stale[bot] commented on issue #7648: Handle exceptions while fetching offsets URL: https://github.com/apache/incubator-druid/pull/7648#issuecomment-518413793 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310802823
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
// Exception caught while setting up the iterator; release
resources.
- Lists.reverse(resources).forEach(CloseQuietly::close);
+ try {
+resources.close();
+ }
+ catch (IOException ex) {
Review comment:
Ah like that, yes that's perfectly possible
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm merged pull request #8227: Web-Console: change go to sql button to more button
gianm merged pull request #8227: Web-Console: change go to sql button to more button URL: https://github.com/apache/incubator-druid/pull/8227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web-Console: change go to sql button to more button (#8227)
This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 5d0805d Web-Console: change go to sql button to more button (#8227)
5d0805d is described below
commit 5d0805dd48890b2a22148ab009d9bdfbf92416d1
Author: mcbrewster <37322608+mcbrews...@users.noreply.github.com>
AuthorDate: Mon Aug 5 14:12:16 2019 -0700
Web-Console: change go to sql button to more button (#8227)
* change go to sql button
* rename to See in SQL view
* update snapshots
---
.../__snapshots__/datasource-view.spec.tsx.snap| 42 +++--
.../src/views/datasource-view/datasource-view.tsx | 45 ++
.../__snapshots__/segments-view.spec.tsx.snap | 43 ++---
.../src/views/segments-view/segments-view.tsx | 55 --
.../__snapshots__/servers-view.spec.tsx.snap | 42 +++--
.../src/views/servers-view/servers-view.tsx| 44 +
.../__snapshots__/tasks-view.spec.tsx.snap | 42 +++--
web-console/src/views/task-view/tasks-view.tsx | 34 +
8 files changed, 286 insertions(+), 61 deletions(-)
diff --git
a/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap
b/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap
index 9162516..0e4bc80 100755
---
a/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap
+++
b/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap
@@ -11,11 +11,43 @@ exports[`data source view matches snapshot 1`] = `
localStorageKey="datasources-refresh-rate"
onRefresh={[Function]}
/>
-
+
+
+
+ }
+ defaultIsOpen={false}
+ disabled={false}
+ fill={false}
+ hasBackdrop={false}
+ hoverCloseDelay={300}
+ hoverOpenDelay={150}
+ inheritDarkTheme={true}
+ interactionKind="click"
+ minimal={false}
+ modifiers={Object {}}
+ openOnTargetFocus={true}
+ position="bottom-left"
+ targetTagName="span"
+ transitionDuration={300}
+ usePortal={true}
+ wrapperTagName="span"
+>
+
+
+{!noSqlMode && (
+ goToQuery(DatasourcesView.DATASOURCE_SQL)}
+ />
+)}
+
+);
+
+return (
+ <>
+
+
+
+
+);
+ }
+
private saveRules = async (datasource: string, rules: any[], comment:
string) => {
try {
await axios.post(`/druid/coordinator/v1/rules/${datasource}`, rules, {
@@ -901,7 +934,7 @@ GROUP BY 1`;
}
render(): JSX.Element {
-const { goToQuery, noSqlMode } = this.props;
+const { noSqlMode } = this.props;
const { showDisabled, hiddenColumns, showChart, chartHeight, chartWidth }
= this.state;
return (
@@ -913,13 +946,7 @@ GROUP BY 1`;
}}
localStorageKey={LocalStorageKeys.DATASOURCES_REFRESH_RATE}
/>
- {!noSqlMode && (
- goToQuery(DatasourcesView.DATASOURCE_SQL)}
-/>
- )}
+ {this.renderBulkDatasourceActions()}
-
+
+
+
+}
+defaultIsOpen={false}
+disabled={false}
+fill={false}
+hasBackdrop={false}
+hoverCloseDelay={300}
+hoverOpenDelay={150}
+inheritDarkTheme={true}
+interactionKind="click"
+minimal={false}
+modifiers={Object {}}
+openOnTargetFocus={true}
+position="bottom-left"
+targetTagName="span"
+transitionDuration={300}
+usePortal={true}
+wrapperTagName="span"
+ >
+
+
Group by
diff --git a/web-console/src/views/segments-view/segments-view.tsx
b/web-console/src/views/segments-view/segments-view.tsx
index 18583ff..6e10b68 100644
--- a/web-console/src/views/segments-view/segments-view.tsx
+++ b/web-console/src/views/segments-view/segments-view.tsx
@@ -16,7 +16,16 @@
* limitations under the License.
*/
-import { Button, ButtonGroup, Intent, Label } from '@blueprintjs/core';
+import {
+ Button,
+ ButtonGroup,
+ Intent,
+ Label,
+ Menu,
+ MenuItem,
+ Popover,
+ Position,
+} from '@blueprintjs/core';
import { IconNames } from '@blueprintjs/icons';
import axios from 'axios';
import React from 'react';
@@ -607,6 +616,35 @@ export class SegmentsView extends
React.PureComponent
+{!noSqlMode && (
+ {
+ if (!lastSegmentsQuery) return;
+ goToQuery(lastSegmentsQuery);
+}}
+ />
+)}
+
+);
+
+return (
+ <>
+
+
+
+
+);
+ }
+
render(): JSX.Element {
const {
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310793741
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
// Exception caught while setting up the iterator; release
resources.
- Lists.reverse(resources).forEach(CloseQuietly::close);
+ try {
+resources.close();
+ }
+ catch (IOException ex) {
Review comment:
Why it can't be weaker? You can catch a weaker Exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8227: Web-Console: change go to sql button to more button
vogievetsky commented on issue #8227: Web-Console: change go to sql button to more button URL: https://github.com/apache/incubator-druid/pull/8227#issuecomment-518401428 Thank you for addressing my feedback, this is good to merge IMO This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ilhanadiyaman commented on issue #8244: Update docker build
ilhanadiyaman commented on issue #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244#issuecomment-518397727 @himanshug I am really sorry for opening this PR again. My forked master branch was messed up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ilhanadiyaman opened a new pull request #8244: Update docker build
ilhanadiyaman opened a new pull request #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244 Sorry for opening again. My master branch was messed up. Fixes #8054 ### Description - [License generation scripts](https://github.com/apache/incubator-druid/blob/master/docs/_bin/generate-license.py) requires python3 and pip3. Python3 and pip3 installation added to docker build. - Docker entrypoint script changed in respond to the new directory structure #7590 - Dockerfile updated to allow dependency caching in multi-stage build This PR has: - [x] been self-reviewed. - [x] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #5872: add method getRequiredColumns for DimFilter
clintropolis commented on a change in pull request #5872: add method getRequiredColumns for DimFilter URL: https://github.com/apache/incubator-druid/pull/5872#discussion_r310786713 ## File path: processing/src/main/java/io/druid/query/filter/DimFilter.java ## @@ -75,4 +77,9 @@ * determine for this DimFilter. */ RangeSet getDimensionRangeSet(String dimension); + + /** + * @return a HashSet that represents all columns' name which the DimFilter required to do filter. + */ + HashSet getRequiredColumns(); Review comment: I'm making this change to use `Set` as part of #8209. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle
jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle URL: https://github.com/apache/incubator-druid/pull/8115#issuecomment-518394084 @himanshug @clintropolis thank you for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in
overshadowableManager and add unit tests
URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785662
##
File path:
core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java
##
@@ -189,89 +247,359 @@ private void
transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou
}
/**
- * Find all atomicUpdateGroups of the given state overshadowed by the given
rootPartitionRange and minorVersion.
+ * Find all atomicUpdateGroups of the given state overshadowed by the
minorVersion in the given rootPartitionRange.
* The atomicUpdateGroup of a higher minorVersion can have a wider
RootPartitionRange.
* To find all atomicUpdateGroups overshadowed by the given
rootPartitionRange and minorVersion,
* we first need to find the first key contained by the given
rootPartitionRange.
* Once we find such key, then we go through the entire map until we see an
atomicUpdateGroup of which
* rootRangePartition is not contained by the given rootPartitionRange.
+ *
+ * @param rangeOfAug the partition range to search for overshadowed groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have lower minor versions
+ * than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
+ */
+ @VisibleForTesting
+ List> findOvershadowedBy(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+true
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contained by the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && rangeOfAug.overlaps(current.getKey())) {
+ if (rangeOfAug.contains(current.getKey())) {
+// versionToGroup is sorted by minorVersion.
+// versionToGroup.headMap(minorVersion) below returns a map containing
all entries of lower minorVersions
+// than the given minorVersion.
+final Short2ObjectSortedMap> versionToGroup =
current.getValue();
+// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation,
especially size(), is not optimized.
+// Note that size() is indirectly called in ArrayList.addAll() when
ObjectSortedSet.toArray() is called.
+// See AbstractObjectCollection.toArray().
+// If you see performance degradation here, probably we need to
improve the below line.
+if (versionToGroup.firstShortKey() < minorVersion) {
+ found.addAll(versionToGroup.headMap(minorVersion).values());
+}
+ }
+ current = stateMap.higherEntry(current.getKey());
+}
+return found;
+ }
+
+ private List> findOvershadows(AtomicUpdateGroup aug,
State fromState)
+ {
+return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(),
fromState);
+ }
+
+ /**
+ * Find all atomicUpdateGroups which overshadow others of the given
minorVersion in the given rootPartitionRange.
+ * Similar to {@link #findOvershadowedBy}.
+ *
+ * Note that one atommicUpdateGroup can overshadow multiple other groups. If
you're finding overshadowing
+ * atomicUpdateGroups by calling this method in a loop, the results of this
method can contain duplicate groups.
+ *
+ * @param rangeOfAug the partition range to search for overshadowing
groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have higher minor
+ * versions than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
*/
- private List>> findOvershadowedBy(
+ @VisibleForTesting
+ List> findOvershadows(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+false
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contains the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && current.getKey().overlaps(rangeOfAug)) {
+ if (current.getKey().contains(rangeOfAug)) {
+// versionToGroup is sorted by
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in
overshadowableManager and add unit tests
URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785630
##
File path:
core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java
##
@@ -189,89 +247,359 @@ private void
transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou
}
/**
- * Find all atomicUpdateGroups of the given state overshadowed by the given
rootPartitionRange and minorVersion.
+ * Find all atomicUpdateGroups of the given state overshadowed by the
minorVersion in the given rootPartitionRange.
* The atomicUpdateGroup of a higher minorVersion can have a wider
RootPartitionRange.
* To find all atomicUpdateGroups overshadowed by the given
rootPartitionRange and minorVersion,
* we first need to find the first key contained by the given
rootPartitionRange.
* Once we find such key, then we go through the entire map until we see an
atomicUpdateGroup of which
* rootRangePartition is not contained by the given rootPartitionRange.
+ *
+ * @param rangeOfAug the partition range to search for overshadowed groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have lower minor versions
+ * than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
+ */
+ @VisibleForTesting
+ List> findOvershadowedBy(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+true
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contained by the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && rangeOfAug.overlaps(current.getKey())) {
+ if (rangeOfAug.contains(current.getKey())) {
+// versionToGroup is sorted by minorVersion.
+// versionToGroup.headMap(minorVersion) below returns a map containing
all entries of lower minorVersions
+// than the given minorVersion.
+final Short2ObjectSortedMap> versionToGroup =
current.getValue();
+// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation,
especially size(), is not optimized.
+// Note that size() is indirectly called in ArrayList.addAll() when
ObjectSortedSet.toArray() is called.
+// See AbstractObjectCollection.toArray().
+// If you see performance degradation here, probably we need to
improve the below line.
+if (versionToGroup.firstShortKey() < minorVersion) {
+ found.addAll(versionToGroup.headMap(minorVersion).values());
+}
+ }
+ current = stateMap.higherEntry(current.getKey());
+}
+return found;
+ }
+
+ private List> findOvershadows(AtomicUpdateGroup aug,
State fromState)
+ {
+return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(),
fromState);
+ }
+
+ /**
+ * Find all atomicUpdateGroups which overshadow others of the given
minorVersion in the given rootPartitionRange.
+ * Similar to {@link #findOvershadowedBy}.
+ *
+ * Note that one atommicUpdateGroup can overshadow multiple other groups. If
you're finding overshadowing
+ * atomicUpdateGroups by calling this method in a loop, the results of this
method can contain duplicate groups.
+ *
+ * @param rangeOfAug the partition range to search for overshadowing
groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have higher minor
+ * versions than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
*/
- private List>> findOvershadowedBy(
+ @VisibleForTesting
+ List> findOvershadows(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+false
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contains the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && current.getKey().overlaps(rangeOfAug)) {
+ if (current.getKey().contains(rangeOfAug)) {
+// versionToGroup is sorted by
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in
overshadowableManager and add unit tests
URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785647
##
File path:
core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java
##
@@ -189,89 +247,359 @@ private void
transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou
}
/**
- * Find all atomicUpdateGroups of the given state overshadowed by the given
rootPartitionRange and minorVersion.
+ * Find all atomicUpdateGroups of the given state overshadowed by the
minorVersion in the given rootPartitionRange.
* The atomicUpdateGroup of a higher minorVersion can have a wider
RootPartitionRange.
* To find all atomicUpdateGroups overshadowed by the given
rootPartitionRange and minorVersion,
* we first need to find the first key contained by the given
rootPartitionRange.
* Once we find such key, then we go through the entire map until we see an
atomicUpdateGroup of which
* rootRangePartition is not contained by the given rootPartitionRange.
+ *
+ * @param rangeOfAug the partition range to search for overshadowed groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have lower minor versions
+ * than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
+ */
+ @VisibleForTesting
+ List> findOvershadowedBy(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+true
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contained by the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && rangeOfAug.overlaps(current.getKey())) {
+ if (rangeOfAug.contains(current.getKey())) {
+// versionToGroup is sorted by minorVersion.
+// versionToGroup.headMap(minorVersion) below returns a map containing
all entries of lower minorVersions
+// than the given minorVersion.
+final Short2ObjectSortedMap> versionToGroup =
current.getValue();
+// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation,
especially size(), is not optimized.
+// Note that size() is indirectly called in ArrayList.addAll() when
ObjectSortedSet.toArray() is called.
+// See AbstractObjectCollection.toArray().
+// If you see performance degradation here, probably we need to
improve the below line.
+if (versionToGroup.firstShortKey() < minorVersion) {
+ found.addAll(versionToGroup.headMap(minorVersion).values());
+}
+ }
+ current = stateMap.higherEntry(current.getKey());
+}
+return found;
+ }
+
+ private List> findOvershadows(AtomicUpdateGroup aug,
State fromState)
+ {
+return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(),
fromState);
+ }
+
+ /**
+ * Find all atomicUpdateGroups which overshadow others of the given
minorVersion in the given rootPartitionRange.
+ * Similar to {@link #findOvershadowedBy}.
+ *
+ * Note that one atommicUpdateGroup can overshadow multiple other groups. If
you're finding overshadowing
+ * atomicUpdateGroups by calling this method in a loop, the results of this
method can contain duplicate groups.
+ *
+ * @param rangeOfAug the partition range to search for overshadowing
groups.
+ * @param minorVersion the minor version to check overshadow relation. The
found groups will have higher minor
+ * versions than this.
+ * @param fromStatethe state to search for overshadowed groups.
+ *
+ * @return a list of found atomicUpdateGroups. It could be empty if no
groups are found.
*/
- private List>> findOvershadowedBy(
+ @VisibleForTesting
+ List> findOvershadows(RootPartitionRange rangeOfAug,
short minorVersion, State fromState)
+ {
+final TreeMap>> stateMap = getStateMap(fromState);
+Entry>>
current = findLowestOverlappingEntry(
+rangeOfAug,
+stateMap,
+false
+);
+
+if (current == null) {
+ return Collections.emptyList();
+}
+
+// Going through the map to find all entries of the RootPartitionRange
contains the given rangeOfAug.
+// Note that RootPartitionRange of entries are always consecutive.
+final List> found = new ArrayList<>();
+while (current != null && current.getKey().overlaps(rangeOfAug)) {
+ if (current.getKey().contains(rangeOfAug)) {
+// versionToGroup is sorted by
[GitHub] [incubator-druid] himanshug closed issue #8054: Issue while using Docker build
himanshug closed issue #8054: Issue while using Docker build URL: https://github.com/apache/incubator-druid/issues/8054 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #8237: Update docker build
himanshug merged pull request #8237: Update docker build URL: https://github.com/apache/incubator-druid/pull/8237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add shuffleSegmentPusher for data shuffle (#8115)
This is an automated email from the ASF dual-hosted git repository.
himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new ab5b3be Add shuffleSegmentPusher for data shuffle (#8115)
ab5b3be is described below
commit ab5b3be6c61d7be540f85385327a6a6489b363b1
Author: Jihoon Son
AuthorDate: Mon Aug 5 13:38:35 2019 -0700
Add shuffleSegmentPusher for data shuffle (#8115)
* Fix race between canHandle() and addSegment() in StorageLocation
* add comment
* Add shuffleSegmentPusher which is a dataSegmentPusher used for writing
shuffle data in local storage.
* add comments
* unused import
* add comments
* fix test
* address comments
* remove tag from javadoc
* address comments
* comparingLong
* Address comments
* fix test
---
.../druid/indexing/common/config/TaskConfig.java | 5 +
.../indexing/worker/IntermediaryDataManager.java | 166 +
.../indexing/worker/ShuffleDataSegmentPusher.java | 77 ++
.../IntermediaryDataManagerAutoCleanupTest.java| 15 +-
...ermediaryDataManagerManualAddAndDeleteTest.java | 47 +++---
.../worker/ShuffleDataSegmentPusherTest.java | 138 +
.../druid/segment/loading/StorageLocation.java | 7 +-
...mentLoaderLocalCacheManagerConcurrencyTest.java | 2 -
.../druid/segment/loading/StorageLocationTest.java | 14 +-
9 files changed, 366 insertions(+), 105 deletions(-)
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
index 31405fe..52bf083 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
@@ -126,6 +126,11 @@ public class TaskConfig
return new File(getTaskDir(taskId), "work");
}
+ public File getTaskTempDir(String taskId)
+ {
+return new File(getTaskDir(taskId), "temp");
+ }
+
public File getTaskLockFile(String taskId)
{
return new File(getTaskDir(taskId), "lock");
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
index c47425a..95acb22 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
@@ -33,11 +33,13 @@ import org.apache.druid.java.util.common.DateTimes;
import org.apache.druid.java.util.common.IOE;
import org.apache.druid.java.util.common.ISE;
import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.io.Closer;
import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
import org.apache.druid.java.util.common.logger.Logger;
import org.apache.druid.segment.loading.StorageLocation;
import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.utils.CompressionUtils;
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.Period;
@@ -45,6 +47,7 @@ import org.joda.time.Period;
import javax.annotation.Nullable;
import java.io.File;
import java.io.IOException;
+import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Collections;
@@ -57,12 +60,14 @@ import java.util.Map.Entry;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
+import java.util.stream.IntStream;
/**
* This class manages intermediary segments for data shuffle between native
parallel index tasks.
- * In native parallel indexing, phase 1 tasks store segment files in local
storage of middleManagers
+ * In native parallel indexing, phase 1 tasks store segment files in local
storage of middleManagers (or indexer)
* and phase 2 tasks read those files via HTTP.
*
* The directory where segment files are placed is structured as
@@ -75,11 +80,12 @@ import java.util.stream.Collectors;
@ManageLifecycle
public class IntermediaryDataManager
{
- private static final Logger log = new Logger(IntermediaryDataManager.class);
+ private static final Logger LOG = new Logger(IntermediaryDataManager.class);
private final long intermediaryPartitionDiscoveryPeriodSec;
private final long intermediaryPartitionCleanupPeriodSec;
private final Period intermediaryPartitionTimeout;
+
[GitHub] [incubator-druid] himanshug merged pull request #8115: Add shuffleSegmentPusher for data shuffle
himanshug merged pull request #8115: Add shuffleSegmentPusher for data shuffle URL: https://github.com/apache/incubator-druid/pull/8115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
himanshug commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518386793 > If I now understand correctly the purpose of the current proposal is to handle the queries aggregation problem. Oak might be a solution also for this problem but this is something we haven't looked at yet. yes, the context here applies to every place aggregation happens and also let it not be dependent on something specific but allow extensions to eventually add optimal behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
clintropolis commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242#issuecomment-518386455 sgtm :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8109: fix merging of groupBy subtotal spec results
himanshug commented on issue #8109: fix merging of groupBy subtotal spec results URL: https://github.com/apache/incubator-druid/pull/8109#issuecomment-518385521 @gianm can you please review it again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new pull request #8243: make double sum/min/max agg work on string columns
himanshug opened a new pull request #8243: make double sum/min/max agg work on string columns URL: https://github.com/apache/incubator-druid/pull/8243 Partial work towards #8148 . There would be follow up PR to do same for other core aggregators once this is merged. ### Description This patch adds handling of single/multi value column handling by double sum/min/max aggregators to do a best effort parsing string as double. `StringColumnDoubleAggregatorWrapper` and `StringColumnDoubleBufferAggregatorWrapper` classes are introduced that can wrap existing double aggregators to handle string columns. Both of the classes are used by `SimpleDoubleAggregatorFactory` to be used when input column is known to be of String type. Currently it does not work if `extractionFn` instead of `fieldName` is provided. (Related #8242 ) This PR has: - [x] been self-reviewed. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [x] added unit tests or modified existing tests to cover new code paths. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7093: [PROPOSAL] Query vectorization
gianm commented on issue #7093: [PROPOSAL] Query vectorization URL: https://github.com/apache/incubator-druid/issues/7093#issuecomment-518383739 Closed by #6794. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm closed issue #7093: [PROPOSAL] Query vectorization
gianm closed issue #7093: [PROPOSAL] Query vectorization URL: https://github.com/apache/incubator-druid/issues/7093 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
himanshug opened a new issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242 they were added before `VirtualColumn` was introduced and I think a better alternative now could be to have a extractionFn based `VirtualColumn` that could be used. That way, all the filter and aggregator implementations don't need to handle them in different places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310750551
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
// Exception caught while setting up the iterator; release
resources.
- Lists.reverse(resources).forEach(CloseQuietly::close);
+ try {
+resources.close();
+ }
+ catch (IOException ex) {
Review comment:
This can't be weaker since it is being thrown by the `.close()` method:
https://github.com/apache/incubator-druid/blob/master/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java#L155
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310750268
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310749986
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
Review comment:
Makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310749285
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
+ * {@linkplain #close closed}.
+ */
+ // close. this word no longer has any meaning to me.
+ public Iterable register(Iterable closeables)
Review comment:
Good suggestion, done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310748843
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
+ * {@linkplain #close closed}.
+ */
+ // close. this word no longer has any meaning to me.
Review comment:
This is copy-paste. It is already in current master :-)
https://github.com/apache/incubator-druid/blob/master/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java#L116
I'll remove it anyway
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei merged pull request #8100: SQL support for t-digest based sketch aggregators
jon-wei merged pull request #8100: SQL support for t-digest based sketch aggregators URL: https://github.com/apache/incubator-druid/pull/8100 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (ac856fe -> 93cf9d4)
This is an automated email from the ASF dual-hosted git repository.
jonwei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.
from ac856fe Web-Console: Adds edit-context-dialog (#8228)
add 93cf9d4 SQL support for t-digest based sketch aggregators (#8100)
No new revisions were added by this update.
Summary of changes:
.../extensions-contrib/tdigestsketch-quantiles.md | 86 ++--
docs/content/querying/sql.md | 2 +
extensions-contrib/tdigestsketch/pom.xml | 84 +++-
.../TDigestMergeSketchAggregator.java | 83
.../TDigestMergeSketchAggregatorFactory.java | 64 ---
.../TDigestMergeSketchBufferAggregator.java| 110 -
...ggregator.java => TDigestSketchAggregator.java} | 12 +-
...ry.java => TDigestSketchAggregatorFactory.java} | 37 +-
...tor.java => TDigestSketchBufferAggregator.java} | 8 +-
.../TDigestSketchComplexMetricSerde.java | 2 +-
.../tdigestsketch/TDigestSketchModule.java | 22 +-
.../tdigestsketch/TDigestSketchObjectStrategy.java | 2 +-
... => TDigestSketchToQuantilePostAggregator.java} | 68 ++--
.../TDigestSketchToQuantilesPostAggregator.java| 6 +-
.../tdigestsketch/TDigestSketchUtils.java | 33 ++
.../sql/TDigestGenerateSketchSqlAggregator.java| 183 +
.../sql/TDigestSketchQuantileSqlAggregator.java| 112 ++
.../tdigestsketch/TDigestSketchAggregatorTest.java | 28 +-
.../sql/TDigestSketchSqlAggregatorTest.java| 447 +
.../druid/query/aggregation/AggregatorUtil.java| 2 -
.../query/aggregation/post/PostAggregatorIds.java | 2 +
21 files changed, 907 insertions(+), 486 deletions(-)
delete mode 100644
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchAggregator.java
delete mode 100644
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchAggregatorFactory.java
delete mode 100644
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchBufferAggregator.java
rename
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchAggregator.java
=> TDigestSketchAggregator.java} (85%)
rename
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchAggregatorFactory.java
=> TDigestSketchAggregatorFactory.java} (79%)
rename
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchBufferAggregator.java
=> TDigestSketchBufferAggregator.java} (93%)
copy
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestSketchToQuantilesPostAggregator.java
=> TDigestSketchToQuantilePostAggregator.java} (67%)
create mode 100644
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestGenerateSketchSqlAggregator.java
copy
extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/sql/DoublesSketchSqlAggregator.java
=>
extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchQuantileSqlAggregator.java
(61%)
create mode 100644
extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Eshcar commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
Eshcar commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518356531 Thanks, that was our impression - that off-heap incremental index is not operational (however it does exist in the code). So, indeed there is no way to compare to it. I also agree that doing the oak-sketches-druid integration in one step might be too complicated. We already have an open issue #5698 and a PR #7676 for getting Oak incremental index into Druid and we hope to get progress there soon. Oak is not based on Memory , yet :). The context of my suggestion is how to support growable sketches off-heap while ingesting data in druid - namely, in the context of Oak. So the order should be first integrating Oak, then having Oak support growable sketches. If I now understand correctly the purpose of the current proposal is to handle the queries aggregation problem. Oak might be a solution also for this problem but this is something we haven't looked at yet. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6541: compaction tasks run for much time
stale[bot] commented on issue #6541: compaction tasks run for much time URL: https://github.com/apache/incubator-druid/issues/6541#issuecomment-518353565 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
leventov commented on a change in pull request #7562: Enable toggling request
logging on/off for different query types
URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310741593
##
File path:
server/src/test/java/org/apache/druid/server/log/FilteredRequestLoggerTest.java
##
@@ -106,6 +123,9 @@ public void testNotFilterAboveThreshold() throws
IOException
EasyMock.expect(nativeRequestLogLine.getQueryStats())
.andReturn(new QueryStats(ImmutableMap.of("query/time", 1000)))
.once();
+EasyMock.expect(nativeRequestLogLine.getQuery())
Review comment:
No. Here, the alignment is on the dot. It's a coincidence that "EasyMock" is
8 characters long.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] capistrant commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
capistrant commented on a change in pull request #7562: Enable toggling request
logging on/off for different query types
URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310739354
##
File path:
server/src/test/java/org/apache/druid/server/log/FilteredRequestLoggerTest.java
##
@@ -106,6 +123,9 @@ public void testNotFilterAboveThreshold() throws
IOException
EasyMock.expect(nativeRequestLogLine.getQueryStats())
.andReturn(new QueryStats(ImmutableMap.of("query/time", 1000)))
.once();
+EasyMock.expect(nativeRequestLogLine.getQuery())
Review comment:
@leventov Does that wrapping indentation of 4 standard include lines such as
these 3? I see that the lines above and below are had been using 8.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
leventov commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310723222 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java ## @@ -146,10 +146,13 @@ private final RetryPolicyFactory retryPolicyFactory; @JsonIgnore - private List indexTaskSpecs; + private final AppenderatorsManager appenderatorsManager; @JsonIgnore - private AppenderatorsManager appenderatorsManager; + private List indexTaskSpecs; + + @Nullable + private volatile IndexTask currentRunningTaskSpec = null; Review comment: https://github.com/code-review-checklists/java-concurrency#justify-volatile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518327693 Currently, the off-heap incremental index is used only in GroupBy V1, which is itself deprecated and may not work at all. So I don't really even see a point in running benchmarks against the current off-heap incremental index to prove that Oak is better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518326114 > Then presenting a solution to this problem using Oak to manage the off-heap index, handling growable sketches using Memory, and showing that it performs as good as or better than the existing implementation would be a win-win-win solution, correct? Yes. I just want to warn you about trying to bring all of that at once - the extent of the change may become unbearable. So even if it incurs more work, it's better to make those changes separately: e. g. first, the transition to `Memory` (in the given subsystem), then - Oak. Doing it in the opposite order may be harder because Oak is already based on `Memory`, despite there is a method [`unsafeByteBufferView`](https://datasketches.github.io/api/memory/snapshot/apidocs/org/apache/datasketches/memory/Memory.html#unsafeByteBufferView-long-int-). > Any documentation on how we should configure the cluster to allow it running in off-heap mode? Not sure what do you mean by that. It's not possible to use the off-heap incremental index for indexing even when all aggregations are constant-sized - it's not implemented. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310691637 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -42,12 +45,14 @@ private final BloomKFilter bloomKFilter; private final HashCode hash; private final ExtractionFn extractionFn; + private final FilterTuning filterTuning; Review comment: Should be `@Nullable`, along with the corresponding constructor parameter This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310705663
##
File path: processing/src/main/java/org/apache/druid/query/Druids.java
##
@@ -361,7 +361,7 @@ public SearchQueryBuilder dataSource(DataSource d)
public SearchQueryBuilder filters(String dimensionName, String value)
{
- dimFilter = new SelectorDimFilter(dimensionName, value, null);
+ dimFilter = new SelectorDimFilter(dimensionName, value, null, null);
Review comment:
Could call the old constructor
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693015
##
File path:
extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java
##
@@ -193,16 +217,17 @@ public boolean equals(Object o)
if (o == null || getClass() != o.getClass()) {
return false;
}
-
BloomDimFilter that = (BloomDimFilter) o;
+return dimension.equals(that.dimension) &&
+ hash.equals(that.hash) &&
+ Objects.equals(extractionFn, that.extractionFn) &&
Review comment:
Everything about `extractionFn`: the field, constructor parameter, getter
should be `@Nullable`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310692390 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -174,6 +191,13 @@ public ExtractionFn getExtractionFn() return extractionFn; } + @JsonInclude(JsonInclude.Include.NON_NULL) + @JsonProperty + public FilterTuning getFilterTuning() Review comment: This method should be `@Nullable` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310707102
##
File path:
processing/src/main/java/org/apache/druid/query/filter/ExpressionDimFilter.java
##
@@ -39,23 +41,42 @@
{
private final String expression;
private final Supplier parsed;
+ private final FilterTuning filterTuning;
Review comment:
Please annotate with `@Nullable` everything that is needed in this class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310695880
##
File path:
processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java
##
@@ -263,43 +290,30 @@ public boolean equals(Object o)
if (o == null || getClass() != o.getClass()) {
return false;
}
-
BoundDimFilter that = (BoundDimFilter) o;
-
-if (isLowerStrict() != that.isLowerStrict()) {
- return false;
-}
-if (isUpperStrict() != that.isUpperStrict()) {
- return false;
-}
-if (!getDimension().equals(that.getDimension())) {
- return false;
-}
-if (getUpper() != null ? !getUpper().equals(that.getUpper()) :
that.getUpper() != null) {
- return false;
-}
-if (getLower() != null ? !getLower().equals(that.getLower()) :
that.getLower() != null) {
- return false;
-}
-if (getExtractionFn() != null
-? !getExtractionFn().equals(that.getExtractionFn())
-: that.getExtractionFn() != null) {
- return false;
-}
-return getOrdering().equals(that.getOrdering());
+return lowerStrict == that.lowerStrict &&
+ upperStrict == that.upperStrict &&
+ dimension.equals(that.dimension) &&
+ Objects.equals(upper, that.upper) &&
+ Objects.equals(lower, that.lower) &&
+ Objects.equals(extractionFn, that.extractionFn) &&
+ Objects.equals(ordering, that.ordering) &&
+ Objects.equals(filterTuning, that.filterTuning);
}
@Override
public int hashCode()
{
-int result = getDimension().hashCode();
-result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0);
-result = 31 * result + (getLower() != null ? getLower().hashCode() : 0);
-result = 31 * result + (isLowerStrict() ? 1 : 0);
-result = 31 * result + (isUpperStrict() ? 1 : 0);
-result = 31 * result + (getExtractionFn() != null ?
getExtractionFn().hashCode() : 0);
-result = 31 * result + getOrdering().hashCode();
-return result;
+return Objects.hash(
+dimension,
+upper,
+lower,
+lowerStrict,
+upperStrict,
+extractionFn,
+ordering,
+filterTuning
+);
}
@Override
Review comment:
Please add `filterTuning` in the `toString()` form.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310705538
##
File path: processing/src/main/java/org/apache/druid/query/Druids.java
##
@@ -204,7 +204,7 @@ public TimeseriesQueryBuilder filters(String
dimensionName, String value)
public TimeseriesQueryBuilder filters(String dimensionName, String value,
String... values)
{
- dimFilter = new InDimFilter(dimensionName, Lists.asList(value, values),
null);
+ dimFilter = new InDimFilter(dimensionName, Lists.asList(value, values),
null, null);
Review comment:
Could call the old constructor
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310706626 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -56,6 +61,17 @@ public BloomDimFilter( this.bloomKFilter = bloomKFilterHolder.getFilter(); this.hash = bloomKFilterHolder.getFilterHash(); this.extractionFn = extractionFn; +this.filterTuning = filterTuning; + } + + @VisibleForTesting + public BloomDimFilter( + String dimension, Review comment: Unnecessary wrapping, also in many other newly added constructors below in this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693642 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/sql/BloomFilterOperatorConversion.java ## @@ -100,7 +100,8 @@ public DimFilter toDruidFilter( return new BloomDimFilter( druidExpression.getSimpleExtraction().getColumn(), holder, - druidExpression.getSimpleExtraction().getExtractionFn() + druidExpression.getSimpleExtraction().getExtractionFn(), Review comment: Constructor annotated `@VisibleForTesting` doesn't mean it can't be used in production code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310695665
##
File path:
extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java
##
@@ -174,6 +191,13 @@ public ExtractionFn getExtractionFn()
return extractionFn;
}
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ @JsonProperty
+ public FilterTuning getFilterTuning()
+ {
+return filterTuning;
+ }
+
@Override
public String toString()
Review comment:
Please add `filterTuning` in the `toString()` form.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310704426 ## File path: processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java ## @@ -114,10 +141,4 @@ public boolean equals(Object o) .collect(Collectors.toSet()) Review comment: Please fix `Sets.newHashSet(...collect(...))`. I've also opened #8241 about fixing this generically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control
filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310704732
##
File path:
processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java
##
@@ -40,14 +43,25 @@
private static final Joiner COMMA_JOINER = Joiner.on(", ");
private final List dimensions;
+ private final FilterTuning filterTuning;
Review comment:
Please annotate with `@Nullable` everything that is needed in this class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693686 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/sql/BloomFilterOperatorConversion.java ## @@ -114,6 +115,7 @@ public DimFilter toDruidFilter( return new BloomDimFilter( virtualColumn.getOutputName(), holder, + null, Review comment: And here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310694567 ## File path: processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java ## @@ -56,6 +58,7 @@ private final Supplier longPredicateSupplier; private final Supplier floatPredicateSupplier; private final Supplier doublePredicateSupplier; + private final FilterTuning filterTuning; Review comment: Please annotate with `@Nullable` everything that needs to be `@Nullable` in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov opened a new issue #8241: Prohibit creating collections from stream.collect(...) using Structural Search inspection
leventov opened a new issue #8241: Prohibit creating collections from stream.collect(...) using Structural Search inspection URL: https://github.com/apache/incubator-druid/issues/8241 1. ``` new $Coll$($stream$.collect($collector$)) ``` The whole match filter: type=java.util.Collection, check "Within type hierarchy". 2. The same about `Map` 3. ``` $Colls$.$factory$($stream$.collect($collector$)) ``` Filter on `Colls`: regex `Lists|Sets|Maps` - this is to catch Guava's factory methods like `Sets.newHashSet()`. There are violations of these rules in the codebase. See the instruction about how to add a rule [here](https://github.com/apache/incubator-druid/blob/ac856fe4c179093c0a5e309a2e988c54a86e89c5/dev/teamcity.md#creating-a-custom-inspection-from-a-structural-search-pattern). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh opened a new pull request #8240: WIP Speedup Travis CI jobs
ccaominh opened a new pull request #8240: WIP Speedup Travis CI jobs
URL: https://github.com/apache/incubator-druid/pull/8240
_**NOTE: This PR is a work-in-progress to see effects of the Travis CI
changes.**_
### Description
Reorganize Travis CI jobs into smaller (and more) jobs. Add various
maven options to skip unnecessary work and refactored Travis CI job
definitions to follow DRY.
Detailed changes:
.travis.yml
- Refactor build logic to get rid of copy-and-paste logic
- Skip static checks and enable parallelism for maven install
- Split static analysis into different jobs to ease triage
- Use "name" attribute instead of NAME environment variable
- Split "indexing" and "web console" out of "other modules test"
- Split 2 integration test jobs into multiple smaller jobs
build.sh
- Enable parallelism
- Disable more static checks
travis_script_integration.sh
travis_script_integration_part2.sh
integration-tests/README.md
- Use TestNG groups instead of shell scripts and move definition of jobs
into Travis CI yaml
integration-tests/pom.xml
- Show elapsed time of individual tests to aid in rebalancing Travis CI
integration test jobs run time
TestNGGroup.java
- Use TestNG groups to make it easy to have multiple Travis CI
integration test jobs. TestNG groups also make it easier to have an
"other" integration test group and makes it less likely a test will
accidentally not be included in a CI job.
ITHadoopIndexTest.java
AbstractITBatchIndexTest.java
AbstractKafkaIndexerTest.java
ITAppenderatorDriverRealtimeIndexTaskTest.java
ITCompactionTaskTest.java
ITIndexerTest.java
ITKafkaIndexingServiceTest.java
ITKafkaIndexingServiceTransactionalTest.java
ITNestedQueryPushDownTest.java
ITParallelIndexTest.java
ITRealtimeIndexTaskTest.java
ITSystemTableBatchIndexTaskTest.java
ITUnionQueryTest.java
ITSystemTableQueryTest.java
ITTwitterQueryTest.java
ITWikipediaQueryTest.java
ITBasicAuthConfigurationTest.java
ITCoordinatorOverlordProxyAuthTest.java
ITTLSTest.java
- Add TestNG group
- Fix various IntelliJ inspection warnings
This PR has:
- [x] been self-reviewed.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6408: Selector filter for multi-value dimensions is not returning all values
stale[bot] closed issue #6408: Selector filter for multi-value dimensions is not returning all values URL: https://github.com/apache/incubator-druid/issues/6408 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6408: Selector filter for multi-value dimensions is not returning all values
stale[bot] commented on issue #6408: Selector filter for multi-value dimensions is not returning all values URL: https://github.com/apache/incubator-druid/issues/6408#issuecomment-518311283 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310683772
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
+ * {@linkplain #close closed}.
+ */
+ // close. this word no longer has any meaning to me.
Review comment:
Please remove this comment, tt doesn't seem to make any sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310684423
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
+ * {@linkplain #close closed}.
+ */
+ // close. this word no longer has any meaning to me.
+ public Iterable register(Iterable closeables)
Review comment:
Some class may implement both `Iterable` and `Closeable`, so ambiguity with
the other `register()` method is possible. I suggest to call this method
`registerAll()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310686115
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
// Exception caught while setting up the iterator; release
resources.
- Lists.reverse(resources).forEach(CloseQuietly::close);
+ try {
+resources.close();
+ }
+ catch (IOException ex) {
Review comment:
I don't think it makes sense to catch anything weaker than `Throable` in
another `catch (Throwable e) {` block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310686168
##
File path:
processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
##
@@ -280,12 +280,17 @@ public AggregateResult call()
return RowBasedGrouperHelper.makeGrouperIterator(
grouper,
query,
- () -> Lists.reverse(resources).forEach(CloseQuietly::close)
+ resources
);
}
catch (Throwable e) {
Review comment:
Please rename to `t`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of
List
URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310684935
##
File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
##
@@ -123,6 +123,17 @@ private Closer()
return closeable;
}
+ /**
+ * Registers a list of {@code closeable} to be closed when this {@code
Closer} is
Review comment:
If `{@code closeable}` is a reference to the interface `Closeable`, then it
should start with a capital letter. If this is a reference to the parameter,
then it should be `closeables`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov opened a new issue #8239: Specify lineWrappingIndentation in Checkstyle's Indentation check
leventov opened a new issue #8239: Specify lineWrappingIndentation in Checkstyle's Indentation check URL: https://github.com/apache/incubator-druid/issues/8239 See https://checkstyle.org/config_misc.html#Indentation Currently, it's `lineWrappingIndentation` and `forceStrictCondition` are not set: https://github.com/apache/incubator-druid/blob/ac856fe4c179093c0a5e309a2e988c54a86e89c5/codestyle/checkstyle.xml#L112-L115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
leventov commented on a change in pull request #7562: Enable toggling request
logging on/off for different query types
URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310683092
##
File path:
server/src/main/java/org/apache/druid/server/log/FileRequestLoggerProvider.java
##
@@ -58,8 +58,12 @@
@Override
public RequestLogger get()
{
-FileRequestLogger logger = new FileRequestLogger(jsonMapper,
factory.create(1, "RequestLogger-%s"), dir,
-filePattern);
+FileRequestLogger logger = new FileRequestLogger(
+jsonMapper,
Review comment:
In Druid, the wrapping indentation is 4, not 8 spaces. Please also fix other
places in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs
stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs URL: https://github.com/apache/incubator-druid/issues/6539#issuecomment-518185577 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] AlexandreYang opened a new pull request #8238: [statsd-emitter] Add config to send Druid process/service as tag
AlexandreYang opened a new pull request #8238: [statsd-emitter] Add config to send Druid process/service as tag URL: https://github.com/apache/incubator-druid/pull/8238 ### Description When using `statsd-emitter` with Dogstatsd, the Druid process/service name (e.g. broker/coordinator) is included in the metric name e.g. `druid.broker.jvm.gc.cpu`. For the same metric e.g. `jvm.gc.cpu`, multiple metric names will be generated (`druid.broker.jvm.gc.cpu`, `druid.coordinator.jvm.gc.cpu`, `druid.historical.jvm.gc.cpu`, etc). **Since we have multiple metric names it's harder to get statistics across all Druid process types.** This PR intend to solve this issue by adding this new configuration: |property|description|required?|default| ||---|-|---| |`druid.emitter.statsd.dogstatsdServiceAsTag`|If `druid.emitter.statsd.dogstatsdServiceAsTag` is true, druid service (e.g. druid/broker, druid/coordinator, etc) is reported as a tag (e.g. `service:druid/broker`) instead of being included in metric name (e.g. `druid.broker.my_metric`).|no|false| So when this `druid.emitter.statsd.dogstatsdServiceAsTag` is true, the resulting metric send would be `jvm.gc.cpu` as name with a tag `service:druid/broker`. If needed, it can be combined with `druid.emitter.statsd.prefix` to add `druid` as prefix to get metric name like `druid.jvm.gc.cpu` instead of `jvm.gc.cpu`. This PR has: - [ ] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
