[GitHub] jon-wei commented on issue #6100: DOCS: ingestion spec documentation should be expanded

jon-wei commented on issue #6100: DOCS: ingestion spec documentation should be expanded URL: https://github.com/apache/incubator-druid/issues/6100#issuecomment-412762182 we have a new tutorial that gives more information on the ingestion spec:

[GitHub] jon-wei commented on issue #6108: more examples needed for druid sql

jon-wei commented on issue #6108: more examples needed for druid sql URL: https://github.com/apache/incubator-druid/issues/6108#issuecomment-412762004 The new query tutorial has more SQL examples: http://druid.io/docs/latest/tutorials/tutorial-query.html (0.12.2/0.12.3)

[GitHub] jon-wei commented on issue #6171: hetrogenous peon configurations

jon-wei commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412755934 @pdeva yes, the supervisor runs on the overlord itself, the tasks created by the supervisor would be assigned based on the affinity configs

[GitHub] pdeva commented on issue #6171: hetrogenous peon configurations

pdeva commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412752366 the documentation for affinity config reference 'workers' not 'supervisors', so can it be used with KIS?

[GitHub] jihoonson commented on issue #6171: hetrogenous peon configurations

jihoonson commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412744634 > 1.7 queries per second * 60 = 102 queries per minute. > to execute 10K queries per minute requires 166 queries per second!

[GitHub] asdf2014 commented on a change in pull request #6158: Make time-related variables more readable

asdf2014 commented on a change in pull request #6158: Make time-related variables more readable URL: https://github.com/apache/incubator-druid/pull/6158#discussion_r209814543 ## File path: common/src/main/java/io/druid/common/config/Log4jShutdown.java ## @@ -28,11 +28,12

[GitHub] gianm commented on issue #6171: hetrogenous peon configurations

gianm commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412724797 IMO the worker affinity approach is a reasonable way to go. An alternative is to use the "taskResource" object to configure how many slots a

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412723764 > If you have 10,000 queries to execute in 1 min, you need to execute about 1.7 queries per sec 1.7 queries per second *

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412723764 > If you have 10,000 queries to execute in 1 min, you need to execute about 1.7 queries per sec 1.7 queries per second *

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412723764 > If you have 10,000 queries to execute in 1 min, you need to execute about 1.7 queries per sec 1.7 queries per second *

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412723764 > If you have 10,000 queries to execute in 1 min, you need to execute about 1.7 queries per sec 1.7 queries per second

[GitHub] jon-wei commented on issue #6171: hetrogenous peon configurations

jon-wei commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412722003 If you want to give tasks different configurations based on datasource, you could deploy middle managers with different worker configurations

[GitHub] pdeva commented on issue #6171: hetrogenous peon configurations

pdeva commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412719814 i think you can just add 1 extra field to supervisor spec: `memoryPerWorkerThread` The peon will use this field along with the

[GitHub] pdeva commented on issue #6171: hetrogenous peon configurations

pdeva commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412718982 you already allow setting number of worker threads in supervisor spec. since direct memory is allocated proportional to worker threads, you are

[GitHub] asdf2014 commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin

asdf2014 commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin URL: https://github.com/apache/incubator-druid/pull/6162#issuecomment-412718575 Hi, @jihoonson . Thanks for your help. The travis job is succeeded. Yes, using profile to control this will be better, it

[GitHub] pdeva commented on issue #6171: hetrogenous peon configurations

pdeva commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412717856 >How many queries exactly do you submit? thousands. the exact number is irrelevant here. the concept of hetroginty is what is relevant

[GitHub] jihoonson commented on issue #6171: hetrogenous peon configurations

jihoonson commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412717704 How many queries exactly do you submit? This is an automated message from

[GitHub] jihoonson commented on issue #6171: hetrogenous peon configurations

jihoonson commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412716916 Ah yes, it depends on not the dataSource size, but query workloads. > Not really. they respond to queries from broker too. The rate

[GitHub] clintropolis commented on a change in pull request #6158: Make time-related variables more readable

clintropolis commented on a change in pull request #6158: Make time-related variables more readable URL: https://github.com/apache/incubator-druid/pull/6158#discussion_r209802009 ## File path: common/src/main/java/io/druid/common/config/Log4jShutdown.java ## @@ -28,11

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412712312 > All peons essentially do similar things: Not really. they respond to queries from broker too. The rate of querying and size of

[GitHub] asdf2014 commented on a change in pull request #6158: Make time-related variables more readable

asdf2014 commented on a change in pull request #6158: Make time-related variables more readable URL: https://github.com/apache/incubator-druid/pull/6158#discussion_r209799605 ## File path: indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java ## @@

[GitHub] pdeva edited a comment on issue #6171: hetrogenous peon configurations

pdeva edited a comment on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412712312 > All peons essentially do similar things: Not really. they respond to queries from broker too. The rate of querying and size of

[GitHub] asdf2014 commented on a change in pull request #6158: Make time-related variables more readable

asdf2014 commented on a change in pull request #6158: Make time-related variables more readable URL: https://github.com/apache/incubator-druid/pull/6158#discussion_r209799544 ## File path: common/src/main/java/io/druid/common/config/Log4jShutdown.java ## @@ -28,11 +28,12

[GitHub] jihoonson commented on issue #6171: hetrogenous peon configurations

jihoonson commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412711583 What do you mean by `the biggest dataSource`? I wonder why peons should have different resource requirements depending on dataSource. All

[GitHub] pdeva opened a new issue #6171: hetrogenous peon configurations

pdeva opened a new issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171 currently the only way to configure memory (heap and direct) for a peon is in a singular location in the config file. This assumes every datasource is equal,

[GitHub] jihoonson commented on issue #6168: Fix NPE for taskGroupId when rolling update

jihoonson commented on issue #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168#issuecomment-412694346 @pdeva thanks. BTW, this patch is incomplete. I think we need a way to find `taskGroup` using `baseSequenceName` to fully

[GitHub] pdeva edited a comment on issue #6168: Fix NPE for taskGroupId when rolling update

pdeva edited a comment on issue #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168#issuecomment-412693803 replied on druid-dev thread. they have recovered for now but will indeed post logs if it occurs again.

[GitHub] pdeva commented on issue #6168: Fix NPE for taskGroupId when rolling update

pdeva commented on issue #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168#issuecomment-412693803 replied on drudi-dev thread. they have recovered for now but will indeed post logs if it occurs again.

[GitHub] jihoonson commented on issue #6168: Fix NPE for taskGroupId when rolling update

jihoonson commented on issue #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168#issuecomment-412692771 @pdeva this error should never happen once the cluster is fully updated. Do you have the overlord and task logs? If you have,

[GitHub] pdeva commented on issue #6168: Fix NPE for taskGroupId when rolling update

pdeva commented on issue #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168#issuecomment-412691645 note that this issue can occur well after the rolling update. we are seeing this on our fully updated cluster

[GitHub] gianm edited a comment on issue #6155: Fix three bugs with segment publishing.

gianm edited a comment on issue #6155: Fix three bugs with segment publishing. URL: https://github.com/apache/incubator-druid/pull/6155#issuecomment-412686059 ``` [INFO] PMD Failure: io.druid.segment.realtime.appenderator.TransactionalSegmentPublisher:22 Rule:UnusedImports Priority:4

[GitHub] gianm commented on issue #6155: Fix three bugs with segment publishing.

gianm commented on issue #6155: Fix three bugs with segment publishing. URL: https://github.com/apache/incubator-druid/pull/6155#issuecomment-412686059 ```[INFO] PMD Failure: io.druid.segment.realtime.appenderator.TransactionalSegmentPublisher:22 Rule:UnusedImports Priority:4 Avoid unused

Build failed in Jenkins: Druid-Master-Nightly #8

See Changes: [fangjinyang] Implemented makeAggregateCombiner() in -- [...truncated 419.48 KB...] at

[GitHub] gianm closed issue #6164: extensions not updated for 0.12.2

gianm closed issue #6164: extensions not updated for 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6164 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub

[GitHub] gianm commented on issue #6164: extensions not updated for 0.12.2

gianm commented on issue #6164: extensions not updated for 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6164#issuecomment-412675266 I just published these to the Central Repository. Usually, it takes a few hours for caches and whatnot to reflect that.

[GitHub] fjy commented on issue #5881: Docs: Fix bound query keys for Filtering on numeric values

fjy commented on issue #5881: Docs: Fix bound query keys for Filtering on numeric values URL: https://github.com/apache/incubator-druid/pull/5881#issuecomment-412668447 @susielu any chance you can resolve the conflict? This

[incubator-druid] branch master updated: Implemented makeAggregateCombiner() in ArrayOfDoublesSketchAggregatorFactory (#6093)

This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new c47032d Implemented

[GitHub] fjy commented on issue #6093: Implemented makeAggregateCombiner() in ArrayOfDoublesSketchAggregatorFactory

fjy commented on issue #6093: Implemented makeAggregateCombiner() in ArrayOfDoublesSketchAggregatorFactory URL: https://github.com/apache/incubator-druid/pull/6093#issuecomment-412668252  This is an automated message from

[GitHub] fjy commented on issue #6094: Introduce SystemSchema tables (#5989)

fjy commented on issue #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#issuecomment-412668118 @surekhasaharan can you fix the merge conflicts? This is an automated message

[GitHub] clintropolis commented on issue #6016: Druid 'Shapeshifting' Columns

clintropolis commented on issue #6016: Druid 'Shapeshifting' Columns URL: https://github.com/apache/incubator-druid/pull/6016#issuecomment-412667799 Just a note if anyone is testing this with their own data, the encoding selection logic change in

[GitHub] clintropolis commented on a change in pull request #6016: Druid 'Shapeshifting' Columns

clintropolis commented on a change in pull request #6016: Druid 'Shapeshifting' Columns URL: https://github.com/apache/incubator-druid/pull/6016#discussion_r209759964 ## File path: processing/src/main/java/io/druid/segment/data/ShapeShiftingColumn.java ## @@ -0,0 +1,332

[GitHub] jihoonson commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin

jihoonson commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin URL: https://github.com/apache/incubator-druid/pull/6162#issuecomment-412655473 @asdf2014 thanks for taking this issue. I've restarted the failed one. BTW, IMO, it would be better to have an

[GitHub] pdeva opened a new issue #6170: 12 hour segment granularity task producing 6 hour segments

pdeva opened a new issue #6170: 12 hour segment granularity task producing 6 hour segments URL: https://github.com/apache/incubator-druid/issues/6170 Title says it all. screenshot of coordinator console: https://www.screencast.com/t/kB6EPZJCK As you can see segments are

[GitHub] clintropolis commented on a change in pull request #6158: Make time-related variables more readable

clintropolis commented on a change in pull request #6158: Make time-related variables more readable URL: https://github.com/apache/incubator-druid/pull/6158#discussion_r209742177 ## File path: indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java ## @@

[GitHub] jihoonson commented on issue #6163: java.lang.NullPointerException: taskGroupId when middlemanager rolling update to 0.12.2

jihoonson commented on issue #6163: java.lang.NullPointerException: taskGroupId when middlemanager rolling update to 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6163#issuecomment-412645662 Thanks @wpcnjupt. Please check #6168.

[GitHub] jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing.

jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing. URL: https://github.com/apache/incubator-druid/pull/6155#discussion_r209740211 ## File path: server/src/main/java/io/druid/metadata/IndexerSQLMetadataStorageCoordinator.java ##

[GitHub] jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing.

jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing. URL: https://github.com/apache/incubator-druid/pull/6155#discussion_r209739448 ## File path: server/src/main/java/io/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java

[GitHub] jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing.

jihoonson commented on a change in pull request #6155: Fix three bugs with segment publishing. URL: https://github.com/apache/incubator-druid/pull/6155#discussion_r209738649 ## File path: server/src/main/java/io/druid/metadata/IndexerSQLMetadataStorageCoordinator.java ##

[incubator-druid] branch master updated: convert all time-intervals in ISO 8601 format to uppercase in doc files (#6118)

This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 69f5550 convert all time-intervals in

[GitHub] fjy closed pull request #6118: Docs: convert all time-intervals in ISO 8601 format to uppercase

fjy closed pull request #6118: Docs: convert all time-intervals in ISO 8601 format to uppercase URL: https://github.com/apache/incubator-druid/pull/6118 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of

[incubator-druid] branch 0.12.3 updated: Fixes for new docs (#6167)

This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch 0.12.3 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/0.12.3 by this push: new 3690cca Fixes for new docs (#6167)

[GitHub] pdeva opened a new issue #6169: task not being scheduled even when plenty of capacity is available

pdeva opened a new issue #6169: task not being scheduled even when plenty of capacity is available URL: https://github.com/apache/incubator-druid/issues/6169 not sure if this related to #6166 but it happened right after that issue. the node (on which this task can be scheuled) has

Build failed in Jenkins: Druid-Master-Nightly #6

See Changes: [github] New doc fixes (#6156) -- [...truncated 419.37 KB...] at

[GitHub] pdeva commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1

pdeva commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1 URL: https://github.com/apache/incubator-druid/issues/6166#issuecomment-412628004 so i just tried increasing the size of the faulty MM instance to an absolutely absurd amount of cpus, and that seems

[GitHub] jihoonson opened a new pull request #6168: Fix NPE for taskGroupId when rolling update

jihoonson opened a new pull request #6168: Fix NPE for taskGroupId when rolling update URL: https://github.com/apache/incubator-druid/pull/6168 Fixes https://github.com/apache/incubator-druid/issues/6163. This is an

[GitHub] jon-wei commented on a change in pull request #6156: New doc fixes

jon-wei commented on a change in pull request #6156: New doc fixes URL: https://github.com/apache/incubator-druid/pull/6156#discussion_r209706349 ## File path: docs/content/tutorials/tutorial-compaction.md ## @@ -31,7 +31,7 @@ There will be 24 segments for this datasource,

[GitHub] pdeva commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1

pdeva commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1 URL: https://github.com/apache/incubator-druid/issues/6166#issuecomment-412605649 @gianm here is a video of the overlord console: https://www.screencast.com/t/z4eXgbaGe5fy as you can see,

[GitHub] gianm commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1

gianm commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1 URL: https://github.com/apache/incubator-druid/issues/6166#issuecomment-412603383 Hi @pdeva, Are you able to see task logs for any of the tasks that are complaining about 'connection

[GitHub] gianm commented on issue #6165: KIS stuck in reboot loop druid 0.12.2

gianm commented on issue #6165: KIS stuck in reboot loop druid 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6165#issuecomment-412602699 Looks like a duplicate of #6166. This is an automated message from the

[GitHub] gianm commented on issue #6164: extensions not updated for 0.12.2

gianm commented on issue #6164: extensions not updated for 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6164#issuecomment-412602506 It seems to be an oversight, we didn't publish extensions for 0.12.2 yet. Will check into it.

[GitHub] pdeva opened a new issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2

pdeva opened a new issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6166 all the tasks keep restarting with this error in the logs: I tried: - upgrading node from 0.12.1 to 0.12.2 - spinning a whole new server and

[GitHub] pdeva opened a new issue #6165: KIS stuck in reboot loop druid 0.12.2

pdeva opened a new issue #6165: KIS stuck in reboot loop druid 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6165 details here: https://groups.google.com/forum/#!topic/druid-user/K7YmcBJM6T4 please help!

[GitHub] pdeva opened a new issue #6164: extensions not updated for 0.12.2

pdeva opened a new issue #6164: extensions not updated for 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6164 example, io.druid.extensions.contrib:druid-redis-cache the latest version remains 0.12.1 This is an

[GitHub] wpcnjupt opened a new issue #6163: java.lang.NullPointerException: taskGroupId when middlemanager rolling update to 0.12.2

wpcnjupt opened a new issue #6163: java.lang.NullPointerException: taskGroupId when middlemanager rolling update to 0.12.2 URL: https://github.com/apache/incubator-druid/issues/6163 hi I am rolling update to 0.12.2. I have updated the Historical nodes and one middlemanager node.

[GitHub] asdf2014 commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin

asdf2014 commented on issue #6162: Add maven.exec.xxx.skip option for exec-maven-plugin URL: https://github.com/apache/incubator-druid/pull/6162#issuecomment-412456329 It seems that those maven options are in effect. The travis

[GitHub] meazza commented on issue #5480: SQL: not in operator can cause NullPointerException

meazza commented on issue #5480: SQL: not in operator can cause NullPointerException URL: https://github.com/apache/incubator-druid/issues/5480#issuecomment-412438737 ``` { "error": "Unknown exception", "errorMessage": "Cannot build plan for query: SELECT COUNT(*) AS TheCount

[GitHub] asdf2014 opened a new pull request #6162: Add maven.exec.xxx.skip option for exec-maven-plugin

asdf2014 opened a new pull request #6162: Add maven.exec.xxx.skip option for exec-maven-plugin URL: https://github.com/apache/incubator-druid/pull/6162 Try to fix [#6160](https://github.com/apache/incubator-druid/issues/6160)