[GitHub] clintropolis commented on issue #6016: Druid 'Shapeshifting' Columns

2018-07-17 Thread GitBox
clintropolis commented on issue #6016: Druid 'Shapeshifting' Columns URL: https://github.com/apache/incubator-druid/pull/6016#issuecomment-405789147 I don't seem to be able to request reviews, but if I could I'd at least ask @leventov, @gianm, and @nishantmonu51 to have a look when

[GitHub] jihoonson opened a new pull request #6015: Check the kafka topic when comparing checkpoints from tasks with the one stored in metastore

2018-07-17 Thread GitBox
jihoonson opened a new pull request #6015: Check the kafka topic when comparing checkpoints from tasks with the one stored in metastore URL: https://github.com/apache/incubator-druid/pull/6015 The key change is:

[GitHub] jihoonson commented on issue #5871: fix push supervisor error

2018-07-17 Thread GitBox
jihoonson commented on issue #5871: fix push supervisor error URL: https://github.com/apache/incubator-druid/pull/5871#issuecomment-405768263 @hellobabygogo do you mean the KIS can avoid the case when task2 fails with this PR?

[GitHub] jihoonson commented on issue #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on issue #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#issuecomment-405766673 @himanshug thank you for reviewing this PR! Yes, I have tested in our cluster by ingesting 100 GB of TPC-H lineitem table. The

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203215047 ## File path: server/src/main/java/io/druid/client/indexing/ClientQuery.java ## @@ -0,0

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203215022 ## File path: indexing-service/src/main/java/io/druid/indexing/common/task/TaskMonitor.java

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203215000 ## File path:

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203214991 ## File path:

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203215030 ## File path: indexing-service/src/main/java/io/druid/indexing/common/task/ParallelIndexSubTask.java

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203214983 ## File path:

[GitHub] drcrallen commented on issue #5913: Move Caching Cluster Client to java streams and allow parallel intermediate merges

2018-07-17 Thread GitBox
drcrallen commented on issue #5913: Move Caching Cluster Client to java streams and allow parallel intermediate merges URL: https://github.com/apache/incubator-druid/pull/5913#issuecomment-405753300 @jihoonson added query context docs

[GitHub] jihoonson commented on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
jihoonson commented on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405753244 https://travis-ci.org/apache/incubator-druid/jobs/405081069 ```Running io.druid.server.coordinator.CuratorDruidCoordinatorTest Tests

[GitHub] drcrallen opened a new pull request #6014: Optionally refuse to consume new data until the prior chunk is being consumed

2018-07-17 Thread GitBox
drcrallen opened a new pull request #6014: Optionally refuse to consume new data until the prior chunk is being consumed URL: https://github.com/apache/incubator-druid/pull/6014 This PR adds in an optional boolean query context field called `enableBrokerBackpressure`. The impact of when

[GitHub] fjy closed pull request #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor

2018-07-17 Thread GitBox
fjy closed pull request #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor URL: https://github.com/apache/incubator-druid/pull/6012 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of

[GitHub] fjy commented on issue #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor

2018-07-17 Thread GitBox
fjy commented on issue #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor URL: https://github.com/apache/incubator-druid/pull/6012#issuecomment-405737906  This is an automated message from the

[GitHub] fjy removed a comment on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy removed a comment on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405737498 `testTaskRunnerRestart(io.druid.indexing.overlord.hrtr.HttpRemoteTaskRunnerTest) Time elapsed: 0.057 sec <<< ERROR!

[GitHub] fjy commented on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy commented on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405737498 `testTaskRunnerRestart(io.druid.indexing.overlord.hrtr.HttpRemoteTaskRunnerTest) Time elapsed: 0.057 sec <<< ERROR! java.lang.RuntimeException:

[GitHub] fjy removed a comment on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy removed a comment on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405736825 testStatusRemoved(io.druid.indexing.overlord.RemoteTaskRunnerTest) Time elapsed: 21.188 sec <<< ERROR!

[GitHub] fjy removed a comment on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy removed a comment on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405736783 testStatusRemoved(io.druid.indexing.overlord.RemoteTaskRunnerTest) Time elapsed: 21.188 sec <<< ERROR!

[GitHub] fjy commented on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy commented on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405736898 `testStatusRemoved(io.druid.indexing.overlord.RemoteTaskRunnerTest) Time elapsed: 21.188 sec <<< ERROR! java.util.concurrent.TimeoutException:

[GitHub] fjy commented on issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
fjy commented on issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013#issuecomment-405736783 testStatusRemoved(io.druid.indexing.overlord.RemoteTaskRunnerTest) Time elapsed: 21.188 sec <<< ERROR! java.util.concurrent.TimeoutException:

[GitHub] jihoonson opened a new issue #6013: Fix transient test failures

2018-07-17 Thread GitBox
jihoonson opened a new issue #6013: Fix transient test failures URL: https://github.com/apache/incubator-druid/issues/6013 Our travis CI is not much stable. Let's list transient test failures here and fix them. This is an

[GitHub] himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203140032 ## File path:

[GitHub] himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203165340 ## File path: indexing-service/src/main/java/io/druid/indexing/common/Counters.java ## @@

[GitHub] himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203139893 ## File path:

[GitHub] himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
himanshug commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203162548 ## File path: indexing-service/src/main/java/io/druid/indexing/common/task/ParallelIndexSubTask.java

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203169187 ## File path: indexing-service/src/main/java/io/druid/indexing/common/IndexTaskClient.java

[GitHub] jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
jihoonson commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203167880 ## File path: indexing-service/src/main/java/io/druid/indexing/common/IndexTaskClient.java

[GitHub] fjy commented on issue #6005: Add concat and textcat SQL functions

2018-07-17 Thread GitBox
fjy commented on issue #6005: Add concat and textcat SQL functions URL: https://github.com/apache/incubator-druid/pull/6005#issuecomment-405716988  This is an automated message from the Apache Git Service. To respond to the

[GitHub] jihoonson opened a new pull request #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor

2018-07-17 Thread GitBox
jihoonson opened a new pull request #6012: [Backport] Fix NPE while handling CheckpointNotice in KafkaSupervisor URL: https://github.com/apache/incubator-druid/pull/6012 Backport of #5996 to 0.12.2. This is an automated

[GitHub] jihoonson edited a comment on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid

2018-07-17 Thread GitBox
jihoonson edited a comment on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid URL: https://github.com/apache/incubator-druid/pull/6007#issuecomment-405699220 @drcrallen thanks for raising #6009.

[GitHub] jihoonson commented on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid

2018-07-17 Thread GitBox
jihoonson commented on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid URL: https://github.com/apache/incubator-druid/pull/6007#issuecomment-405699220 @drcrallen thanks for rasing #6009. This is

[GitHub] drcrallen commented on issue #5921: Support annotation binding in Druid extensions

2018-07-17 Thread GitBox
drcrallen commented on issue #5921: Support annotation binding in Druid extensions URL: https://github.com/apache/incubator-druid/pull/5921#issuecomment-405695124 The jackson injector we use was written before the guice module was available IIRC. It is worth replacing with the general

[GitHub] fjy commented on issue #6011: Update readme

2018-07-17 Thread GitBox
fjy commented on issue #6011: Update readme URL: https://github.com/apache/incubator-druid/pull/6011#issuecomment-405691144 @gianm I think we should try to direct people to the Druid webpage as much as possible and avoid repeating information in too many places.

[GitHub] fjy closed pull request #6011: Update readme

2018-07-17 Thread GitBox
fjy closed pull request #6011: Update readme URL: https://github.com/apache/incubator-druid/pull/6011 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a

[GitHub] leventov closed pull request #6007: Move build badge to https://travis-ci.org/apache/incubator-druid

2018-07-17 Thread GitBox
leventov closed pull request #6007: Move build badge to https://travis-ci.org/apache/incubator-druid URL: https://github.com/apache/incubator-druid/pull/6007 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of

[GitHub] drcrallen commented on issue #6003: Can we switch to Tez or Spark for batch ingestion ingestion instead of Map Reduce?

2018-07-17 Thread GitBox
drcrallen commented on issue #6003: Can we switch to Tez or Spark for batch ingestion ingestion instead of Map Reduce? URL: https://github.com/apache/incubator-druid/issues/6003#issuecomment-405688698 IMHO something like an apache beam focused indexing workflow would be more portable.

Re: Multi-threaded Druid Tests/Benchmarks

2018-07-17 Thread Charles Allen
Unfortunately I think multi-threaded test coverage is kind of weak and historically very hart to test. There are some topN benchmarks but they are very limited as they don't scale well (heap gets blasted from incremental index) with a large concurrency level. On Sun, Jul 15, 2018 at 6:35 AM

[GitHub] drcrallen commented on issue #5913: Move Caching Cluster Client to java streams and allow parallel intermediate merges

2018-07-17 Thread GitBox
drcrallen commented on issue #5913: Move Caching Cluster Client to java streams and allow parallel intermediate merges URL: https://github.com/apache/incubator-druid/pull/5913#issuecomment-405666760 lots of unrelated test failures

TopN folding and result ordering (and maybe group by)

2018-07-17 Thread Charles Allen
I brought this up in the Dev Sync but thought I would write up a couple of findings here. We have some large results in TopN queries that come back, and have been looking at optimizations in the TopN (or GroupBy) query path in order to accommodate these larger results sets returning from many

[GitHub] leventov commented on issue #5980: Various changes about a few coding specifications

2018-07-17 Thread GitBox
leventov commented on issue #5980: Various changes about a few coding specifications URL: https://github.com/apache/incubator-druid/pull/5980#issuecomment-405661972 @asdf2014 1. I don't know why `CostBalancerStrategyBenchmark` and `WrappedRoaringBitmapTest` are written in such an

[GitHub] fjy commented on issue #6011: Update readme

2018-07-17 Thread GitBox
fjy commented on issue #6011: Update readme URL: https://github.com/apache/incubator-druid/pull/6011#issuecomment-405659153 @gianm added disclaimer This is an automated message from the Apache Git Service. To respond to the

[GitHub] gianm commented on issue #6011: Update readme

2018-07-17 Thread GitBox
gianm commented on issue #6011: Update readme URL: https://github.com/apache/incubator-druid/pull/6011#issuecomment-405658112 @fjy could you include the incubation disclaimer too (https://incubator.apache.org/guides/branding.html#disclaimers)? Perhaps also some of the same explanatory

Re: This week's dev sync

2018-07-17 Thread Charles Allen
Some on the call mentioned there were some oddities with logins this week. Using incognito mode worked to fix the login issues On Tue, Jul 17, 2018 at 9:50 AM Charles Allen wrote: > To join the video meeting, click this link: > https://meet.google.com/ozi-rtfg-ags > Otherwise, to join by phone,

[GitHub] QubitPi closed issue #5922: Logging in Druid extensions

2018-07-17 Thread GitBox
QubitPi closed issue #5922: Logging in Druid extensions URL: https://github.com/apache/incubator-druid/issues/5922 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and

[GitHub] QubitPi commented on issue #5922: Logging in Druid extensions

2018-07-17 Thread GitBox
QubitPi commented on issue #5922: Logging in Druid extensions URL: https://github.com/apache/incubator-druid/issues/5922#issuecomment-405655142 io.druid.java.util.common.logger.Logger works. Thanks @gianm ! This is an

[GitHub] QubitPi removed a comment on issue #5922: Logging in Druid extensions

2018-07-17 Thread GitBox
QubitPi removed a comment on issue #5922: Logging in Druid extensions URL: https://github.com/apache/incubator-druid/issues/5922#issuecomment-401893414 Thanks, @gianm . I have the following log4j2 under resource folder, but problem is still there when I tried to print a debug() message:

[GitHub] drcrallen commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
drcrallen commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203102303 ## File path: indexing-service/src/main/java/io/druid/indexing/common/IndexTaskClient.java

[GitHub] drcrallen commented on a change in pull request #5492: Native parallel batch indexing without shuffle

2018-07-17 Thread GitBox
drcrallen commented on a change in pull request #5492: Native parallel batch indexing without shuffle URL: https://github.com/apache/incubator-druid/pull/5492#discussion_r203101307 ## File path:

[GitHub] fjy opened a new pull request #6010: Update

2018-07-17 Thread GitBox
fjy opened a new pull request #6010: Update URL: https://github.com/apache/incubator-druid/pull/6010 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL

This week's dev sync

2018-07-17 Thread Charles Allen
To join the video meeting, click this link: https://meet.google.com/ozi-rtfg-ags Otherwise, to join by phone, dial +1 442-666-1256 and enter this PIN: 6867# To view more phone numbers, click this link: https://tel.meet/ozi-rtfg-ags?hs=5

[GitHub] drcrallen opened a new issue #6009: Coveralls badge is pointing to old repo

2018-07-17 Thread GitBox
drcrallen opened a new issue #6009: Coveralls badge is pointing to old repo URL: https://github.com/apache/incubator-druid/issues/6009 As noted in https://github.com/apache/incubator-druid/pull/6007 the coveralls badge points to the old repo via https://coveralls.io/github/druid-io/druid

[GitHub] drcrallen commented on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid

2018-07-17 Thread GitBox
drcrallen commented on issue #6007: Move build badge to https://travis-ci.org/apache/incubator-druid URL: https://github.com/apache/incubator-druid/pull/6007#issuecomment-405648172 @jihoonson unfortunately it isn't a simple drop and replace. It requires some permissions or settings

[GitHub] salman028 opened a new issue #6008: groupBy Query took too much time

2018-07-17 Thread GitBox
salman028 opened a new issue #6008: groupBy Query took too much time URL: https://github.com/apache/incubator-druid/issues/6008 i am using groupBy Query over data that is above the limit of 500k . I am using druid 0.11.0 and groupBy v2 still it took too much time to get the results .Is

[GitHub] hoesler edited a comment on issue #5221: Support Hadoop batch ingestion for druid-azure-extensions

2018-07-17 Thread GitBox
hoesler edited a comment on issue #5221: Support Hadoop batch ingestion for druid-azure-extensions URL: https://github.com/apache/incubator-druid/pull/5221#issuecomment-405304318 @spyk Thanks for adding this feature! I tried to get this working with a local mapred job but got an