[jira] [Resolved] (HIVE-21263) DruidStorageHandler should not disable the druid datasource when a hive external table is dropped

[Seung-Hyun Cheong (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong resolved HIVE-21263.
--
Resolution: Won't Fix

> DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped
> -
>
> Key: HIVE-21263
> URL: https://issues.apache.org/jira/browse/HIVE-21263
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> DruidStorageHandler should not disable the druid datasource of the hive 
> external table.
>  
> [DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
> {code:java}
> if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
>   LOG.info("Dropping with purge all the data for data source {}", 
> dataSourceName);
>   List
>   dataSegmentList =
>   DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
>   getDruidMetadataStorageTablesConfig(),
>   dataSourceName);
>   if (dataSegmentList.isEmpty()) {
> LOG.info("Nothing to delete for data source {}", dataSourceName);
> return;
>   }
>   for (DataSegment dataSegment : dataSegmentList) {
> try {
>   deleteSegment(dataSegment);
> } catch (SegmentLoadingException e) {
>   LOG.error(String.format("Error while deleting segment [%s]", 
> dataSegment.getIdentifier()), e);
> }
>   }
> }
> if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
> getDruidMetadataStorageTablesConfig(),
> dataSourceName)) {
>   LOG.info("Successfully dropped druid data source {}", dataSourceName);
> }
> {code}
>  
> If a druid datasource is disabled
>  # We can not query to the datasource anymore.
>  # It takes too much time to enable again a datasource, if the datasource is 
> big.
>  
> The data of an external table should be usable, even after the external table 
> is dropped from hive.
>  
> So, DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong resolved HIVE-21194.
--
Resolution: Won't Fix

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: HIVE-21194.patch)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Status: Open  (was: Patch Available)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: HIVE-21194.patch)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: HIVE-21194.patch
Status: Patch Available  (was: In Progress)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766948#comment-16766948
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/13/19 11:59 AM:


I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"a51e6aeaf816bdeea5e91ba3a0fab8a31b3a496d".
{code:java}
git checkout a51e6aeaf816bdeea5e91ba3a0fab8a31b3a496d{code}
 

 

The fixed DruidStorageHandler.java
{code:java}
Index: 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
--- 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(revision b0d01eb68a604bb6efb68455193803f0651dc265)
+++ 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(date 1550046272509)
@@ -98,6 +98,7 @@
import org.jboss.netty.handler.codec.http.HttpMethod;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Period;
import org.skife.jdbi.v2.exceptions.CallbackFailedException;
import org.slf4j.Logger;
@@ -752,7 +753,7 @@

@Override public void configureOutputJobProperties(TableDesc tableDesc, 
Map jobProperties) {
jobProperties.put(Constants.DRUID_DATA_SOURCE, tableDesc.getTableName());
- jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
+ jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime(DateTimeZone.UTC).toString());
jobProperties.put(DruidConstants.DRUID_JOB_WORKING_DIRECTORY, 
getStagingWorkingDir().toString());
// DruidOutputFormat will write segments in an intermediate directory
jobProperties.put(DruidConstants.DRUID_SEGMENT_INTERMEDIATE_DIRECTORY,
{code}
 

Building jar
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted hive service.

 

 

Now, the version of the segment inserted after patched is UTC.

!https://media.oss.navercorp.com/user/7420/files/25a18e30-2fb0-11e9-829a-6fc37a382da9!

 

Finally, KillTask worked properly. (I don't need to wait for 9 hours in UTC+9)
{code:java}
{
  "task": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
  "status": {
"id": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
"status": "SUCCESS",
"duration": 7084
  }
}
{code}
 

 

 


was (Author: seunghyun.cheong):
I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java
{code:java}
Index: 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
--- 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(revision b0d01eb68a604bb6efb68455193803f0651dc265)
+++ 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(date 1550046272509)
@@ -98,6 +98,7 @@
import org.jboss.netty.handler.codec.http.HttpMethod;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Period;
import org.skife.jdbi.v2.exceptions.CallbackFailedException;
import org.slf4j.Logger;
@@ -752,7 +753,7 @@

@Override public void configureOutputJobProperties(TableDesc tableDesc, 
Map jobProperties) {
jobProperties.put(Constants.DRUID_DATA_SOURCE, tableDesc.getTableName());
- jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
+ jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime(DateTimeZone.UTC).toString());
jobProperties.put(DruidConstants.DRUID_JOB_WORKING_DIRECTORY, 
getStagingWorkingDir().toString());
// DruidOutputFormat will write segments in an intermediate directory
jobProperties.put(DruidConstants.DRUID_SEGMENT_INTERMEDIATE_DIRECTORY,
{code}
 

Building jar
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Status: Open  (was: Patch Available)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21194 started by Seung-Hyun Cheong.

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21263) DruidStorageHandler should not disable the druid datasource when a hive external table is dropped

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong reassigned HIVE-21263:



> DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped
> -
>
> Key: HIVE-21263
> URL: https://issues.apache.org/jira/browse/HIVE-21263
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> DruidStorageHandler should not disable the druid datasource of the hive 
> external table.
>  
> [DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
>  
>  
> {code:java}
> if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
>   LOG.info("Dropping with purge all the data for data source {}", 
> dataSourceName);
>   List
>   dataSegmentList =
>   DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
>   getDruidMetadataStorageTablesConfig(),
>   dataSourceName);
>   if (dataSegmentList.isEmpty()) {
> LOG.info("Nothing to delete for data source {}", dataSourceName);
> return;
>   }
>   for (DataSegment dataSegment : dataSegmentList) {
> try {
>   deleteSegment(dataSegment);
> } catch (SegmentLoadingException e) {
>   LOG.error(String.format("Error while deleting segment [%s]", 
> dataSegment.getIdentifier()), e);
> }
>   }
> }
> if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
> getDruidMetadataStorageTablesConfig(),
> dataSourceName)) {
>   LOG.info("Successfully dropped druid data source {}", dataSourceName);
> }
> {code}
>  
>  
> If a druid datasource is disabled
>  # We can not query to a datasource.
>  # It takes too much time to enable again a datasource, if the datasource is 
> big.
>  
> The data of an external table should be usable, even after the external table 
> is dropped from hive.
>  
> So, DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21263) DruidStorageHandler should not disable the druid datasource when a hive external table is dropped

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21263:
-
Description: 
DruidStorageHandler should not disable the druid datasource of the hive 
external table.

 

[DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
{code:java}
if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
  LOG.info("Dropping with purge all the data for data source {}", 
dataSourceName);
  List
  dataSegmentList =
  DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
  getDruidMetadataStorageTablesConfig(),
  dataSourceName);
  if (dataSegmentList.isEmpty()) {
LOG.info("Nothing to delete for data source {}", dataSourceName);
return;
  }
  for (DataSegment dataSegment : dataSegmentList) {
try {
  deleteSegment(dataSegment);
} catch (SegmentLoadingException e) {
  LOG.error(String.format("Error while deleting segment [%s]", 
dataSegment.getIdentifier()), e);
}
  }
}
if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
getDruidMetadataStorageTablesConfig(),
dataSourceName)) {
  LOG.info("Successfully dropped druid data source {}", dataSourceName);
}
{code}
 

If a druid datasource is disabled
 # We can not query to the datasource anymore.
 # It takes too much time to enable again a datasource, if the datasource is 
big.

 

The data of an external table should be usable, even after the external table 
is dropped from hive.

 

So, DruidStorageHandler should not disable the druid datasource when a hive 
external table is dropped.

 

 

 

  was:
DruidStorageHandler should not disable the druid datasource of the hive 
external table.

 

[DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
{code:java}
if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
  LOG.info("Dropping with purge all the data for data source {}", 
dataSourceName);
  List
  dataSegmentList =
  DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
  getDruidMetadataStorageTablesConfig(),
  dataSourceName);
  if (dataSegmentList.isEmpty()) {
LOG.info("Nothing to delete for data source {}", dataSourceName);
return;
  }
  for (DataSegment dataSegment : dataSegmentList) {
try {
  deleteSegment(dataSegment);
} catch (SegmentLoadingException e) {
  LOG.error(String.format("Error while deleting segment [%s]", 
dataSegment.getIdentifier()), e);
}
  }
}
if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
getDruidMetadataStorageTablesConfig(),
dataSourceName)) {
  LOG.info("Successfully dropped druid data source {}", dataSourceName);
}
{code}
 

If a druid datasource is disabled
 # We can not query to a datasource.
 # It takes too much time to enable again a datasource, if the datasource is 
big.

 

The data of an external table should be usable, even after the external table 
is dropped from hive.

 

So, DruidStorageHandler should not disable the druid datasource when a hive 
external table is dropped.

 

 

 


> DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped
> -
>
> Key: HIVE-21263
> URL: https://issues.apache.org/jira/browse/HIVE-21263
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> DruidStorageHandler should not disable the druid datasource of the hive 
> external table.
>  
> [DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
> {code:java}
> if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
>   LOG.info("Dropping with purge all the data for data source {}", 
> dataSourceName);
>   List
>   dataSegmentList =
>   DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
>   getDruidMetadataStorageTablesConfig(),
>   dataSourceName);
>   if (dataSegmentList.isEmpty()) {
> LOG.info("Nothing to delete for data source {}", dataSourceName);
> return;
>   }
>   for (DataSegment dataSegment : dataSegmentList) {
> try {
>   deleteSegment(dataSegment);
> } catch (SegmentLoadingException e) {
>   LOG.error(String.format("Error while deleting segment [%s]", 
> dataSegment.getIdentifier()), e);
> }
>   }
> }
> if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
> 

[jira] [Updated] (HIVE-21263) DruidStorageHandler should not disable the druid datasource when a hive external table is dropped

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21263:
-
Description: 
DruidStorageHandler should not disable the druid datasource of the hive 
external table.

 

[DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
{code:java}
if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
  LOG.info("Dropping with purge all the data for data source {}", 
dataSourceName);
  List
  dataSegmentList =
  DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
  getDruidMetadataStorageTablesConfig(),
  dataSourceName);
  if (dataSegmentList.isEmpty()) {
LOG.info("Nothing to delete for data source {}", dataSourceName);
return;
  }
  for (DataSegment dataSegment : dataSegmentList) {
try {
  deleteSegment(dataSegment);
} catch (SegmentLoadingException e) {
  LOG.error(String.format("Error while deleting segment [%s]", 
dataSegment.getIdentifier()), e);
}
  }
}
if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
getDruidMetadataStorageTablesConfig(),
dataSourceName)) {
  LOG.info("Successfully dropped druid data source {}", dataSourceName);
}
{code}
 

If a druid datasource is disabled
 # We can not query to a datasource.
 # It takes too much time to enable again a datasource, if the datasource is 
big.

 

The data of an external table should be usable, even after the external table 
is dropped from hive.

 

So, DruidStorageHandler should not disable the druid datasource when a hive 
external table is dropped.

 

 

 

  was:
DruidStorageHandler should not disable the druid datasource of the hive 
external table.

 

[DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]

 

 
{code:java}
if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
  LOG.info("Dropping with purge all the data for data source {}", 
dataSourceName);
  List
  dataSegmentList =
  DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
  getDruidMetadataStorageTablesConfig(),
  dataSourceName);
  if (dataSegmentList.isEmpty()) {
LOG.info("Nothing to delete for data source {}", dataSourceName);
return;
  }
  for (DataSegment dataSegment : dataSegmentList) {
try {
  deleteSegment(dataSegment);
} catch (SegmentLoadingException e) {
  LOG.error(String.format("Error while deleting segment [%s]", 
dataSegment.getIdentifier()), e);
}
  }
}
if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
getDruidMetadataStorageTablesConfig(),
dataSourceName)) {
  LOG.info("Successfully dropped druid data source {}", dataSourceName);
}
{code}
 

 

If a druid datasource is disabled
 # We can not query to a datasource.
 # It takes too much time to enable again a datasource, if the datasource is 
big.

 

The data of an external table should be usable, even after the external table 
is dropped from hive.

 

So, DruidStorageHandler should not disable the druid datasource when a hive 
external table is dropped.

 

 

 


> DruidStorageHandler should not disable the druid datasource when a hive 
> external table is dropped
> -
>
> Key: HIVE-21263
> URL: https://issues.apache.org/jira/browse/HIVE-21263
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> DruidStorageHandler should not disable the druid datasource of the hive 
> external table.
>  
> [DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
> {code:java}
> if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
>   LOG.info("Dropping with purge all the data for data source {}", 
> dataSourceName);
>   List
>   dataSegmentList =
>   DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
>   getDruidMetadataStorageTablesConfig(),
>   dataSourceName);
>   if (dataSegmentList.isEmpty()) {
> LOG.info("Nothing to delete for data source {}", dataSourceName);
> return;
>   }
>   for (DataSegment dataSegment : dataSegmentList) {
> try {
>   deleteSegment(dataSegment);
> } catch (SegmentLoadingException e) {
>   LOG.error(String.format("Error while deleting segment [%s]", 
> dataSegment.getIdentifier()), e);
> }
>   }
> }
> if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
> getDruidMetadataStorageTablesConfig(),

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766948#comment-16766948
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/13/19 9:10 AM:
---

I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java
{code:java}
Index: 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
--- 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(revision b0d01eb68a604bb6efb68455193803f0651dc265)
+++ 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(date 1550046272509)
@@ -98,6 +98,7 @@
import org.jboss.netty.handler.codec.http.HttpMethod;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Period;
import org.skife.jdbi.v2.exceptions.CallbackFailedException;
import org.slf4j.Logger;
@@ -752,7 +753,7 @@

@Override public void configureOutputJobProperties(TableDesc tableDesc, 
Map jobProperties) {
jobProperties.put(Constants.DRUID_DATA_SOURCE, tableDesc.getTableName());
- jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
+ jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime(DateTimeZone.UTC).toString());
jobProperties.put(DruidConstants.DRUID_JOB_WORKING_DIRECTORY, 
getStagingWorkingDir().toString());
// DruidOutputFormat will write segments in an intermediate directory
jobProperties.put(DruidConstants.DRUID_SEGMENT_INTERMEDIATE_DIRECTORY,
{code}
 

Building jar
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted hive service.

 

 

Now, the version of the segment inserted after patched is UTC.

!https://media.oss.navercorp.com/user/7420/files/25a18e30-2fb0-11e9-829a-6fc37a382da9!

 

Finally, KillTask worked properly. (I don't need to wait for 9 hours in UTC+9)
{code:java}
{
  "task": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
  "status": {
"id": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
"status": "SUCCESS",
"duration": 7084
  }
}
{code}
 

 

 


was (Author: seunghyun.cheong):
I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java
{code:java}
Index: 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
--- 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(revision b0d01eb68a604bb6efb68455193803f0651dc265)
+++ 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(date 1550046272509)
@@ -98,6 +98,7 @@
import org.jboss.netty.handler.codec.http.HttpMethod;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Period;
import org.skife.jdbi.v2.exceptions.CallbackFailedException;
import org.slf4j.Logger;
@@ -752,7 +753,7 @@

@Override public void configureOutputJobProperties(TableDesc tableDesc, 
Map jobProperties) {
jobProperties.put(Constants.DRUID_DATA_SOURCE, tableDesc.getTableName());
- jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
+ jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime(DateTimeZone.UTC).toString());
jobProperties.put(DruidConstants.DRUID_JOB_WORKING_DIRECTORY, 
getStagingWorkingDir().toString());
// DruidOutputFormat will write segments in an intermediate directory
jobProperties.put(DruidConstants.DRUID_SEGMENT_INTERMEDIATE_DIRECTORY,
{code}
 

Building jar
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted 

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766948#comment-16766948
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/13/19 9:04 AM:
---

I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java
{code:java}
Index: 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
--- 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(revision b0d01eb68a604bb6efb68455193803f0651dc265)
+++ 
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
(date 1550046272509)
@@ -98,6 +98,7 @@
import org.jboss.netty.handler.codec.http.HttpMethod;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Period;
import org.skife.jdbi.v2.exceptions.CallbackFailedException;
import org.slf4j.Logger;
@@ -752,7 +753,7 @@

@Override public void configureOutputJobProperties(TableDesc tableDesc, 
Map jobProperties) {
jobProperties.put(Constants.DRUID_DATA_SOURCE, tableDesc.getTableName());
- jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
+ jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime(DateTimeZone.UTC).toString());
jobProperties.put(DruidConstants.DRUID_JOB_WORKING_DIRECTORY, 
getStagingWorkingDir().toString());
// DruidOutputFormat will write segments in an intermediate directory
jobProperties.put(DruidConstants.DRUID_SEGMENT_INTERMEDIATE_DIRECTORY,
{code}
 

Building jar
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted hive service.

 

 

Now, the version of the segment inserted after patched is UTC.

!https://media.oss.navercorp.com/user/7420/files/25a18e30-2fb0-11e9-829a-6fc37a382da9!

 

Finally, KillTask worked properly.

 
{code:java}
{
  "task": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
  "status": {
"id": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
"status": "SUCCESS",
"duration": 7084
  }
}
{code}
 

 

 

 

 


was (Author: seunghyun.cheong):
I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".

 
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java

!image-2019-02-13-17-53-24-488.png!

 

Building jar

 
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted hive service.

 

 

Now, the version of the segment inserted after patched is UTC.

!https://media.oss.navercorp.com/user/7420/files/25a18e30-2fb0-11e9-829a-6fc37a382da9!

 

Finally, KillTask worked properly.

 
{code:java}
{
  "task": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
  "status": {
"id": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
"status": "SUCCESS",
"duration": 7084
  }
}
{code}
 

 

 

 

 

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png, image-2019-02-13-17-53-24-488.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  

[jira] [Commented] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766948#comment-16766948
 ] 

Seung-Hyun Cheong commented on HIVE-21194:
--

I'm using HDP 3.1.1. And the version of hive in HDP 3.1.1 is 3.1.0. But it's 
also contains several patches.

(Ref. 
[https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_hive.html])

 

To build exactly same hive-druid-handler, I checked out the commit 
"2ff9c5229d1470afab12258862f801cf14dc2a2d".

 
{code:java}
git checkout 2ff9c5229d1470afab12258862f801cf14dc2a2d{code}
 

 

The fixed DruidStorageHandler.java

!image-2019-02-13-17-53-24-488.png!

 

Building jar

 
{code:java}
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"

mvn clean install -DskipTests
{code}
 

I deployed fixed jar to all nodes, and restarted hive service.

 

 

Now, the version of the segment inserted after patched is UTC.

!https://media.oss.navercorp.com/user/7420/files/25a18e30-2fb0-11e9-829a-6fc37a382da9!

 

Finally, KillTask worked properly.

 
{code:java}
{
  "task": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
  "status": {
"id": 
"kill_upload_2019-02-12T00:00:00.000Z_2019-02-13T00:00:00.000Z_2019-02-13T08:00:08.754Z",
"status": "SUCCESS",
"duration": 7084
  }
}
{code}
 

 

 

 

 

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png, image-2019-02-13-17-53-24-488.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-13-17-53-24-488.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: HIVE-21194.patch
Status: Patch Available  (was: In Progress)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png, image-2019-02-13-17-53-24-488.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-13-17-53-24-488.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png, image-2019-02-13-17-53-24-488.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758063#comment-16758063
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/13/19 8:29 AM:
---

[~bslim] 

 

I'm using HDP 3.1.1. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 

 

So, in KST(UTC+9), I can't delete a segment for 9 hours...

 

 

 

In Druid, a segment published by IndexTask has a version UTC (below)

!image-2019-02-01-16-32-17-093.png!
 * The interval of the segment: UTC
 * The version of the segment: UTC

 

So, there is no such problem.

 

 

 

And, I didn't test my patch actually... sorry for that. I'll submit a patch 
again after testing. (Currently I deleted my patch)

(QA failed, because I forgot the import statement on my patch. "import 
org.joda.time.DateTimeZone;")

 

 


was (Author: seunghyun.cheong):
[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-01-16-17-52-419.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-01-15-07-06-893.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-01-12-34-44-731.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-01-12-44-22-331.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: image-2019-02-01-16-17-36-598.png)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png, 
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Affects Version/s: 3.1.0

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Status: Open  (was: Patch Available)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758063#comment-16758063
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/1/19 10:33 AM:
---

[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 

 

So, in KST(UTC+9), I can't delete a segment for 9 hours...

 

 

 

In Druid, a segment published by IndexTask has a version UTC (below)

!image-2019-02-01-16-32-17-093.png!
 * The interval of the segment: UTC
 * The version of the segment: UTC

 

So, there is no such problem.

 

 

 

And, I didn't test my patch actually... sorry for that. I'll submit a patch 
again after testing. (Currently I deleted my patch)

(QA failed, because I forgot the import statement on my patch. "import 
org.joda.time.DateTimeZone;")

 

 


was (Author: seunghyun.cheong):
[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.

 
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted

 
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs

 
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 

[jira] [Work started] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21194 started by Seung-Hyun Cheong.

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758063#comment-16758063
 ] 

Seung-Hyun Cheong edited comment on HIVE-21194 at 2/1/19 8:03 AM:
--

[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.

 
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted

 
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs

 
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 

 

So, in KST(UTC+9), I can't delete a segment for 9 hours...

 

 

 

In Druid, a segment published by IndexTask has a version UTC (below)

!image-2019-02-01-16-32-17-093.png!
 * The interval of the segment: UTC
 * The version of the segment: UTC

 

So, there is no such problem.

 

 

 

And, I didn't test my patch actually... sorry for that. I'll submit a patch 
again after testing. (Currently I deleted my patch)

(QA failed, because I forgot the import statement on my patch. "import 
org.joda.time.DateTimeZone;")

 

 


was (Author: seunghyun.cheong):
[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.

 
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted

 
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs

 
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 

[jira] [Commented] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758063#comment-16758063
 ] 

Seung-Hyun Cheong commented on HIVE-21194:
--

[~bslim] 

 

I'm using HDP 3.1.0. (Hive 3.1.0, Druid 0.12.1)

 

A query to insert data from HDFS to druid.

 
{code:java}
INSERT
INTO
  TABLE druid.data_table
SELECT
  `time` AS `__time`,
  .
  .
  .
FROM
  hdfs.data_table
WHERE
  .
  .
  .{code}
 

The inserted segment meta

!image-2019-02-01-16-31-56-958.png!
 * The interval of the segment: UTC (Green one)
 * The version of the segment: UTC+9 (Red one, it's KST.)

 

To delete one of the segments I inserted

 
{code:java}
// Disabling the segment
DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

// Deleting the segment
DELETE 
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}{code}
 

 

Then the exception occurs

 
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 

 

So, in KST(UTC+9), I can't delete a segment for 9 hours...

 

 

 

In Druid, a segment published by IndexTask has a version UTC (below)

!image-2019-02-01-16-32-17-093.png!
 * The interval of the segment: UTC
 * The version of the segment: UTC

 

So, there is no such problem.

 

 

 

And, I didn't test my patch actually... sorry for that. I'll submit a patch 
again after testing.

(QA failed, because I forgot the import statement on my patch. "import 
org.joda.time.DateTimeZone;")

 

 

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-16-32-17-093.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-16-31-56-958.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png, 
> image-2019-02-01-16-31-56-958.png, image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-16-17-36-598.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-16-17-52-419.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png, 
> image-2019-02-01-16-17-36-598.png, image-2019-02-01-16-17-52-419.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-15-07-06-893.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png, image-2019-02-01-15-07-06-893.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-12-44-22-331.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png, 
> image-2019-02-01-12-44-22-331.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: image-2019-02-01-12-34-44-731.png

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-12-34-44-731.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: (was: HIVE-21194.patch)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
> segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. KillTask compares versions

[KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")

[TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
  
h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")

[DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only, DruidStorageHandler should set a version of 
segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. KillTask compares versions

[KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. KillTask compares versions

[KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")

[TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
  
h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")

[DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
version of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. KillTask compares versions 
[KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z") 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. KillTask compares versions 
[KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z") 
[TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
  
h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00") 
[DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
version of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. [KillTask version (UTC, e.g. 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Attachment: HIVE-21194.patch
Status: Patch Available  (was: In Progress)

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. [KillTask compares 
> versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. [KillTask version (UTC, e.g. 
> "2019-01-30T16:58:29.992Z")|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. [Segment version (UTC+9, e.g. 
> "2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
> version of segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21194 started by Seung-Hyun Cheong.

> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. Reason
> h3. [KillTask compares 
> versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>   
> h3. [KillTask version (UTC, e.g. 
> "2019-01-30T16:58:29.992Z")|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>   
> h3. [Segment version (UTC+9, e.g. 
> "2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. Suggestion
> h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
> version of segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, e.g. 
"2019-01-30T16:58:29.992Z")|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
 

 
h3. [Segment version (UTC+9, e.g. 
"2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
version of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, 
"{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
  
h3. [KillTask version (UTC, e.g. 
"2019-01-30T16:58:29.992Z")|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
  
h3. [Segment version (UTC+9, e.g. 
"2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
version of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, e.g. 
"2019-01-30T16:58:29.992Z")|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. Reason
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, 
"{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
 

 
h3. [Segment version (UTC+9, 
"2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. Suggestion
h3. Because druid uses UTC only for now, DruidStorageHandler should set a 
version of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. AS-IS
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, 
"{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = 

[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong updated HIVE-21194:
-
Description: 
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. AS-IS
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, 
"{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
{code:java}
version = DateTimes.nowUtc().toString();
{code}
 

 
h3. [Segment version (UTC+9, 
"2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
{code:java}
jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
DateTime().toString());
{code}
 

 
h1. TO-BE

 

Because druid uses UTC only for now, DruidStorageHandler should set a version 
of segment to UTC.

 

 

 

  was:
h1. Exception while running a KillTask
{code:java}
2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
 type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused 
segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
 has version[2019-01-31T01:12:32.289+09:00] > task 
version[2019-01-30T16:58:29.992Z]
at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
 [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
 
h1. AS-IS
h3. [KillTask compares 
versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]

 
{code:java}
if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}
{code}
 

 
h3. [KillTask version (UTC, 
"{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]

 
{code:java}
version = 

[jira] [Assigned] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC

[Seung-Hyun Cheong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seung-Hyun Cheong reassigned HIVE-21194:



> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] 
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running 
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
>  type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused 
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
>  has version[2019-01-31T01:12:32.289+09:00] > task 
> version[2019-01-30T16:58:29.992Z]
>   at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) 
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
>  [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>  
> h1. AS-IS
> h3. [KillTask compares 
> versions|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
>  
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
>   throw new ISE(
>   "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
>   unusedSegment.getId(),
>   unusedSegment.getVersion(),
>   myLock.getVersion()
>   );
> }
> {code}
>  
>  
> h3. [KillTask version (UTC, 
> "{{2019-01-30T16:58:29.992Z}}{{")}}|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
>  
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>  
>  
> h3. [Segment version (UTC+9, 
> "2019-01-31T01:12:32.289+09:00")|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new 
> DateTime().toString());
> {code}
>  
>  
> h1. TO-BE
>  
> Because druid uses UTC only for now, DruidStorageHandler should set a version 
> of segment to UTC.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)