[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[
https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882309#comment-15882309
]
Hive QA commented on HIVE-15951:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853996/HIVE-15951.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10258 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3747/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3747/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3747/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853996 - PreCommit-HIVE-Build
> Make sure base persist directory is unique and deleted
> --
>
> Key: HIVE-15951
> URL: https://issues.apache.org/jira/browse/HIVE-15951
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-15951.2.patch, HIVE-15951.patch
>
>
> In some cases the base persist directory will contain old data or shared
> between reducer in the same physical VM.
> That will lead to the failure of the job till that the directory is cleaned.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881093#comment-15881093 ] Ashutosh Chauhan commented on HIVE-15951: - +1 seems like we commons-io indeed is available. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881094#comment-15881094 ] Ashutosh Chauhan commented on HIVE-15951: - [~bslim] Would you like to make it Patch Available so we get a QA run. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879410#comment-15879410 ] Ashutosh Chauhan commented on HIVE-15951: - I checked commons-io is not packaged with hive-exec so it wont be available on task nodes. All callers of FileUtils in hive-common are in front end, so it wont be an issue for it. But if we add it here, we will need commons-io jar on task nodes. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879355#comment-15879355 ] slim bouguerra commented on HIVE-15951: --- [~ashutoshc] valid point but hive common is using that method as well so i think it is ok to use it. https://github.com/b-slim/hive/blob/38ad77929980dc155dcc4a5d009a9a855eb5b017/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L755-L755 > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879280#comment-15879280 ] Ashutosh Chauhan commented on HIVE-15951: - I am not sure if we package apache-commons-io in druid handler jar. If not, we probably dont want to add a dependency on it just for this minor convenience method. Other than that, looks good. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878627#comment-15878627 ] slim bouguerra commented on HIVE-15951: --- [~ashutoshc] please checkout the new fix. The delete is done on the close call. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.2.patch, HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872832#comment-15872832 ] Ashutosh Chauhan commented on HIVE-15951: - deleteOnExit() is called when process exists. For container mode, process will terminate immediately and clean up will happen. For llap, process won't exit once RecordWriter is done, since its long running process waiting to execute next query. So, clean up wont happen until llap deamon goes down, which likely is not useful. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872796#comment-15872796 ] slim bouguerra commented on HIVE-15951: --- [~ashutoshc] i am not sure i am getting this, is it the delete on exit that won't work ? > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872651#comment-15872651 ] Ashutosh Chauhan commented on HIVE-15951: - This will work in container mode, but may not in llap mode with long running process. There we may need to keep track of lifecycle of the file. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15951) Make sure base persist directory is unique and deleted
[ https://issues.apache.org/jira/browse/HIVE-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870792#comment-15870792 ] slim bouguerra commented on HIVE-15951: --- [~ashutoshc] can you please checkout this bug. > Make sure base persist directory is unique and deleted > -- > > Key: HIVE-15951 > URL: https://issues.apache.org/jira/browse/HIVE-15951 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15951.patch > > > In some cases the base persist directory will contain old data or shared > between reducer in the same physical VM. > That will lead to the failure of the job till that the directory is cleaned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
