[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741636#comment-15741636 ] Szehon Ho commented on HIVE-15385: -- Sorry for late reply, glad it's figured out, thanks guys for taking care of it. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 2.2.0 > > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736071#comment-15736071 ] Sahil Takiar commented on HIVE-15385: - The test failures seem unrelated, and they are failing in other QA runs. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736061#comment-15736061 ] Hive QA commented on HIVE-15385: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12842388/HIVE-15385.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10798 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision] (batchId=150) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2525/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2525/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2525/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12842388 - PreCommit-HIVE-Build > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736021#comment-15736021 ] Sahil Takiar commented on HIVE-15385: - Thanks Ashutosh! > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736015#comment-15736015 ] Ashutosh Chauhan commented on HIVE-15385: - yeah.. that was not conscious choice to alter behavior. Sounds good to restore documented behavior. Thanks for fixing this up. +1 > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735990#comment-15735990 ] Sahil Takiar commented on HIVE-15385: - Thanks Sergio. I believe [~ashutoshc] was working on HIVE-12988, HIVE-13716, and HIVE-13933 - any chance you could comment on this JIRA. To summarize, Hive documentation claims that when {{hive.warehouse.subdir.inherit.perms}} is {{true}}, any failure to inherit permissions will not cause queries to fail, only a warning will be logged. It looks like the aforementioned JIRAs changed that by not catching exceptions thrown by {{HdfsUtils.setFullFileStatus}}, just wondering if that was intentional or not. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735908#comment-15735908 ] Sergio Peña commented on HIVE-15385: I deleted the HiveQA comment and allow retriggering the Jenkins job. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735896#comment-15735896 ] Sergio Peña commented on HIVE-15385: Thanks [~stakiar] for the explanation. I spent some time investigating the history of this permission issue, and as you mentioned, the places where the IOException is not ignored could have been by accident. Based on the history that permissions should not throw an exception and make the query to fail, I agree on just send a warning to the log inside the setFullFileStatus() instead of throwing an exception. This is confusing for people that want to use such method. +1 I don't think tests failures are related, but could you re-attach the patch to see if the others stop failing? > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733503#comment-15733503 ] Hive QA commented on HIVE-15385: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12842388/HIVE-15385.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10745 tests executed *Failed tests:* {noformat} TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=143) [vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=126) [ppd_transform.q,auto_join9.q,auto_join1.q,vector_data_types.q,input14.q,union30.q,input12.q,union_remove_22.q,vectorization_3.q,groupby1_map_nomap.q,cbo_union.q,disable_merge_for_bucketing.q,reduce_deduplicate_exclude_join.q,filter_join_breaktask2.q,join30.q] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision] (batchId=150) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=92) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2498/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2498/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2498/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12842388 - PreCommit-HIVE-Build > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733460#comment-15733460 ] Sahil Takiar commented on HIVE-15385: - I think HIVE-7015 has some info for why failure to set permissions shouldn't result in failures. Here is the failure that I am hitting: HDFS permission inheritance follows the BSD rule: when a new file is created, it inherits the group of the parent directory. There are situations in Hive when trying to inherit the group of the parent can fail, failure to set the group can occur in the following situation: 1: {{user1}} uploads a file into HDFS directory {{/test1/}} where {{/test1/}} has group {{user1}} 2: The {{/user/hive/warehouse}} directory was created by the {{hive}} user and has group {{hive}} 3: {{user1}} runs a query that loads data from {{/test1/}} and writes it to {{/user/hive/warehouse/}} 4: The problem is that when the data is moved from {{/test1/}} to {{/user/hive/warehouse/}}, it retains its original group {{user1}} 5: Hive then tries to set the group to {{hive}} in order to preserve permission inheritance, but this fails because the query is running as {{user1}}, but {{user1}} is not part of the {{hive}} group and is not allowed to change the group of a file to {{hive}} I think the {{chgrp}} logic is similar to other standard POSIX systems. I think there can be other situations too, but I'm not an expert in this area so I would have to play around with it some more. I've confirmed that everywhere this method is used inherit perms is true, except for one place: {{DDLTask.truncateTable}} (which looks like it should be checking if inherit perms is true). Most of the invocations of {{setFullFileStatus}} are wrapped by a try-catch block which just logs the exception. There are only a few place where they are propagated up to the caller, these were all introduced by HIVE-12988, HIVE-13716, and HIVE-13933 - seemingly by accident. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733279#comment-15733279 ] Sergio Peña commented on HIVE-15385: What are the reasons of a inherit permission failure? Why do we want to continue the query if we cannot set the correct permissions? [~szehon] You wrote the permission wiki page, any info why this case? The solution on the RB is good and simple for how we want to fix it. But, I don't know if HdfsUtils.setFullFileStatus() may be used by other code parts where we really want to get a failure in case a permission cannot be set. I noticed the method throws an IOException when recursion is used too. Do we want to remove that exception? > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733011#comment-15733011 ] Sahil Takiar commented on HIVE-15385: - [~spena] can you review? RB link: https://reviews.apache.org/r/54541/ > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch, HIVE-15385.2.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732857#comment-15732857 ] Hive QA commented on HIVE-15385: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12842306/HIVE-15385.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10788 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision] (batchId=150) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=92) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2492/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2492/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2492/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12842306 - PreCommit-HIVE-Build > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15385.1.patch > > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731059#comment-15731059 ] Sahil Takiar commented on HIVE-15385: - HIVE-7015 changed the logic so that failures to inherit permissions don't fail queries. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731056#comment-15731056 ] Sahil Takiar commented on HIVE-15385: - There are also a few places throughout {{Hive.java}} where exceptions thrown by {{setFullFileStatus}} are swallowed, but there also a bunch of places where exceptions are propagated to the caller. > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[ https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730435#comment-15730435 ] Sahil Takiar commented on HIVE-15385: - [~spena] does my explanation of the situation make sense? Any comments on whether or not failure to inherit permissions should cause query failures or not? > Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., > false) causes queries to fail > -- > > Key: HIVE-15385 > URL: https://issues.apache.org/jira/browse/HIVE-15385 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > According to > https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive, > failure to inherit permissions should not cause queries to fail. > It looks like this was the case until HIVE-13716, which added some code to > use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set > permissions instead of shelling out and running {{-chgrp -R ...}}. > When shelling out, the return status of each command is ignored, so if there > are any failures when inheriting permissions, a warning is logged, but the > query still succeeds. > However, when invoked the {{FileSystem}} API, any failures will be propagated > up to the caller, and the query will fail. > This is problematic because {{setFulFileStatus}} shells out when the > {{recursive}} parameter is set to {{true}}, and when it is false it invokes > the {{FileSystem}} API. So the behavior is inconsistent depending on the > value of {{recursive}}. > We should decide whether or not permission inheritance should fail queries or > not, and then ensure the code consistently follows that decision. -- This message was sent by Atlassian JIRA (v6.3.4#6332)