[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326086#comment-15326086 ] Pengcheng Xiong commented on HIVE-13662: [~ashutoshc], the test case failures are either not related or not reproducible. Thanks. > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch, HIVE-13662.02.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[
https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322844#comment-15322844
]
Hive QA commented on HIVE-13662:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809056/HIVE-13662.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 10188 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-cbo_windowing.q-tez_join.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_dynamic_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_coalesce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_round_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_elt
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_orderby_5
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/62/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/62/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-62/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809056 - PreCommit-HIVE-MASTER-Build
> Set file permission and ACL in file sink operator
> -
>
> Key: HIVE-13662
> URL: https://issues.apache.org/jira/browse/HIVE-13662
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13662.01.patch, HIVE-13662.02.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[
https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321043#comment-15321043
]
Pengcheng Xiong commented on HIVE-13662:
More info here
{code}
FileStatus{path=hdfs://localhost:63064/base/warehouse/singledynamicpart/.hive-staging_hive_2016-06-07_23-09-19_277_7503414655019469213-1/_task_tmp.-ext-10002/part1=1/_tmp.00_0;
isDirectory=false; length=0; replication=3; blocksize=134217728;
modification_time=1465386667125; access_time=1465366204678; owner=pxiong;
group=supergroup; permission=rwxr-xr-x; isSymlink=false}
{code}
> Set file permission and ACL in file sink operator
> -
>
> Key: HIVE-13662
> URL: https://issues.apache.org/jira/browse/HIVE-13662
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13662.01.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321019#comment-15321019 ] Pengcheng Xiong commented on HIVE-13662: Here is what they say "When the new create(path, permission, …) method (with the permission parameter P) is used, the mode of the new file is P & ^umask & 0666." In my case, P=777, umask=022, the result should be 0644. But it is 0655. Even if i got umask wrong, there is no way that we can get the last bit x in rwxr-xr-x with & 0666 > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320084#comment-15320084 ] Ashutosh Chauhan commented on HIVE-13662: - sounds like your need to & apply reverse umask to get desired behavior : https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Changes_to_the_File_System_API > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[
https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320074#comment-15320074
]
Pengcheng Xiong commented on HIVE-13662:
Hi [~ashutoshc], I found a bigger problem. It sounds like that when we call
{code}
HiveFileFormatUtils.getHiveRecordWriter(jc, conf.getTableInfo(),
outputClass, conf, fsp.outPaths[filesIdx], permission, reporter);
{code}
in case of HiveIgnoreKeyTextOutputFormat, for example,
{code}
createOutStream = Utilities.createCompressedStream(
jc,
fs.create(outPath, permission, true,
fs.getConf().getInt("io.file.buffer.size", 4096),
fs.getDefaultReplication(outPath),
fs.getDefaultBlockSize(outPath), progress),
isCompressed);
{code}
if {code} permission{code} is rwxrwxrwx, the outPath is still rwxr-xr-x. It
sounds like this fs.create with permission does not work
> Set file permission and ACL in file sink operator
> -
>
> Key: HIVE-13662
> URL: https://issues.apache.org/jira/browse/HIVE-13662
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13662.01.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315768#comment-15315768 ] Pengcheng Xiong commented on HIVE-13662: Hi [~ashutoshc], after some investigation, I still have some questions on this. Please correct me if I am wrong. (1) FileSinkOperator will call "HiveFileFormatUtils.getHiveRecordWriter() which calls HiveOutputFormat::getHiveRecordWriter() which will eventually call FileSystem.create() which accepts permissions as an argument." This is true, but it is for SOMEPATH/warehouse/t2/.hive-staging_hive_2016-06-04_12-08-56_977_8632631213427778083-1/_task_tmp.-ext-1/p1=5/_tmp.00_0, for example. This file includes the taskID, i.e., _tmp.00_0. (2) However, in Hive.java, when we move file, we do not care about the permission of _tmp.00_0, we care about the permission of /p1=5. If /p1=5 is there, we need to use its existing permission, otherwise we need to inherit from its parents. Thus, if we would like to set file permission and ACL in file sink operator (the goal of this jira), we need to create the directory /p1=5 with a correct permission, rather than to call FileSystem.create() with permissions. > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[
https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288558#comment-15288558
]
Hive QA commented on HIVE-13662:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804262/HIVE-13662.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 134 failed/errored test(s), 9901 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_16.q-skewjoin.q-vectorization_div0.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7.q-auto_join17.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into_with_schema
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_merge2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats12
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285379#comment-15285379 ] Ashutosh Chauhan commented on HIVE-13662: - FileSinkOperator uses HiveFileFormatUtils.getHiveRecordWriter() which calls HiveOutputFormat::getHiveRecordWriter() which will eventually call FileSystem.create() which accepts permissions as an argument. That is where we need to pass in permissions so that we have permissions set at the time of file creation itself, so that we can avoid two calls on Namenode. > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282534#comment-15282534 ] Rui Li commented on HIVE-13662: --- Sure [~pxiong], thanks for working on it. > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281997#comment-15281997 ] Pengcheng Xiong commented on HIVE-13662: assigned to myself as per [~ashutoshc]'s request. :) > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
