yikuanlee commented on issue #5817:
URL: https://github.com/apache/iceberg/issues/5817#issuecomment-1255364692
Also, Hive metastore reported that iceberg table report
AllRequestedHiveResources={db2iuser/t32_iceberg; null://null; }
not sure why iceberg won't return full hdfs directory but null://null
instead.
```
2022-09-22 17:30:48,009 DEBUG org.apache.ranger.perf.resourcetrie.retrieval:
[pool-5-thread-17]: [PERF]
RangerPolicyRepository.getLikelyMatchEvaluators(resource=null://null): 0
2022-09-22 17:30:48,009 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyRepository:
[pool-5-thread-17]: <==
RangerPolicyRepository.getLikelyMatchPolicyEvaluators(null://null):
evaluatorCount=0
2022-09-22 17:30:48,009 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyRepository:
[pool-5-thread-17]: ==> RangerPolicyRepository.storeAuditEnabledInCache()
2022-09-22 17:30:48,010 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyRepository:
[pool-5-thread-17]: <== RangerPolicyRepository.storeAuditEnabledInCache()
2022-09-22 17:30:48,010 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.evaluatePoliciesNoAudit(RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } }, policyType
=0, zoneName=null): RangerAccessResult={isAccessDetermined={true}
isAllowed={false} isAuditedDetermined={false} isAudited={false}
auditLogId={null} policyType={0} policyId={-1} zoneName={null} auditPolicy
Id={-1} policyVersion={null} evaluatedPoliciesCount={0} reason={null}
additionalInfo={}}
2022-09-22 17:30:48,010 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.zoneAwareAccessEvaluationWithNoAudit(RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } }, policyType
=0): RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId
={-1} policyVersion={null} evaluatedPoliciesCount={0} reason={null}
additionalInfo={}}
2022-09-22 17:30:48,011 DEBUG org.apache.ranger.perf.policyengine.request:
[pool-5-thread-17]: [PERF]
RangerPolicyEngine.evaluatePolicies(requestHashCode=53a8da6e_0): 4
2022-09-22 17:30:48,011 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.evaluatePolicies(RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } },
policyType=0): RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId={-1} policyVersion={
null} evaluatedPoliciesCount={0} reason={null} additionalInfo={}}
2022-09-22 17:30:48,011 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: ==>
RangerPolicyEngineImpl.evaluateAuditPolicies(result=RangerAccessResult={isAccessDetermined={true}
isAllowed={false} isAuditedDetermined={false} isAudited={false}
auditLogId={null} policyType={0} policyId={-1} zoneName={null}
auditPolicyId={-1} policyVersion={null} evaluatedPoliciesCount={0}
reason={null} additionalInfo={}})
2022-09-22 17:30:48,011 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: ==>
RangerPolicyEngineImpl.evaluateTagAuditPolicies(request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } },
result=RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId={-1} policyV
ersion={null} evaluatedPoliciesCount={0} reason={null} additionalInfo={}})
2022-09-22 17:30:48,012 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.evaluateTagAuditPolicies(request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } },
result=RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId={-1} policyV
ersion={null} evaluatedPoliciesCount={0} reason={null} additionalInfo={}})
2022-09-22 17:30:48,012 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: ==>
RangerPolicyEngineImpl.evaluateResourceAuditPolicies(request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } },
result=RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId={-1} po
licyVersion={null} evaluatedPoliciesCount={0} reason={null} additionalInfo={}})
2022-09-22 17:30:48,012 DEBUG org.apache.ranger.perf.resourcetrie.retrieval:
[pool-5-thread-17]: [PERF]
RangerPolicyRepository.getLikelyMatchEvaluators(resource=null://null): 0
2022-09-22 17:30:48,012 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyRepository:
[pool-5-thread-17]: <==
RangerPolicyRepository.getLikelyMatchPolicyEvaluators(null://null):
evaluatorCount=0
2022-09-22 17:30:48,012 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.evaluateResourceAuditPolicies(request=RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={db2iuser}
elements={} }} accessType={rwstorage} user={db2iuser} userGroups={hive
db2iuser } userRoles={} accessTime={Thu Sep 22 17:30:47 UTC 2022}
clientIPAddress={10.89.0.3} forwardedAddresses={} remoteIPAddress={10.89.0.3}
clientType={HIVEMETASTORE} action={ALTERTABLE_ADDCOLS} requestData={alter table
t32_iceberg} sessionId={HiveMetaStore} resourceMatchingScope={SELF}
clusterName={DV} clusterType={}
context={REQUESTED_RESOURCES={AllRequestedHiveResources={db2iuser/t32_iceberg;
null://null; } } token:OWNER={db2iuser} token:USER={db2iuser} } },
result=RangerAccessResult={isAccessDetermined={true} isAllowed={false}
isAuditedDetermined={false} isAudited={false} auditLogId={null} policyType={0}
policyId={-1} zoneName={null} auditPolicyId={-1} po
licyVersion={null} evaluatedPoliciesCount={0} reason={null}
additionalInfo={}}): ret=false
2022-09-22 17:30:48,012 DEBUG
org.apache.ranger.plugin.policyengine.RangerPolicyEngineImpl:
[pool-5-thread-17]: <==
RangerPolicyEngineImpl.evaluateAuditPolicies(result=RangerAccessResult={isAccessDetermined={true}
isAllowed={false} isAuditedDetermined={false} isAudited={false}
auditLogId={null} policyType={0} policyId={-1} zoneName={null}
auditPolicyId={-1} policyVersion={null} evaluatedPoliciesCount={0}
reason={null} additionalInfo={}})
2022-09-22 17:30:48,039 DEBUG
org.apache.ranger.plugin.audit.RangerDefaultAuditHandler: [pool-5-thread-17]:
==>
RangerDefaultAuditHandler.logAuthzAudit(AuthzAuditEvent{repositoryType=3;repositoryName=cm_hive;user=db2iuser;eventTime=Thu
Sep 22 17:30:47 UTC
2022;accessType=ALTER;resourcePath=db2iuser/t32_iceberg;resourceType=@table;action=alter;accessResult=1;agentId=hiveMetastore;policyId=8;resultReason=null;aclEnforcer=ranger-acl;sessionId=HiveMetaStore;clientType=HIVEMETASTORE;clientIP=10.89.0.3;requestData=alter
table
t32_iceberg;agentHostname=cms.dv.net;logType=RangerAudit;eventId=ab214aac-e979-4888-8052-1a243f638e07-0;seq_num=0;event_count=1;event_dur_ms=0;tags=[];clusterName=DV;zoneName=null;policyVersion=5;additionalInfo={"remote-ip-address":10.89.0.3,
"forwarded-ip-addresses":[]})
2022-09-22 17:30:48,039 DEBUG
org.apache.ranger.plugin.audit.RangerDefaultAuditHandler: [pool-5-thread-17]:
<==
RangerDefaultAuditHandler.logAuthzAudit(AuthzAuditEvent{repositoryType=3;repositoryName=cm_hive;user=db2iuser;eventTime=Thu
Sep 22 17:30:47 UTC
2022;accessType=ALTER;resourcePath=db2iuser/t32_iceberg;resourceType=@table;action=alter;accessResult=1;agentId=hiveMetastore;policyId=8;resultReason=null;aclEnforcer=ranger-acl;sessionId=HiveMetaStore;clientType=HIVEMETASTORE;clientIP=10.89.0.3;requestData=alter
table
t32_iceberg;agentHostname=cms.dv.net;logType=RangerAudit;eventId=ab214aac-e979-4888-8052-1a243f638e07-0;seq_num=1;event_count=1;event_dur_ms=0;tags=[];clusterName=DV;zoneName=null;policyVersion=5;additionalInfo={"remote-ip-address":10.89.0.3,
"forwarded-ip-addresses":[]})
2022-09-22 17:30:48,040 DEBUG org.apache.ranger.perf.hiveauth.request:
[pool-5-thread-17]: [PERF]
RangerHiveAuthorizer.checkPrivileges(hiveOpType=ALTERTABLE_ADDCOLS): 55
2022-09-22 17:30:48,042 ERROR
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer:
[pool-5-thread-17]: HiveMetaStoreAuthorizer.onEvent(): failed
org.apache.hadoop.hive.metastore.api.MetaException: Permission denied: user
[db2iuser] does not have [RWSTORAGE] privilege on [null://null]
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.checkPrivileges(HiveMetaStoreAuthorizer.java:532)
~[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:106)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:3979)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:5879)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_req(HiveMetaStore.java:5825)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_232]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_232]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at com.sun.proxy.$Proxy28.alter_table_req(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_req.getResult(ThriftHiveMetastore.java:17439)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_req.getResult(ThriftHiveMetastore.java:17423)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_232]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_232]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
