[jira] [Commented] (SENTRY-2310) Sentry is not be able to fetch full updates when there is HMS restart in the snapshot process.
[ https://issues.apache.org/jira/browse/SENTRY-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542499#comment-16542499 ] Arjun Mishra commented on SENTRY-2310: -- Option 1 is better. Closing and opening a connection every half a second is not feasible in my opinion > Sentry is not be able to fetch full updates when there is HMS restart in the > snapshot process. > -- > > Key: SENTRY-2310 > URL: https://issues.apache.org/jira/browse/SENTRY-2310 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > If the communication between sentry and HMS goes down for any reason while > sentry is fetching full update from HMS, SentryHMSClient in HMSFollower would > be left with a reference to closed socket. As sentry is not handling the > failure and closing the SentryHMSClient, it continues using the same > SentryHMSClient. This will result in "java.net.SocketException: Broken pipe" > as the client tries to write on socket that is closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2310) Sentry is not be able to fetch full updates when there is HMS restart in the snapshot process.
[ https://issues.apache.org/jira/browse/SENTRY-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542269#comment-16542269 ] kalyan kumar kalvagadda commented on SENTRY-2310: - There are two options here # close the SentryHMSClient when there is an exception. Drawback with this approach is that we need to make sure that this is done when ever new API's are added to SentryHMSClient. # Make sure that SentryHMSClient in HMSFollower is closed every time execution of HMSFollower is done. Over head is that sentry would creating new connection and closing it for every 500 milli sec. > Sentry is not be able to fetch full updates when there is HMS restart in the > snapshot process. > -- > > Key: SENTRY-2310 > URL: https://issues.apache.org/jira/browse/SENTRY-2310 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > If the communication between sentry and HMS goes down for any reason while > sentry is fetching full update from HMS, SentryHMSClient in HMSFollower would > be left with a reference to closed socket. As sentry is not handling the > failure and closing the SentryHMSClient, it continues using the same > SentryHMSClient. This will result in "java.net.SocketException: Broken pipe" > as the client tries to write on socket that is closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2299) NPE In Sentry HDFS Sync Plugin
[
https://issues.apache.org/jira/browse/SENTRY-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542267#comment-16542267
]
Hadoop QA commented on SENTRY-2299:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12931376/SENTRY-2299.001.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3960/console
This message is automatically generated.
> NPE In Sentry HDFS Sync Plugin
> --
>
> Key: SENTRY-2299
> URL: https://issues.apache.org/jira/browse/SENTRY-2299
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2299.001.patch
>
>
> Sentry HDFS ACL synchronization stopped working and throws
> NullPointerException. The HDFS logs showed repeating errors like the
> following:
> {code}
> 11:16:15.743 AM WARNSentryAuthorizationInfo
> Failed to update, will retry in [3]ms, error:
> java.lang.NullPointerException
> at org.apache.sentry.hdfs.HMSPaths$Entry.access$200(HMSPaths.java:146)
> at org.apache.sentry.hdfs.HMSPaths.renameAuthzObject(HMSPaths.java:879)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.applyPartialUpdate(UpdateableAuthzPaths.java:118)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.updatePartial(UpdateableAuthzPaths.java:81)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.processUpdates(SentryAuthorizationInfo.java:211)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:139)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.run(SentryAuthorizationInfo.java:232)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> The customer checked the Sentry logs and didn't see any corresponding errors.
> The issue stopped occurring, apparently not through any specific user
> intervention. (The customer tried manually failing over the active NameNode,
> with no change.)
> {code}
> Arjun mentioned the reason is that some delta update from sentry server was
> lost, so the oldEntry at line HMSPaths.java:879 was null. That caused null
> exception.
> {code}
> void renameAuthzObject(String oldName, List> oldPathElems,
> String newName, List> newPathElems) {
> if (LOG.isDebugEnabled()) {
> LOG.debug(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> }
> if (oldPathElems == null || oldPathElems.isEmpty() ||
> newPathElems == null || newPathElems.isEmpty() ||
> newName == null || newName.equals(oldName)) {
> LOG.warn(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})" +
> ": invalid inputs, skipping",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> return;
> }
> // if oldPath == newPath, that is path has not changed as part of rename
> and hence new table
> // needs to have old paths => new_table.add(old_table_partition_paths)
> List oldPathElements = oldPathElems.get(0);
> List newPathElements = newPathElems.get(0);
> if (!oldPathElements.equals(newPathElements)) {
> Entry oldEntry = root.find(oldPathElements.toArray(new String[0]),
> false);
> Entry newParent = root.createParent(newPathElements);
> oldEntry.moveTo(newParent, newPathElements.get(newPathElements.size() -
> 1)); -> oldEntry is null
> }
> {code}
> There are several possible reasons why some delta changes are lost.
> {code}
> 1. Sentry server does not save the rename update as delta update. The chance
> is really low
> 2. The delta change is lost from sentry server to name node plugin. The
> chance is also low
> 3. When applying delta change about old entry, it is lost
> {code}
> The fix for this issue
> 1. Check if oldEntry is null. If so, don't call oldEntry.moveTo. Instead,
> throw exception, which will be caught by its caller and causes sentry client
> at name node plugin gets path full snapshot from sentry server.
> 2. Find out why the oldEntry is
[jira] [Assigned] (SENTRY-2310) Sentry is not be able to fetch full updates when there is HMS restart in the snapshot process.
[ https://issues.apache.org/jira/browse/SENTRY-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda reassigned SENTRY-2310: --- Assignee: kalyan kumar kalvagadda > Sentry is not be able to fetch full updates when there is HMS restart in the > snapshot process. > -- > > Key: SENTRY-2310 > URL: https://issues.apache.org/jira/browse/SENTRY-2310 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > If the communication between sentry and HMS goes down for any reason while > sentry is fetching full update from HMS, SentryHMSClient in HMSFollower would > be left with a reference to closed socket. As sentry is not handling the > failure and closing the SentryHMSClient, it continues using the same > SentryHMSClient. This will result in "java.net.SocketException: Broken pipe" > as the client tries to write on socket that is closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2310) Sentry is not be able to fetch full updates when there is HMS restart in the snapshot process.
kalyan kumar kalvagadda created SENTRY-2310: --- Summary: Sentry is not be able to fetch full updates when there is HMS restart in the snapshot process. Key: SENTRY-2310 URL: https://issues.apache.org/jira/browse/SENTRY-2310 Project: Sentry Issue Type: Bug Reporter: kalyan kumar kalvagadda If the communication between sentry and HMS goes down for any reason while sentry is fetching full update from HMS, SentryHMSClient in HMSFollower would be left with a reference to closed socket. As sentry is not handling the failure and closing the SentryHMSClient, it continues using the same SentryHMSClient. This will result in "java.net.SocketException: Broken pipe" as the client tries to write on socket that is closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2158) Update notification handler to update privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542251#comment-16542251 ] Na Li commented on SENTRY-2158: --- Right now, we do not expose public API at Sentry server for granting or revoking user privileges. Therefore, this jira cannot be done until we expose such API, like alter_sentry_user_grant_privilege and alter_sentry_user_revoke_privilege, in SentryPolicyService. > Update notification handler to update privileges to user > > > Key: SENTRY-2158 > URL: https://issues.apache.org/jira/browse/SENTRY-2158 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > SentryPolicyStoreProcessor calls NotificationHandlerInvoker when processing > permission related commands. We should update notification handler in the > following files when granting privileges to user. > When authorizable changes, need to change user privileges too > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandler.java > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandlerInvoker.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2158) Update notification handler to update privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2158: -- Attachment: (was: SENTRY-2158.001.patch) > Update notification handler to update privileges to user > > > Key: SENTRY-2158 > URL: https://issues.apache.org/jira/browse/SENTRY-2158 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > SentryPolicyStoreProcessor calls NotificationHandlerInvoker when processing > permission related commands. We should update notification handler in the > following files when granting privileges to user. > When authorizable changes, need to change user privileges too > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandler.java > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandlerInvoker.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2158) Update notification handler to update privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2158: -- Status: In Progress (was: Patch Available) > Update notification handler to update privileges to user > > > Key: SENTRY-2158 > URL: https://issues.apache.org/jira/browse/SENTRY-2158 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > SentryPolicyStoreProcessor calls NotificationHandlerInvoker when processing > permission related commands. We should update notification handler in the > following files when granting privileges to user. > When authorizable changes, need to change user privileges too > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandler.java > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandlerInvoker.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2158) Update notification handler to update privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2158: -- Issue Type: New Feature (was: Sub-task) Parent: (was: SENTRY-2151) > Update notification handler to update privileges to user > > > Key: SENTRY-2158 > URL: https://issues.apache.org/jira/browse/SENTRY-2158 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > SentryPolicyStoreProcessor calls NotificationHandlerInvoker when processing > permission related commands. We should update notification handler in the > following files when granting privileges to user. > When authorizable changes, need to change user privileges too > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandler.java > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandlerInvoker.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2299) NPE In Sentry HDFS Sync Plugin
[
https://issues.apache.org/jira/browse/SENTRY-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2299:
--
Summary: NPE In Sentry HDFS Sync Plugin (was: NPE In Sentry HDFS Sync Plug)
> NPE In Sentry HDFS Sync Plugin
> --
>
> Key: SENTRY-2299
> URL: https://issues.apache.org/jira/browse/SENTRY-2299
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2299.001.patch
>
>
> Sentry HDFS ACL synchronization stopped working and throws
> NullPointerException. The HDFS logs showed repeating errors like the
> following:
> {code}
> 11:16:15.743 AM WARNSentryAuthorizationInfo
> Failed to update, will retry in [3]ms, error:
> java.lang.NullPointerException
> at org.apache.sentry.hdfs.HMSPaths$Entry.access$200(HMSPaths.java:146)
> at org.apache.sentry.hdfs.HMSPaths.renameAuthzObject(HMSPaths.java:879)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.applyPartialUpdate(UpdateableAuthzPaths.java:118)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.updatePartial(UpdateableAuthzPaths.java:81)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.processUpdates(SentryAuthorizationInfo.java:211)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:139)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.run(SentryAuthorizationInfo.java:232)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> The customer checked the Sentry logs and didn't see any corresponding errors.
> The issue stopped occurring, apparently not through any specific user
> intervention. (The customer tried manually failing over the active NameNode,
> with no change.)
> {code}
> Arjun mentioned the reason is that some delta update from sentry server was
> lost, so the oldEntry at line HMSPaths.java:879 was null. That caused null
> exception.
> {code}
> void renameAuthzObject(String oldName, List> oldPathElems,
> String newName, List> newPathElems) {
> if (LOG.isDebugEnabled()) {
> LOG.debug(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> }
> if (oldPathElems == null || oldPathElems.isEmpty() ||
> newPathElems == null || newPathElems.isEmpty() ||
> newName == null || newName.equals(oldName)) {
> LOG.warn(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})" +
> ": invalid inputs, skipping",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> return;
> }
> // if oldPath == newPath, that is path has not changed as part of rename
> and hence new table
> // needs to have old paths => new_table.add(old_table_partition_paths)
> List oldPathElements = oldPathElems.get(0);
> List newPathElements = newPathElems.get(0);
> if (!oldPathElements.equals(newPathElements)) {
> Entry oldEntry = root.find(oldPathElements.toArray(new String[0]),
> false);
> Entry newParent = root.createParent(newPathElements);
> oldEntry.moveTo(newParent, newPathElements.get(newPathElements.size() -
> 1)); -> oldEntry is null
> }
> {code}
> There are several possible reasons why some delta changes are lost.
> {code}
> 1. Sentry server does not save the rename update as delta update. The chance
> is really low
> 2. The delta change is lost from sentry server to name node plugin. The
> chance is also low
> 3. When applying delta change about old entry, it is lost
> {code}
> The fix for this issue
> 1. Check if oldEntry is null. If so, don't call oldEntry.moveTo. Instead,
> throw exception, which will be caught by its caller and causes sentry client
> at name node plugin gets path full snapshot from sentry server.
> 2. Find out why the oldEntry is null and fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2299) NPE In Sentry HDFS Sync Plug
[
https://issues.apache.org/jira/browse/SENTRY-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2299:
--
Status: Patch Available (was: Open)
> NPE In Sentry HDFS Sync Plug
>
>
> Key: SENTRY-2299
> URL: https://issues.apache.org/jira/browse/SENTRY-2299
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2299.001.patch
>
>
> Sentry HDFS ACL synchronization stopped working and throws
> NullPointerException. The HDFS logs showed repeating errors like the
> following:
> {code}
> 11:16:15.743 AM WARNSentryAuthorizationInfo
> Failed to update, will retry in [3]ms, error:
> java.lang.NullPointerException
> at org.apache.sentry.hdfs.HMSPaths$Entry.access$200(HMSPaths.java:146)
> at org.apache.sentry.hdfs.HMSPaths.renameAuthzObject(HMSPaths.java:879)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.applyPartialUpdate(UpdateableAuthzPaths.java:118)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.updatePartial(UpdateableAuthzPaths.java:81)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.processUpdates(SentryAuthorizationInfo.java:211)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:139)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.run(SentryAuthorizationInfo.java:232)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> The customer checked the Sentry logs and didn't see any corresponding errors.
> The issue stopped occurring, apparently not through any specific user
> intervention. (The customer tried manually failing over the active NameNode,
> with no change.)
> {code}
> Arjun mentioned the reason is that some delta update from sentry server was
> lost, so the oldEntry at line HMSPaths.java:879 was null. That caused null
> exception.
> {code}
> void renameAuthzObject(String oldName, List> oldPathElems,
> String newName, List> newPathElems) {
> if (LOG.isDebugEnabled()) {
> LOG.debug(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> }
> if (oldPathElems == null || oldPathElems.isEmpty() ||
> newPathElems == null || newPathElems.isEmpty() ||
> newName == null || newName.equals(oldName)) {
> LOG.warn(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})" +
> ": invalid inputs, skipping",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> return;
> }
> // if oldPath == newPath, that is path has not changed as part of rename
> and hence new table
> // needs to have old paths => new_table.add(old_table_partition_paths)
> List oldPathElements = oldPathElems.get(0);
> List newPathElements = newPathElems.get(0);
> if (!oldPathElements.equals(newPathElements)) {
> Entry oldEntry = root.find(oldPathElements.toArray(new String[0]),
> false);
> Entry newParent = root.createParent(newPathElements);
> oldEntry.moveTo(newParent, newPathElements.get(newPathElements.size() -
> 1)); -> oldEntry is null
> }
> {code}
> There are several possible reasons why some delta changes are lost.
> {code}
> 1. Sentry server does not save the rename update as delta update. The chance
> is really low
> 2. The delta change is lost from sentry server to name node plugin. The
> chance is also low
> 3. When applying delta change about old entry, it is lost
> {code}
> The fix for this issue
> 1. Check if oldEntry is null. If so, don't call oldEntry.moveTo. Instead,
> throw exception, which will be caught by its caller and causes sentry client
> at name node plugin gets path full snapshot from sentry server.
> 2. Find out why the oldEntry is null and fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2299) NPE In Sentry HDFS Sync Plug
[
https://issues.apache.org/jira/browse/SENTRY-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2299:
--
Attachment: SENTRY-2299.001.patch
> NPE In Sentry HDFS Sync Plug
>
>
> Key: SENTRY-2299
> URL: https://issues.apache.org/jira/browse/SENTRY-2299
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2299.001.patch
>
>
> Sentry HDFS ACL synchronization stopped working and throws
> NullPointerException. The HDFS logs showed repeating errors like the
> following:
> {code}
> 11:16:15.743 AM WARNSentryAuthorizationInfo
> Failed to update, will retry in [3]ms, error:
> java.lang.NullPointerException
> at org.apache.sentry.hdfs.HMSPaths$Entry.access$200(HMSPaths.java:146)
> at org.apache.sentry.hdfs.HMSPaths.renameAuthzObject(HMSPaths.java:879)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.applyPartialUpdate(UpdateableAuthzPaths.java:118)
> at
> org.apache.sentry.hdfs.UpdateableAuthzPaths.updatePartial(UpdateableAuthzPaths.java:81)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.processUpdates(SentryAuthorizationInfo.java:211)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:139)
> at
> org.apache.sentry.hdfs.SentryAuthorizationInfo.run(SentryAuthorizationInfo.java:232)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> The customer checked the Sentry logs and didn't see any corresponding errors.
> The issue stopped occurring, apparently not through any specific user
> intervention. (The customer tried manually failing over the active NameNode,
> with no change.)
> {code}
> Arjun mentioned the reason is that some delta update from sentry server was
> lost, so the oldEntry at line HMSPaths.java:879 was null. That caused null
> exception.
> {code}
> void renameAuthzObject(String oldName, List> oldPathElems,
> String newName, List> newPathElems) {
> if (LOG.isDebugEnabled()) {
> LOG.debug(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> }
> if (oldPathElems == null || oldPathElems.isEmpty() ||
> newPathElems == null || newPathElems.isEmpty() ||
> newName == null || newName.equals(oldName)) {
> LOG.warn(String.format("%s renameAuthzObject({%s, %s} -> {%s, %s})" +
> ": invalid inputs, skipping",
> this, oldName, assemblePaths(oldPathElems), newName,
> assemblePaths(newPathElems)));
> return;
> }
> // if oldPath == newPath, that is path has not changed as part of rename
> and hence new table
> // needs to have old paths => new_table.add(old_table_partition_paths)
> List oldPathElements = oldPathElems.get(0);
> List newPathElements = newPathElems.get(0);
> if (!oldPathElements.equals(newPathElements)) {
> Entry oldEntry = root.find(oldPathElements.toArray(new String[0]),
> false);
> Entry newParent = root.createParent(newPathElements);
> oldEntry.moveTo(newParent, newPathElements.get(newPathElements.size() -
> 1)); -> oldEntry is null
> }
> {code}
> There are several possible reasons why some delta changes are lost.
> {code}
> 1. Sentry server does not save the rename update as delta update. The chance
> is really low
> 2. The delta change is lost from sentry server to name node plugin. The
> chance is also low
> 3. When applying delta change about old entry, it is lost
> {code}
> The fix for this issue
> 1. Check if oldEntry is null. If so, don't call oldEntry.moveTo. Instead,
> throw exception, which will be caught by its caller and causes sentry client
> at name node plugin gets path full snapshot from sentry server.
> 2. Find out why the oldEntry is null and fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
