[jira] [Updated] (HIVE-12043) UGI instances being used in IO elevator threads are incorrect
[ https://issues.apache.org/jira/browse/HIVE-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12043: -- Fix Version/s: llap > UGI instances being used in IO elevator threads are incorrect > - > > Key: HIVE-12043 > URL: https://issues.apache.org/jira/browse/HIVE-12043 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-12043.1.txt > > > ... which leads to FileSystem closed exceptions. > I'm not sure yet if this is a result of the threadpool being used, and UGI > not working well with threadpools, or something else. > The UGI instance which was setup - at what looks to be thread creation time - > ends up being used for several different reads, ignoring the actual UGI > passed in. At some point this changes to a new incorrect UGI. > A simple fix is to propagate the correct UGI all the way to the reader, and > that fixes the FileSystem Closed exception. Figuring out the precise reason > would be good though. > Related to HIVE-9898. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12043) UGI instances being used in IO elevator threads are incorrect
[ https://issues.apache.org/jira/browse/HIVE-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12043: -- Attachment: HIVE-12043.1.txt Patch stores the UGI at object creation time, and then uses it during execution in a thread. After this patch, the behaviour is to use a separate UGI instance for each task - with the tokens associated with the task. (That would have been the previous behaviour as well - except for the linking of the first UGI to a thread). We can look at using a single UGI for all IO-elevator work as an alternate approach, with SQL standard auth instead of storage auth. [~sershe], [~gopalv] - please review. > UGI instances being used in IO elevator threads are incorrect > - > > Key: HIVE-12043 > URL: https://issues.apache.org/jira/browse/HIVE-12043 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-12043.1.txt > > > ... which leads to FileSystem closed exceptions. > I'm not sure yet if this is a result of the threadpool being used, and UGI > not working well with threadpools, or something else. > The UGI instance which was setup - at what looks to be thread creation time - > ends up being used for several different reads, ignoring the actual UGI > passed in. At some point this changes to a new incorrect UGI. > A simple fix is to propagate the correct UGI all the way to the reader, and > that fixes the FileSystem Closed exception. Figuring out the precise reason > would be good though. > Related to HIVE-9898. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12043) UGI instances being used in IO elevator threads are incorrect
[ https://issues.apache.org/jira/browse/HIVE-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12043: -- Attachment: HIVE-12043.2.txt Renamed the method. Yes, it does work :) Without generating the Filesystem closed exception. We will likely need to add some more synchronization for cases where tasks complete wihle the IO elevator is still reading data for them. There's a good chance of the IO Elevator seing a FileSystem closed in this case - since the task closes the FS as soon as it's done. orc_llap.q passed. Committing to the LLAP branch. > UGI instances being used in IO elevator threads are incorrect > - > > Key: HIVE-12043 > URL: https://issues.apache.org/jira/browse/HIVE-12043 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-12043.1.txt, HIVE-12043.2.txt > > > ... which leads to FileSystem closed exceptions. > I'm not sure yet if this is a result of the threadpool being used, and UGI > not working well with threadpools, or something else. > The UGI instance which was setup - at what looks to be thread creation time - > ends up being used for several different reads, ignoring the actual UGI > passed in. At some point this changes to a new incorrect UGI. > A simple fix is to propagate the correct UGI all the way to the reader, and > that fixes the FileSystem Closed exception. Figuring out the precise reason > would be good though. > Related to HIVE-9898. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
