[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547880#comment-13547880 ] Hudson commented on HIVE-3709: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-3709. Stop storing default ConfVars in temp file (Kevin Wilfong via cws) (Revision 1415038) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1415038 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/LoopingByteArrayInputStream.java Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.11.0 Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506407#comment-13506407 ] Hudson commented on HIVE-3709: -- Integrated in Hive-trunk-h0.21 #1824 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1824/]) HIVE-3709. Stop storing default ConfVars in temp file (Kevin Wilfong via cws) (Revision 1415038) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1415038 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/LoopingByteArrayInputStream.java Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.11 Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[
https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505292#comment-13505292
]
Carl Steinbach commented on HIVE-3709:
--
Another option for fixing this problem is to give administrators the ability to
specify the location where the temporary file should be written (for setuid
processes running on POSIX systems the conventional location is somewhere under
/var). For example we could add a configuration property named
hive.process.local.temporary.dir and have it default to
${user.home}/.hive/${process_id}.
Stop storing default ConfVars in temp file
--
Key: HIVE-3709
URL: https://issues.apache.org/jira/browse/HIVE-3709
Project: Hive
Issue Type: Improvement
Components: Configuration
Affects Versions: 0.10.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt,
HIVE-3709.3.patch.txt
To work around issues with Hadoop's Configuration object, specifically it's
addResource(InputStream), default configurations are written to a temp file
(I think HIVE-2362 introduced this).
This, however, introduces the problem that once that file is deleted from
/tmp the client crashes. This is particularly problematic for long running
services like the metastore server.
Writing a custom InputStream to deal with the problems in the Configuration
object should provide a work around, which does not introduce a time bomb
into Hive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505455#comment-13505455 ] Chris McConnell commented on HIVE-3709: --- I was also looking into this with 3596, I was able to fix utilizing a location similar to the suggestion Carl made above, however I think that pushes the problem to another location, rather than addressing the actual issue. I like where Kevin is going with this fix, I had thought about the possibility of checking the confVarURL in the copy constructor, removing and re-creating if it did not exist, but even that would not be perfect depending upon timing. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505838#comment-13505838 ] Kevin Wilfong commented on HIVE-3709: - I had considered the solution Carl mentioned, but I didn't go that route because it could result in clutter in the directory specified, in particular we can't guarantee the file will be deleted in the presence of catastrophic failures. To solve this the user would need to set up some sort of periodic cleanup which puts us back in the same position. We might be able to work around this by regularly touching the file, but I'm not 100% sure. Carl, are you still seeing threading problems with the most recent patch? TestHiveServerSessions has been succeeding for me consistently. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506216#comment-13506216 ] Carl Steinbach commented on HIVE-3709: -- @Kevin: I tried running TestHiveServerSessions again and wasn't able to provoke the failure I saw before. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.11 Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504437#comment-13504437 ] Ashutosh Chauhan commented on HIVE-3709: Kevin, Will HADOOP-8573 fix this? Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504814#comment-13504814 ] Kevin Wilfong commented on HIVE-3709: - It looks like that fixes the issue on a single thread where it ends up reading from the same InputStream repeatedly, which is why I overrode the close method to reset the InputStream. It does not look like it will fix the multi-threaded issue. If two threads get Configuration objects constructed using the copy constructor, and hence get the same InputStream since the resources are not cloned themselves, and loadResources has not been called before the copy constructor, it looks like it could be possible that both threads call loadResources at about the same time causing the issues Carl was seeing in TestHiveServerSessions. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498291#comment-13498291 ] Kevin Wilfong commented on HIVE-3709: - Thanks Carl, I switched to caching the byte[] and returning a new InputStream wrapping that byte[]. Now those two tests pass. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498325#comment-13498325 ] Carl Steinbach commented on HIVE-3709: -- +1. Running tests. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13496751#comment-13496751 ] Ashutosh Chauhan commented on HIVE-3709: +1 for getting rid of writing temp file. I have also been hit by this. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13496763#comment-13496763 ] Kevin Wilfong commented on HIVE-3709: - https://reviews.facebook.net/D6723 Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13496810#comment-13496810 ] Carl Steinbach commented on HIVE-3709: -- +1. Will commit if tests pass. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
