Re: Discuss: End of static, thread local
HIVE-4226 is a very welcome patch but in the long run I think we want to move away from using static thread local everywhere which that patch uses. Rightfully so at this juncture. Meaning I think that HIVE-4226 is a good interim solution but long term I think we want to be passing around some kind of context object as Edward suggests in his original mail. This will make the code cleaner and more testable. This is a big effort and will require some analysis and planning before execution. On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 navis@nexr.com wrote: https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this. 2013/8/11 Brock Noland br...@cloudera.com: I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Discuss: End of static, thread local
+1 ! That's should be the right approach IMO too. thanks Prasad On Sun, Aug 11, 2013 at 10:03 AM, Brock Noland br...@cloudera.com wrote: HIVE-4226 is a very welcome patch but in the long run I think we want to move away from using static thread local everywhere which that patch uses. Rightfully so at this juncture. Meaning I think that HIVE-4226 is a good interim solution but long term I think we want to be passing around some kind of context object as Edward suggests in his original mail. This will make the code cleaner and more testable. This is a big effort and will require some analysis and planning before execution. On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 navis@nexr.com wrote: https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this. 2013/8/11 Brock Noland br...@cloudera.com: I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Discuss: End of static, thread local
+1 That's definitely the right way forward. These thread local variables should be verboten. I also agree with Brock's assessment that this will need a design doc first. My guess is that this will be quite a bit of work. Thanks, Gunther. On Sun, Aug 11, 2013 at 12:34 PM, Prasad Mujumdar pras...@cloudera.comwrote: +1 ! That's should be the right approach IMO too. thanks Prasad On Sun, Aug 11, 2013 at 10:03 AM, Brock Noland br...@cloudera.com wrote: HIVE-4226 is a very welcome patch but in the long run I think we want to move away from using static thread local everywhere which that patch uses. Rightfully so at this juncture. Meaning I think that HIVE-4226 is a good interim solution but long term I think we want to be passing around some kind of context object as Edward suggests in his original mail. This will make the code cleaner and more testable. This is a big effort and will require some analysis and planning before execution. On Sat, Aug 10, 2013 at 11:25 PM, Navis류승우 navis@nexr.com wrote: https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this. 2013/8/11 Brock Noland br...@cloudera.com: I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Discuss: End of static, thread local
I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment.
Re: Discuss: End of static, thread local
I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.comwrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Discuss: End of static, thread local
https://issues.apache.org/jira/browse/HIVE-4226 seemed addressing this. 2013/8/11 Brock Noland br...@cloudera.com: I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.comwrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
