Re: Using UserGroupInformation in multithread process

2015-06-19 Thread Zhijie Shen
Do you mean TGT tickets or tokens? Anyway, they should be across threads. Did you check if you're using the same UGI object in different threads? Thanks, Zhijie From: Gaurav Gupta gaurav.gopi...@gmail.com Sent: Thursday, June 18, 2015 11:35 PM To:

How to monitor what hdfs block is served to a client?

2015-06-19 Thread Shiyao Ma
Hi. How to monitor the block transmission log of datanodes? A more detailed example: My hdfs block size is 128MB. I have a file stored on hdfs with size 167.08MB. Also, I have a client, requesting the whole file with three splits, e.g., hdfs://myserver:9000/myfile:0+58397994 (0-56MB)

Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp

2015-06-19 Thread max scalf
Not to hijack this post but how would you deal with data that is maintained by hive(Orc format file, hive created tables etc..)...Would we copy the hivemetastore(MySQL) and move that over to new cluster? On Friday, June 19, 2015, Joep Rottinghuis jrottingh...@gmail.com wrote: You can't set up a

Using UserGroupInformation in multithread process

2015-06-19 Thread Gaurav Gupta
I am using UserGroupInformation to get the Kerberos tokens. I have a process in a Yarn container that is spawning another thread (slave). I am renewing the Kerberos Tokens in master thread but the slave thread is still using older Tokens. Are tokens not shared across threads in same JVM? Thanks

Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp

2015-06-19 Thread Nitin Pawar
yes On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot divya.htco...@gmail.com wrote: In thats It will be like three step process . 1. first cluster (secure zone) HDFS - copytoLocal - user local file system 2. user local space - copy data - second cluster user local file system 3. second

Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp

2015-06-19 Thread Joep Rottinghuis
You can't set up a proxy ? You probably want to avoid writing to local file system because aside from that being slow, it limits the size of your file to the free space on your local disc. If you do need to go commando and go through a single client machine that can see both clusters you

Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp

2015-06-19 Thread Divya Gehlot
In thats It will be like three step process . 1. first cluster (secure zone) HDFS - copytoLocal - user local file system 2. user local space - copy data - second cluster user local file system 3. second cluster user local file system - copyfromlocal - second clusterHDFS Am I on the right track ?