Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-07 Thread Sriram Nookala
Thanks, setting hbase.bulkload.retries.retryOnIOException to true in the configuration worked. My Hbase cluster is colocated with the Yarn cluster on EMR. On Thu, Sep 7, 2017 at 4:08 AM, Ankit Singhal wrote: > bq. This runs successfully if I split this into 2 files,

Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-07 Thread Ankit Singhal
bq. This runs successfully if I split this into 2 files, but I'd like to avoid doing that. do you run a different job for each file? if your HBase cluster is not co-located with your yarn cluster then it may be possible that copying of large HFile is timing out(this may happen due to the fewer

Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-06 Thread Ted Yu
bq. hbase.bulkload.retries.retryOnIOException is disabled. Unable to recover The above is from HBASE-17165. See if the load can pass after enabling the config. On Wed, Sep 6, 2017 at 3:11 PM, Sriram Nookala wrote: > It finally times out with these exceptions > > ed Sep

Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-06 Thread Sriram Nookala
It finally times out with these exceptions ed Sep 06 21:38:07 UTC 2017, RpcRetryingCaller{globalStartTime=1504731276347, pause=100, retries=35}, java.io.IOException: Call to ip-10-123-0-60.ec2.internal/10.123.0.60:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:

Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-06 Thread Sriram Nookala
Phoenix 4.11.0, HBase 1.3.1 This is what I get from jstack "main" #1 prio=5 os_prio=0 tid=0x7fb3d0017000 nid=0x5de7 waiting on condition [0x7fb3d75f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xf588> (a

Re: Phoenix CSV Bulk Load fails to load a large file

2017-09-06 Thread Sergey Soldatov
Do you have more details on the version of Phoenix/HBase you are using as well as how it hangs (Exceptions/messages that may help to understand the problem)? Thanks, Sergey On Wed, Sep 6, 2017 at 1:13 PM, Sriram Nookala wrote: > I'm trying to load a 3.5G file with 60

Phoenix CSV Bulk Load fails to load a large file

2017-09-06 Thread Sriram Nookala
I'm trying to load a 3.5G file with 60 million rows using CsvBulkLoadTool. It hangs while loading HFiles. This runs successfully if I split this into 2 files, but I'd like to avoid doing that. This is on Amazon EMR, is this an issue due to disk space or memory. I have a single master and 2 region