Re: help diagnosing issue

t >>HBase >> code it is thrown in a few places. Would be good to check where the >> resource crunch is occurring at. >> >> >> >> On Tue, Sep 1, 2015 at 2:26 PM, Perko, Ralph J <ralph.pe...@pnnl.gov> >> wrote: >> >>> Hi I have run into an issu

Re: Salt bucket count recommendation

:34 AM, Perko, Ralph J ralph.pe...@pnnl.gov wrote: Hi – following up on this. Is it generally recommended to roughly match the salt bucket count to region server count? Or is it more arbitrary? Should I use something like 255 because the regions are going to split anyway? Thanks, Ralph

Re: Salt bucket count recommendation

Hi – following up on this. Is it generally recommended to roughly match the salt bucket count to region server count? Or is it more arbitrary? Should I use something like 255 because the regions are going to split anyway? Thanks, Ralph From: Perko, Ralph J Reply-To: user

Salt bucket count recommendation

Hi, We have a 40 node cluster with 8 core tables and around 35 secondary index tables. The tables get very large – billions of records and terabytes of data. What salt bucket count do you recommend? Thanks, Ralph

Re: Phoenix version

. Thanks, James [1] http://phoenix.apache.org/pherf.html On Tue, Apr 7, 2015 at 3:23 PM, Perko, Ralph J ralph.pe...@pnnl.gov wrote: Hi, What is the recommended Phoenix version for HBase 0.98.4? Should I downgrade to 4.2.2? Thanks, Ralph

Re: indexed query question

not use PK skip scan according to its semantics. A query similar to the above example with OR logic would only run slower than one simply doing a full-scan on the main data table (without using the index table at all). Thanks, Maryann On Mon, Jan 19, 2015 at 1:25 PM, Perko, Ralph J ralph.pe

Re: hbase / phoenix errors

https://issues.apache.org/jira/browse/PHOENIX-1634. Rajeshbabu - would be interesting to get your opinion on this too. On Tue, Apr 7, 2015 at 1:19 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: Based on the Phoenix compatibility chart at the download page I did not expect

Re: hbase / phoenix errors

HDP-2.3 (the HBase/Phoenix code both supports/uses the 0 argument getCatalogTracker). From: Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov Sent: Tuesday, April 07, 2015 10:28 AM To: user@phoenix.apache.orgmailto:user@phoenix.apache.org Subject: RE

RE: hbase / phoenix errors

log lines around the time of a regionserver's crash (look for exceptions etc around that time in the regionserver logs). Thanks Devaraj On Apr 6, 2015, at 3:00 PM, Perko, Ralph J ralph.pe...@pnnl.govjavascript:_e(%7B%7D,'cvml','ralph.pe...@pnnl.gov'); wrote: Hi, we recently upgraded to Phoenix

RE: hbase / phoenix errors

this discussion to the hbase user group? Thanks, Ralph From: Perko, Ralph J Sent: Tuesday, April 07, 2015 10:28 AM To: user@phoenix.apache.org Subject: RE: hbase / phoenix errors Thank you for the response I am using Phoenix 4.3 as a separate installation. Unfortunately I have no way to copy

RE: hbase / phoenix errors

argument, but in Phoenix versions from the open source, it works with 0 arguments. This has been taken care of in the-yet-to-be-released HDP-2.3 (the HBase/Phoenix code both supports/uses the 0 argument getCatalogTracker). From: Perko, Ralph J ralph.pe

hbase / phoenix errors

Hi, we recently upgraded to Phoenix 4.3 and Hortonworks 2.2 (HBase .98.4) and we are running into some issues. I am wondering if I am missing something easy and hoping you can help. I have 34 regions servers and many keep crashing but without much in the way of error messages. Here are the

RE: bulk loader MR counters

version of Phoenix were you using before? Are you running with exactly the same data and table definitions as before, or have those changed by any chance? - Gabriel On Thu, Apr 2, 2015 at 11:43 PM Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: Thanks - I will try your

RE: bulk loader MR counters

sorting map output . Regards Ravi From: Perko, Ralph J Sent: Thursday, April 02, 2015 2:36 PM To: user@phoenix.apache.org Subject: RE: bulk loader MR counters My apologies, the formatting did not come out as planned. Here is another go: Hi, we recently upgraded our cluster (Phoenix 4.3 - HDP

bulk load issue

Hi, I recently ran into a new issue with the csv bulk loader. The MapReduce jobs run fine but then the hbase loading portion seems to get stuck in a cycle of RpcRetryCaller cycles on the index tables. Sample output – there are many of these for all index tables 15/03/30 09:55:21 INFO

Re: TTL

-11 12:29 GMT-05:00 Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov: Hi – I attempted to set the TTL parameter using ALTER TABLE through phoenix and received error 1025: Unsupported property set in ALTER TABLE command I was, however, able to set this directly through the hbase shell

Re: Pig vs Bulk Load record count

at 3:29 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: I have solved the problem. This was a mystery because the same data loaded into the same schema gave conflicting counts depending on the load technique. While the data itself had no duplicate keys the behavior

Re: Pig vs Bulk Load record count

: Tuesday, February 03, 2015 2:42 AM To: user@phoenix.apache.orgmailto:user@phoenix.apache.org Subject: Re: Pig vs Bulk Load record count Thanks Ralph. I will try to reproduce this on my end with a sample data set and get back to you. Regards Ravi On Mon, Feb 2, 2015 at 5:27 PM, Perko, Ralph J

RE: Pig vs Bulk Load record count

Load record count Hi Ralph, Is it possible to share the CREATE TABLE command as I would like to reproduce the error on my side with a sample dataset with the specific data types of yours. Regards Ravi On Mon, Feb 2, 2015 at 1:29 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe

Pig vs Bulk Load record count

Hi, I’ve run into a peculiar issue between loading data using Pig vs the CsvBulkLoadTool. I have 42M csv records to load and I am comparing the performance. In both cases the MR jobs are successful, and there are no errors. In both cases the MR job counters state there are 42M Map input and

Re: Pig vs Bulk Load record count

with an entry Phoenix Generic Upsert Statement: .. Also, what do the MapReduce counters say for the job. If possible can you share the pig script as sometimes the order of columns in the STORE command impacts. Regards Ravi On Mon, Feb 2, 2015 at 10:46 AM, Perko, Ralph J ralph.pe

Re: Query performance question

). Will try to work out a solution ASAP. On Mon, Dec 15, 2014 at 3:38 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: I am using version 4.2.2 From: Perko, Ralph Perko ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov Reply-To: user@phoenix.apache.orgmailto:user

Re: Query performance question

to run a full-scan join query on BULK_TABLE, like select * from BULK_TABLE join (select file_id, recnum from BULK_TABLE) as SS on BULK_TABLE.file_id = SS.file_id and BULK_TABLE.recnum = SS.recnum? Thanks, Maryann On Thu, Dec 11, 2014 at 6:28 PM, Perko, Ralph J ralph.pe

Re: Query performance question

, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: My apologies for the delay. I had to switch clusters and use a smaller dataset. The discrepancy still exists but the numbers are a little different: I ran the same queries as in the original email (below) Total records: 581M

Re: Query performance question

of your table and index so that I can try reproducing the problem? Thanks, Maryann On Mon, Dec 15, 2014 at 2:46 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: My apologies for the delay. I had to switch clusters and use a smaller dataset. The discrepancy still exists

Re: Phoenix loading via psql.py - specifying tab separator

I have encountered the same issue with the CsvBulkLoadTool and used the same workaround On 12/12/14, 11:53 AM, Gabriel Reid gabriel.r...@gmail.com wrote: On Fri, Dec 12, 2014 at 1:48 AM, James Taylor jamestay...@apache.org wrote: Thanks for that somewhat tricky workaround, Gabriel. Might be

Query performance question

Hi, Thanks for all your help thus far with Phoenix. I am trying to understand the best way to construct a query that returns all the fields from a table but still takes advantage of a single field secondary index. I have a table with upwards of 50 fields and do not wish to index them all

Re: pig and phoenix

@phoenix.apache.orgmailto:user@phoenix.apache.org Subject: Re: pig and phoenix Hi Ralph Glad that worked partly. For the issue that you are mentioning I am not sure of any easy way out as there could be some rows with null column values Regards Ravi Magham. On Monday, December 8, 2014, Perko, Ralph J

Re: pig and phoenix

as there could be some rows with null column values Regards Ravi Magham. On Monday, December 8, 2014, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: Ravi, Your suggestion worked – thank you! But I am now getting a org.apache.phoenix.schema.ConstraintViolationException on some

pig and phoenix

Hi, I wrote a series of pig scripts to load data that were working well with 4.0, but since upgrading to 4.2.x (4.2.1 currently) are now failing. Here is an example: Table def: CREATE TABLE IF NOT EXISTS t1_log_dns ( period BIGINT NOT NULL, deployment VARCHAR NOT NULL, file_id VARCHAR

Re: RegionTooBusyException

), than play with compaction config values (above). -Vladimir Rodionov On Thu, Nov 6, 2014 at 12:31 PM, Perko, Ralph J ralph.pe...@pnnl.govmailto:ralph.pe...@pnnl.gov wrote: Hi, I am using a combination of Pig, Phoenix and HBase to load data on a test cluster and I continue to run into an issue

RegionTooBusyException

Hi, I am using a combination of Pig, Phoenix and HBase to load data on a test cluster and I continue to run into an issue with larger, longer running jobs (smaller jobs succeed). After the job has run for several hours, the first set of mappers have finished and the second begin, the job dies

data ingestion

Hi, What is the best way to ingest large amounts of csv data coming in at regular intervals (about every 15min for a total of about 500G/daily or 1.5B records/daily) that requires a few transformations before being inserted? By transformation I mean the following: 1) 1 field is converted to a