date:20170620


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057009#comment-16057009
 ] 

vishal edited comment on PHOENIX-3962 at 6/21/17 5:38 AM:
--

[~an...@apache.org] Hi,It is in stand alone mode and I am creating only one 
connection object.
It is throwing error that Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }
Only problem is with SYSTEM.MUTEX while other tables from SYSTEM namespace are 
getting enabled and moved to system namespace.
Please tell me the solution.
After deleting snapshot I am getting error that SYSTEM.MUTEX is disabled.
Then again I enabled it and run the program then again same error failed to 
move directory.


was (Author: vishalb.e...@gmail.com):
[~an...@apache.org] Hi,It is in stand alone mode and I am creating only one 
connection object.
It is throwing error that Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }
Only problem is with SYSTEM.MUTEX while other tables from SYSTEM namespace are 
getting enabled and moved to system namespace.
Please tell me the solution.

> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3962) Not creating the table in custom schema


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057009#comment-16057009
 ] 

vishal commented on PHOENIX-3962:
-

[~an...@apache.org] Hi,It is in stand alone mode and I am creating only one 
connection object.
It is throwing error that Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }.
Only problem is with SYSTEM.MUTEX while other tables from SYSTEM namespace are 
getting enabled and moved to system namespace.
Please tell me the solution.

> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3962) Not creating the table in custom schema


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057009#comment-16057009
 ] 

vishal edited comment on PHOENIX-3962 at 6/21/17 5:21 AM:
--

[~an...@apache.org] Hi,It is in stand alone mode and I am creating only one 
connection object.
It is throwing error that Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }
Only problem is with SYSTEM.MUTEX while other tables from SYSTEM namespace are 
getting enabled and moved to system namespace.
Please tell me the solution.


was (Author: vishalb.e...@gmail.com):
[~an...@apache.org] Hi,It is in stand alone mode and I am creating only one 
connection object.
It is throwing error that Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }.
Only problem is with SYSTEM.MUTEX while other tables from SYSTEM namespace are 
getting enabled and moved to system namespace.
Please tell me the solution.

> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vishal updated PHOENIX-3966:

Description: 
While creating schema or table I am getting this error:
Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
table=SYSTEM.MUTEX type=DISABLED }
So because of this error it is not creating the schema or table.

java-program:
--
{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = 
DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
statement = connection.createStatement();
statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
connection.commit();
{code}

hdfs-site.xml

{code:java}
 
phoenix.schema.isNamespaceMappingEnabled
true
   

phoenix.schema.mapSystemTablesToNamespace
true
   
{code}

please help me.

  was:
While creating schema or table I am getting this error:
Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
table=SYSTEM.MUTEX type=DISABLED }
So because of this error it is not creating the schema or table.

java-program:
--
{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = 
DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
statement = connection.createStatement();
statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
{code}

hdfs-site.xml

{code:java}
 
phoenix.schema.isNamespaceMappingEnabled
true
   

phoenix.schema.mapSystemTablesToNamespace
true
   
{code}

please help me.


> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3966
> URL: https://issues.apache.org/jira/browse/PHOENIX-3966
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> connection.commit();
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3944) ReadOnlyTableException occurs when we map Phoenix view to an existing HBase table with Namespace Mapping enabled


[ 
https://issues.apache.org/jira/browse/PHOENIX-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056998#comment-16056998
 ] 

ASF GitHub Bot commented on PHOENIX-3944:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154068
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/FromCompiler.java ---
@@ -196,7 +196,8 @@ public static ColumnResolver 
getResolverForCreation(final CreateTableStatement s
 if (htable != null) Closeables.closeQuietly(htable);
 }
 tableNode = NamedTableNode.create(null, baseTable, 
statement.getColumnDefs());
-return new SingleTableColumnResolver(connection, 
tableNode, e.getTimeStamp(), new HashMap(1), false);
+boolean isNamespaceMapped = 
SchemaUtil.isNamespaceMappingEnabled(statement.getTableType(), 
connection.getQueryServices().getProps());
--- End diff --


To handle view mapped \"NS.TBL\"(without schema)

```
-byte[] fullTableName = SchemaUtil.getPhysicalName(
-
SchemaUtil.getTableNameAsBytes(baseTable.getSchemaName(), 
baseTable.getTableName()),
-
connection.getQueryServices().getProps()).getName();
+byte[] fullTableName = SchemaUtil
+
.getPhysicalName(Bytes.toBytes(baseTable.toString()),
+connection.getQueryServices().getProps())
+.getName();
```



> ReadOnlyTableException occurs when we map Phoenix view to an existing HBase 
> table with Namespace Mapping enabled
> 
>
> Key: PHOENIX-3944
> URL: https://issues.apache.org/jira/browse/PHOENIX-3944
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>
> Firstly, I created a namespace and a table in hbase shell:
> {code}
> hbase> create_namespace "NS"
> hbase> create "NS:TBL", "CF"
> {code}
> After that, I tried to create a phoenix view for this existing hbase table in 
> phoenix-sqlline:
> {code}
> phoenix> CREATE VIEW ns.tbl (pk VARCHAR PRIMARY KEY, cf.col VARCHAR);
> {code}
> However, I encountered the following error and I was not able to create the 
> view:
> {code}
> org.apache.phoenix.schema.ReadOnlyTableException: ERROR 505 (42000): Table is 
> read only.
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1072)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1434)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2624)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1040)
>   at 
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:212)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:393)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:376)
>   at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:375)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:363)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1707)
>   at org.apache.PhoenixTest.main(PhoenixTest.java:55)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3944) ReadOnlyTableException occurs when we map Phoenix view to an existing HBase table with Namespace Mapping enabled


[ 
https://issues.apache.org/jira/browse/PHOENIX-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056997#comment-16056997
 ] 

ASF GitHub Bot commented on PHOENIX-3944:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154290
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -2135,7 +2135,7 @@ private PTable 
createTableInternal(CreateTableStatement statement, byte[][] spli
 // Upsert physical name for mapped view only if the full 
physical table name is different than the full table name
 // Otherwise, we end up with a self-referencing link and 
then cannot ever drop the view.
 if (viewType != ViewType.MAPPED
-|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getTableName(schemaName, 
tableName))) {
+|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getPhysicalName(schemaName, 
tableName))) {
--- End diff --

can you include isNamespaceMapped as table name with schema can also be in 
default namespace.
```
-|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getTableName(schemaName, 
tableName))) {
+|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getPhysicalTableName(
+SchemaUtil.getTableNameAsBytes(schemaName, 
tableName), isNamespaceMapped).getNameAsString())) {
```



> ReadOnlyTableException occurs when we map Phoenix view to an existing HBase 
> table with Namespace Mapping enabled
> 
>
> Key: PHOENIX-3944
> URL: https://issues.apache.org/jira/browse/PHOENIX-3944
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>
> Firstly, I created a namespace and a table in hbase shell:
> {code}
> hbase> create_namespace "NS"
> hbase> create "NS:TBL", "CF"
> {code}
> After that, I tried to create a phoenix view for this existing hbase table in 
> phoenix-sqlline:
> {code}
> phoenix> CREATE VIEW ns.tbl (pk VARCHAR PRIMARY KEY, cf.col VARCHAR);
> {code}
> However, I encountered the following error and I was not able to create the 
> view:
> {code}
> org.apache.phoenix.schema.ReadOnlyTableException: ERROR 505 (42000): Table is 
> read only.
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1072)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1434)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2624)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1040)
>   at 
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:212)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:393)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:376)
>   at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:375)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:363)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1707)
>   at org.apache.PhoenixTest.main(PhoenixTest.java:55)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3944) ReadOnlyTableException occurs when we map Phoenix view to an existing HBase table with Namespace Mapping enabled


[ 
https://issues.apache.org/jira/browse/PHOENIX-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056996#comment-16056996
 ] 

ASF GitHub Bot commented on PHOENIX-3944:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154332
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/SchemaUtil.java ---
@@ -302,6 +302,10 @@ public static String getTableName(byte[] schemaName, 
byte[] tableName) {
 return getName(schemaName, tableName);
 }
 
+public static String getPhysicalName(String schemaName, String 
tableName) {
+return schemaName + QueryConstants.NAMESPACE_SEPARATOR + tableName;
+}
+
 public static String getColumnDisplayName(byte[] cf, byte[] cq) {
--- End diff --

we probably don't need this method after above changes. 


> ReadOnlyTableException occurs when we map Phoenix view to an existing HBase 
> table with Namespace Mapping enabled
> 
>
> Key: PHOENIX-3944
> URL: https://issues.apache.org/jira/browse/PHOENIX-3944
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>
> Firstly, I created a namespace and a table in hbase shell:
> {code}
> hbase> create_namespace "NS"
> hbase> create "NS:TBL", "CF"
> {code}
> After that, I tried to create a phoenix view for this existing hbase table in 
> phoenix-sqlline:
> {code}
> phoenix> CREATE VIEW ns.tbl (pk VARCHAR PRIMARY KEY, cf.col VARCHAR);
> {code}
> However, I encountered the following error and I was not able to create the 
> view:
> {code}
> org.apache.phoenix.schema.ReadOnlyTableException: ERROR 505 (42000): Table is 
> read only.
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1072)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1434)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2624)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1040)
>   at 
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:212)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:393)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:376)
>   at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:375)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:363)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1707)
>   at org.apache.PhoenixTest.main(PhoenixTest.java:55)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3944) ReadOnlyTableException occurs when we map Phoenix view to an existing HBase table with Namespace Mapping enabled


[ 
https://issues.apache.org/jira/browse/PHOENIX-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056995#comment-16056995
 ] 

ASF GitHub Bot commented on PHOENIX-3944:
-

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123153895
  
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java 
---
@@ -710,6 +714,27 @@ public void testCreateViewWithUpdateCacheFrquency() 
throws Exception {
   assertTrue(rs.next());
 }
 
+@Test
+public void 
testCreateViewMappedToExistingHbaseTableWithNamespaceMappingEnabled() throws 
Exception {
+Properties props = new Properties();
+props.setProperty(QueryServices.IS_NAMESPACE_MAPPING_ENABLED, 
"true");
+Connection conn = DriverManager.getConnection(getUrl(),props);
+
+String ns = "NS_" + generateUniqueName();
+String table = "TABLE_" + generateUniqueName();
+String fam = "CF";
+
+HBaseAdmin admin = 
conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin();
+admin.createNamespace(NamespaceDescriptor.create(ns).build());
+HTableDescriptor desc = new HTableDescriptor(TableName.valueOf(ns, 
table));
+desc.addFamily(new HColumnDescriptor(fam));
+admin.createTable(desc);
+
+conn.createStatement().execute("CREATE SCHEMA " + ns);
+conn.createStatement().execute("CREATE VIEW " + ns + "." + table + 
" "
++ "(PK VARCHAR PRIMARY KEY, " + fam + ".COL VARCHAR)");
--- End diff --

can you update a test to include a case where user has "NS.TBL" in default 
namespace and he wants to map a view.(\"NS.TBL\")(for backward compatibility).
```
desc = new HTableDescriptor(TableName.valueOf(ns+"."+table));
desc.addFamily(new HColumnDescriptor(fam));
admin.createTable(desc);

conn.createStatement().execute("CREATE VIEW \"" + ns + "." + table + "\""
+ "(PK VARCHAR PRIMARY KEY, " + fam + ".COL VARCHAR)");
```


> ReadOnlyTableException occurs when we map Phoenix view to an existing HBase 
> table with Namespace Mapping enabled
> 
>
> Key: PHOENIX-3944
> URL: https://issues.apache.org/jira/browse/PHOENIX-3944
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>
> Firstly, I created a namespace and a table in hbase shell:
> {code}
> hbase> create_namespace "NS"
> hbase> create "NS:TBL", "CF"
> {code}
> After that, I tried to create a phoenix view for this existing hbase table in 
> phoenix-sqlline:
> {code}
> phoenix> CREATE VIEW ns.tbl (pk VARCHAR PRIMARY KEY, cf.col VARCHAR);
> {code}
> However, I encountered the following error and I was not able to create the 
> view:
> {code}
> org.apache.phoenix.schema.ReadOnlyTableException: ERROR 505 (42000): Table is 
> read only.
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1072)
>   at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1434)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2624)
>   at 
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1040)
>   at 
> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:212)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:393)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:376)
>   at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:375)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:363)
>   at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1707)
>   at org.apache.PhoenixTest.main(PhoenixTest.java:55)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #261: PHOENIX-3944 ReadOnlyTableException occurs when w...

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123153895
  
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java 
---
@@ -710,6 +714,27 @@ public void testCreateViewWithUpdateCacheFrquency() 
throws Exception {
   assertTrue(rs.next());
 }
 
+@Test
+public void 
testCreateViewMappedToExistingHbaseTableWithNamespaceMappingEnabled() throws 
Exception {
+Properties props = new Properties();
+props.setProperty(QueryServices.IS_NAMESPACE_MAPPING_ENABLED, 
"true");
+Connection conn = DriverManager.getConnection(getUrl(),props);
+
+String ns = "NS_" + generateUniqueName();
+String table = "TABLE_" + generateUniqueName();
+String fam = "CF";
+
+HBaseAdmin admin = 
conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin();
+admin.createNamespace(NamespaceDescriptor.create(ns).build());
+HTableDescriptor desc = new HTableDescriptor(TableName.valueOf(ns, 
table));
+desc.addFamily(new HColumnDescriptor(fam));
+admin.createTable(desc);
+
+conn.createStatement().execute("CREATE SCHEMA " + ns);
+conn.createStatement().execute("CREATE VIEW " + ns + "." + table + 
" "
++ "(PK VARCHAR PRIMARY KEY, " + fam + ".COL VARCHAR)");
--- End diff --

can you update a test to include a case where user has "NS.TBL" in default 
namespace and he wants to map a view.(\"NS.TBL\")(for backward compatibility).
```
desc = new HTableDescriptor(TableName.valueOf(ns+"."+table));
desc.addFamily(new HColumnDescriptor(fam));
admin.createTable(desc);

conn.createStatement().execute("CREATE VIEW \"" + ns + "." + table + "\""
+ "(PK VARCHAR PRIMARY KEY, " + fam + ".COL VARCHAR)");
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #261: PHOENIX-3944 ReadOnlyTableException occurs when w...

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154332
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/SchemaUtil.java ---
@@ -302,6 +302,10 @@ public static String getTableName(byte[] schemaName, 
byte[] tableName) {
 return getName(schemaName, tableName);
 }
 
+public static String getPhysicalName(String schemaName, String 
tableName) {
+return schemaName + QueryConstants.NAMESPACE_SEPARATOR + tableName;
+}
+
 public static String getColumnDisplayName(byte[] cf, byte[] cq) {
--- End diff --

we probably don't need this method after above changes. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #261: PHOENIX-3944 ReadOnlyTableException occurs when w...

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154290
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -2135,7 +2135,7 @@ private PTable 
createTableInternal(CreateTableStatement statement, byte[][] spli
 // Upsert physical name for mapped view only if the full 
physical table name is different than the full table name
 // Otherwise, we end up with a self-referencing link and 
then cannot ever drop the view.
 if (viewType != ViewType.MAPPED
-|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getTableName(schemaName, 
tableName))) {
+|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getPhysicalName(schemaName, 
tableName))) {
--- End diff --

can you include isNamespaceMapped as table name with schema can also be in 
default namespace.
```
-|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getTableName(schemaName, 
tableName))) {
+|| 
!physicalNames.get(0).getString().equals(SchemaUtil.getPhysicalTableName(
+SchemaUtil.getTableNameAsBytes(schemaName, 
tableName), isNamespaceMapped).getNameAsString())) {
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #261: PHOENIX-3944 ReadOnlyTableException occurs when w...

Github user ankitsinghal commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/261#discussion_r123154068
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/FromCompiler.java ---
@@ -196,7 +196,8 @@ public static ColumnResolver 
getResolverForCreation(final CreateTableStatement s
 if (htable != null) Closeables.closeQuietly(htable);
 }
 tableNode = NamedTableNode.create(null, baseTable, 
statement.getColumnDefs());
-return new SingleTableColumnResolver(connection, 
tableNode, e.getTimeStamp(), new HashMap(1), false);
+boolean isNamespaceMapped = 
SchemaUtil.isNamespaceMappingEnabled(statement.getTableType(), 
connection.getQueryServices().getProps());
--- End diff --


To handle view mapped \"NS.TBL\"(without schema)

```
-byte[] fullTableName = SchemaUtil.getPhysicalName(
-
SchemaUtil.getTableNameAsBytes(baseTable.getSchemaName(), 
baseTable.getTableName()),
-
connection.getQueryServices().getProps()).getName();
+byte[] fullTableName = SchemaUtil
+
.getPhysicalName(Bytes.toBytes(baseTable.toString()),
+connection.getQueryServices().getProps())
+.getName();
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3962) Not creating the table in custom schema

2017-06-20 Thread Ankit Singhal (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056988#comment-16056988
 ] 

Ankit Singhal commented on PHOENIX-3962:


It seems error while creating SYSTEM.MUTEX snapshot for upgrade. can you check 
master logs for the similar errors as well.
I'm assuming that you are not running multiple connections when you are doing 
upgrade(means during first time after enabling namespace properties). 

You can also try after deleting the snapshot if it exists (from hbase shell: 
list_snapshots and delete_snapshot '_UPGRADING_TABLE_SYSTEM.MUTEX')

> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

vishal created PHOENIX-3966:
---

 Summary: Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }
 Key: PHOENIX-3966
 URL: https://issues.apache.org/jira/browse/PHOENIX-3966
 Project: Phoenix
  Issue Type: Bug
 Environment: HBase version : 1.2.6
Phoenix Version : 4.10.0-HBase-1.2.0

Reporter: vishal


While creating schema or table I am getting this error:
Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
table=SYSTEM.MUTEX type=DISABLED }
So because of this error it is not creating the schema or table.

java-program:
--
{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = 
DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
statement = connection.createStatement();
statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
{code}

hdfs-site.xml

{code:java}
 
phoenix.schema.isNamespaceMappingEnabled
true
   

phoenix.schema.mapSystemTablesToNamespace
true
   
{code}

please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3962) Not creating the table in custom schema


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056945#comment-16056945
 ] 

vishal edited comment on PHOENIX-3962 at 6/21/17 4:05 AM:
--

Yes I have added in server side i.e. in hdfs-site.xml and as well as in client 
side i.e. in my java program.
But every time I am getting new errors LIKE..
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=FLUSH } had an error.  
_UPGRADING_TABLE_SYSTEM.MUTEX not found in proclist []

hdfs-site.xml


{code:java}
   
phoenix.schema.isNamespaceMappingEnabled
true
   


phoenix.schema.mapSystemTablesToNamespace
true
   
{code}


java application

{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = DriverManager.getConnection("jdbc:phoenix:localhost", 
connectionProps);
{code}




was (Author: vishalb.e...@gmail.com):
Yes I have added in server side i.e. in hdfs-site.xml and as well as in client 
side i.e. in my java program.
But every time I am getting new errors..

hdfs-site.xml


{code:java}
   
phoenix.schema.isNamespaceMappingEnabled
true
   


phoenix.schema.mapSystemTablesToNamespace
true
   
{code}


java application

{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = DriverManager.getConnection("jdbc:phoenix:localhost", 
connectionProps);
{code}



> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3962) Not creating the table in custom schema


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vishal updated PHOENIX-3962:

Description: 
I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
But instead of creating the table in that namespace it is creating it in 
default namespace with the table name  *MYTEST.MYTAB1*.
This not my requirement.

I have done like this:
1) hbase(main):059:0> create_namespace 'MYTEST'
2)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> result will be empty as i have not created any tables

3)createing table using phoenix using sql as below:

{code:java}
connection = DriverManager.getConnection("jdbc:phoenix:localhost");
statement = connection.createStatement();
statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
null primary key, name varchar)");
connection.commit();
{code}

4)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> still it is returning empty.

Please suggest me the right syntax to create table in my own schema.

  was:
I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
But instead of creating the table in that namespace it is creating it in 
default namespace with the table name  *MYTEST.MYTAB1*.
This not my requirement.

I have done like this:
1) hbase(main):059:0> create_namespace 'MYTEST'
2)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> result will be empty as i have not created any tables

3)createing table using phoenix using sql as below:

{code:java}
connection = DriverManager.getConnection("jdbc:phoenix:localhost:12181");
statement = connection.createStatement();
statement.executeUpdate("create table *MYTEST.MYTAB1* (employee_id integer not 
null primary key, name varchar)");
connection.commit();
{code}

4)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> still it is returning empty.

Please suggest me the right syntax to create table in my own schema.


> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost");
> statement = connection.createStatement();
> statement.executeUpdate("create table MYTEST.MYTAB1 (employee_id integer not 
> null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3962) Not creating the table in custom schema


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056945#comment-16056945
 ] 

vishal commented on PHOENIX-3962:
-

Yes I have added in server side i.e. in hdfs-site.xml and as well as in client 
side i.e. in my java program.
But every time I am getting new errors..

hdfs-site.xml


{code:java}
   
phoenix.schema.isNamespaceMappingEnabled
true
   


phoenix.schema.mapSystemTablesToNamespace
true
   
{code}


java application

{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = DriverManager.getConnection("jdbc:phoenix:localhost", 
connectionProps);
{code}



> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost:12181");
> statement = connection.createStatement();
> statement.executeUpdate("create table *MYTEST.MYTAB1* (employee_id integer 
> not null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3964) Index.preWALRestore should handle index write failure

2017-06-20 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056811#comment-16056811
 ] 

Andrew Purtell commented on PHOENIX-3964:
-

WDYT [~samarthjain] [~jamestaylor]

> Index.preWALRestore should handle index write failure
> -
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: chenglei
> Attachments: PHOENIX-3964_v1.patch
>
>
> When I restarted my hbase cluster a certain time, I noticed some regions are 
> in FAILED_OPEN state and the RegionServers have some error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
> handler.OpenRegionHandler: Failed open of 
> region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
>  starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
> Failed to write to multiple index tables
> at 
> org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at 
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at 
> org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at 
> org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When I look the code of Index.preWALRestore method, I find 
> RecoveryIndexWriter.write method is used to write the indexUpdates in 
> following line 543: 
>
> {code:java}
>  
> 526  public void preWALRestore(ObserverContext 
> env, HRegionInfo info,
> 527  HLogKey logKey, WALEdit logEdit) throws IOException {
> 528  if (this.disabled) {
> 529  super.preWALRestore(env, info, logKey, logEdit);
> 530  return;
> 531}
> 532// TODO check the regions in transition. If the server on which the 
> region lives is this one,
> 533// then we should rety that write later in postOpen.
> 534// we might be able to get even smarter here and pre-split the edits 
> that are server-local
> 535// into their own recovered.edits file. This then lets us do a 
> straightforward recovery of each
> 536// region (and more efficiently as we aren't writing quite as 
> hectically from this one place).
> 537
> 538/*
> 539 * Basically, we let the index regions recover for a little while long 
> before retrying in the
> 540 * hopes they come up before the primary table finishes.
> 541 */
> 542

[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle index write failure


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenglei updated PHOENIX-3964:
--
Description: 
When I restarted my hbase cluster a certain time, I noticed some regions are in 
FAILED_OPEN state and the RegionServers have some error logs as following:

{code:java}
2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
handler.OpenRegionHandler: Failed open of 
region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
 starting to roll back the global memstore size.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
Failed to write to multiple index tables
at 
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
at 
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at 
org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
at 
org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}

When I look the code of Index.preWALRestore method, I find 
RecoveryIndexWriter.write method is used to write the indexUpdates in following 
line 543: 
   

{code:java}
 
526  public void preWALRestore(ObserverContext 
env, HRegionInfo info,
527  HLogKey logKey, WALEdit logEdit) throws IOException {
528  if (this.disabled) {
529  super.preWALRestore(env, info, logKey, logEdit);
530  return;
531}
532// TODO check the regions in transition. If the server on which the 
region lives is this one,
533// then we should rety that write later in postOpen.
534// we might be able to get even smarter here and pre-split the edits 
that are server-local
535// into their own recovered.edits file. This then lets us do a 
straightforward recovery of each
536// region (and more efficiently as we aren't writing quite as hectically 
from this one place).
537
538/*
539 * Basically, we let the index regions recover for a little while long 
before retrying in the
540 * hopes they come up before the primary table finishes.
541 */
542Collection> indexUpdates = 
extractIndexUpdate(logEdit);
543recoveryWriter.write(indexUpdates, true);
544  }
{code}

but the RecoveryIndexWriter.write method is as following, it directly throws 
Exception except non-existing tables, so RecoveryIndexWriter's 
failurePolicy(which is StoreFailuresInCachePolicy by default) even has no 
opportunity to be used,  and it leads to Index.failedIndexEdits which is filled 
by the StoreFailuresInCachePolicy is always empty.

{code:java}
 public void write(Collection> toWrite, boolean 
allowLocalUpdates) throws IOException {
try {

[jira] [Created] (PHOENIX-3965) Github repo read me need to update

Ethan Wang created PHOENIX-3965:
---

 Summary: Github repo read me need to update
 Key: PHOENIX-3965
 URL: https://issues.apache.org/jira/browse/PHOENIX-3965
 Project: Phoenix
  Issue Type: Improvement
Reporter: Ethan Wang
Priority: Trivial


Github repo read me need to update. Both logo and read me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus


[ 
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056673#comment-16056673
 ] 

Ethan Wang commented on PHOENIX-3390:
-

[~talktoswapna][~talktoswa...@gmail.com]
Thanks! Forked!

> Custom UDAF for HyperLogLogPlus
> ---
>
> Key: PHOENIX-3390
> URL: https://issues.apache.org/jira/browse/PHOENIX-3390
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Swapna Kasula
>Priority: Minor
>
> With ref # PHOENIX-2069
> Custome UDAF to aggregate/union of Hyperloglog's of a column and returns a 
> Hyperloglog.
> select hllUnion(col1) from table;  //returns a Hyperloglog, which is the 
> union of all hyperloglog's from all rows for column 'col1'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

[
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
]

Ethan Wang edited comment on PHOENIX-3390 at 6/20/17 11:36 PM:
---

Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

First, HyperLogLog(as describe in
http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) is an algorithm that
is only good at one particular task: Count of distinct items in a multi-set. It
does not help on generic user-defined aggregation function, such as APPROX_SUM,
APPROX_MAX etc. (The APPROX_SUM example Swapna provided above is actually the
count of the union of two multi-sets, but it's still count)

Secondly, HyperLogLog is optimizing on "Space usage", so that it is able to do
the count on a huge data set (before it is impossible do so, because you have
to remember each distinct item in memory). So HLL is a "can't-do" to "can-do"
improvement, not a "slow" to "faster" improvement: It still has to full scan
through every item, there is no "sampling" going on in the process!

Besides HLL, there do exist other algorithms and their implementations that
will achieve other types of user defined approximate aggregation. Those works
such as SuperLogLog, KLL(sketching algorithm by Zohar et al), the work by
Manku, Rajagopalan et al (1), and the work by Munro and Paterson et al (2).
Among which, Yahoo's library DataSketches(https://datasketches.github.io.
thanks for recommending @Rahul G) provides the following Approximate
aggregations that may be particularly interesting for Phoenix. Besides HLL
based APPROX_COUNT, there are :
APPROX_MEDIAN
APPROX_PERCENTILE (like95 Percentile)
APPROX_MIN/MAX
APPROX_KMV (the famous Kth Minimum Value)

=
Now, everything above are space-optimized approaches. We could, have a
different set of user defined functions for approximate aggregation that is
time-optimized based. i.e., run faster. Such as a use case below: "I'm able to
do select count( * ) from A, but I don't want to, because full scan of A take
too long. Can I just count on 10% sample and get a approximate that way?"
Along this line, the similar approaches would be
BlinkDB(https://sameeragarwal.github.io/blinkdb_eurosys13.pdf), PHOENIX-153
tablesample.

Phoenix community please advice the direction we are going.

[~jamestaylor][~gjacoby]

(1). Gurmeet Singh Manku, Sridhar Rajagopalan, and Bruce G. Lindsay. Random
sampling tech-niques for space efficient online computation of order statistics
of large datasets.
(2). J.I. Munro and M.S. Paterson. Selection and sorting with limited storage.

was (Author: aertoria):
Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

First, HyperLogLog( as describe in
http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) is an algorithm that
is only good at one particular task: Count of distinct items in a multi-set. It
does not help on generic user-defined aggregation function, such as APPROX_SUM,
APPROX_MAX etc. (The APPROX_SUM example Swapna provided above is actually the
count of the union of two multi-sets, but it's still count)

[jira] [Comment Edited] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

[
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
]

Ethan Wang edited comment on PHOENIX-3390 at 6/20/17 11:36 PM:
---

Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

First, HyperLogLog is an algorithm that is only good at one particular task:
Count of distinct items in a multi-set. It does not help on generic
user-defined aggregation function, such as APPROX_SUM, APPROX_MAX etc. (The
APPROX_SUM example Swapna provided above is actually the count of the union of
two multi-sets, but it's still count)

Phoenix community please advice the direction we are going.

[~jamestaylor][~gjacoby]

[jira] [Comment Edited] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

[
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
]

Ethan Wang edited comment on PHOENIX-3390 at 6/20/17 11:36 PM:
---

Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

First, HyperLogLog( as describe in
http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) is an algorithm that
is only good at one particular task: Count of distinct items in a multi-set. It
does not help on generic user-defined aggregation function, such as APPROX_SUM,
APPROX_MAX etc. (The APPROX_SUM example Swapna provided above is actually the
count of the union of two multi-sets, but it's still count)

Phoenix community please advice the direction we are going.

[~jamestaylor][~gjacoby]

[jira] [Commented] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

2017-06-20 Thread Swapna Kasula (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056669#comment-16056669
 ] 

Swapna Kasula commented on PHOENIX-3390:


[~aertoria] I have pushed my code to git repo. Below is the URL, check it out.
https://github.com/SwapnaKasula/apache_phoenix_cdh5

> Custom UDAF for HyperLogLogPlus
> ---
>
> Key: PHOENIX-3390
> URL: https://issues.apache.org/jira/browse/PHOENIX-3390
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Swapna Kasula
>Priority: Minor
>
> With ref # PHOENIX-2069
> Custome UDAF to aggregate/union of Hyperloglog's of a column and returns a 
> Hyperloglog.
> select hllUnion(col1) from table;  //returns a Hyperloglog, which is the 
> union of all hyperloglog's from all rows for column 'col1'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

[
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
]

Ethan Wang edited comment on PHOENIX-3390 at 6/20/17 11:16 PM:
---

Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

Phoenix community please advice the direction we are going.

[~jamestaylor][~gjacoby]

=
Now, everything above are space-optimized approaches. We could, have a
different set of user defined functions for approximate aggregation that is
time-optimized based. i.e., run faster.
Such as a use case below: "I'm able to do select count( * ) from A, but I don't
want to, because full scan of A take too long. Can I just count on 10% sample
and get a approximate that way?"
Along this line, the similar approaches would be

[jira] [Comment Edited] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

[
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
]

Ethan Wang edited comment on PHOENIX-3390 at 6/20/17 11:12 PM:
---

Had a discussion with [~talktoswapna] today. Thanks to her time we clarified
some items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

=
Now, everything above are space-optimized approaches. We could, have a
different set of user defined functions for approximate aggregation that is
time-optimized based. i.e., run faster.
Such as a use case below: "I'm able to do select count( * ) from A, but I don't
want to, because full scan of A take too long. Can I just count on 10% sample
and get a approximate that way?"
Along this line, the similar approaches would be BlinkDB, PHOENIX-153
tablesample.

Phoenix community please advice the direction we are going.

[~jamestaylor][~gjacoby]

was (Author: aertoria):
Had a discussion with [~swapna] today. Thanks to her time we clarified some
items regarding this feature. Now I'd like to add some more detail to the
description of this ticket.

=
Now, everything above are space-optimized approaches. We could, have a
different set of user defined functions for approximate aggregation that is
time-optimized based. i.e., run faster.
Such as a use case below: "I'm able to do select count( * ) from A, but I don't
want to, because full scan of A take too long. Can I just count on 10% sample
and get a approximate that way?"
Along this line, the similar approaches would be BlinkDB, PHOENIX-153
tablesample.

Phoenix community please

[jira] [Commented] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus

2017-06-20 Thread Geoffrey Jacoby (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056629#comment-16056629
 ] 

Geoffrey Jacoby commented on PHOENIX-3390:
--

[~julianhyde], fyi. I recently saw your ApacheCon talk on sketches/hyperloglog 
in Calcite and thought this might interest you given the ongoing work on 
Phoenix/Calcite integration. 

> Custom UDAF for HyperLogLogPlus
> ---
>
> Key: PHOENIX-3390
> URL: https://issues.apache.org/jira/browse/PHOENIX-3390
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Swapna Kasula
>Priority: Minor
>
> With ref # PHOENIX-2069
> Custome UDAF to aggregate/union of Hyperloglog's of a column and returns a 
> Hyperloglog.
> select hllUnion(col1) from table;  //returns a Hyperloglog, which is the 
> union of all hyperloglog's from all rows for column 'col1'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #236: PHOENIX-3654 Load Balancer for thin client

2017-06-20 Thread rahulsIOT

Github user rahulsIOT commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116803
  
--- Diff: phoenix-queryserver/src/it/resources/log4j.properties ---
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Define some default values that can be overridden by system properties
-hbase.root.logger=DEBUG,console
--- End diff --

yes. sorry. will quickly fix this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3654) Load Balancer for thin client


[ 
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056608#comment-16056608
 ] 

ASF GitHub Bot commented on PHOENIX-3654:
-

Github user rahulsIOT commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116803
  
--- Diff: phoenix-queryserver/src/it/resources/log4j.properties ---
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Define some default values that can be overridden by system properties
-hbase.root.logger=DEBUG,console
--- End diff --

yes. sorry. will quickly fix this.


> Load Balancer for thin client
> -
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
>  Issue Type: New Feature
>Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
>Reporter: Rahul Shrivastava
>Assignee: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments: LoadBalancerDesign.pdf, Loadbalancer.patch
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> We have been having internal discussion on load balancer for thin client for 
> PQS. The general consensus we have is to have an embedded load balancer with 
> the thin client instead of using external load balancer such as haproxy. The 
> idea is to not to have another layer between client and PQS. This reduces 
> operational cost for system, which currently leads to delay in executing 
> projects.
> But this also comes with challenge of having an embedded load balancer which 
> can maintain sticky sessions, do fair load balancing knowing the load 
> downstream of PQS server. In addition, load balancer needs to know location 
> of multiple PQS server. Now, the thin client needs to keep track of PQS 
> servers via zookeeper ( or other means). 
> In the new design, the client ( PQS client) , it is proposed to  have an 
> embedded load balancer.
> Where will the load Balancer sit ?
> The load load balancer will embedded within the app server client.  
> How will the load balancer work ? 
> Load balancer will contact zookeeper to get location of PQS. In this case, 
> PQS needs to register to ZK itself once it comes online. Zookeeper location 
> is in hbase-site.xml. It will maintain a small cache of connection to the 
> PQS. When a request comes in, it will check for an open connection from the 
> cache. 
> How will load balancer know load on PQS ?
> To start with, it will pick a random open connection to PQS. This means that 
> load balancer does not know PQS load. Later , we can augment the code so that 
> thin client can receive load info from PQS and make intelligent decisions.  
> How will load balancer maintain sticky sessions ?
> While we still need to investigate how to implement sticky sessions. We can 
> look for some open source implementation for the same.
> How will PQS register itself to service locator ?
> PQS will have location of zookeeper in hbase-site.xml and it would register 
> itself to the zookeeper. Thin client will find out PQS location using 
> zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3654) Load Balancer for thin client


[ 
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056607#comment-16056607
 ] 

ASF GitHub Bot commented on PHOENIX-3654:
-

Github user joshelser commented on the issue:

https://github.com/apache/phoenix/pull/236
  
Ok, thanks for the pointer, Rahul.


> Load Balancer for thin client
> -
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
>  Issue Type: New Feature
>Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
>Reporter: Rahul Shrivastava
>Assignee: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments: LoadBalancerDesign.pdf, Loadbalancer.patch
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> We have been having internal discussion on load balancer for thin client for 
> PQS. The general consensus we have is to have an embedded load balancer with 
> the thin client instead of using external load balancer such as haproxy. The 
> idea is to not to have another layer between client and PQS. This reduces 
> operational cost for system, which currently leads to delay in executing 
> projects.
> But this also comes with challenge of having an embedded load balancer which 
> can maintain sticky sessions, do fair load balancing knowing the load 
> downstream of PQS server. In addition, load balancer needs to know location 
> of multiple PQS server. Now, the thin client needs to keep track of PQS 
> servers via zookeeper ( or other means). 
> In the new design, the client ( PQS client) , it is proposed to  have an 
> embedded load balancer.
> Where will the load Balancer sit ?
> The load load balancer will embedded within the app server client.  
> How will the load balancer work ? 
> Load balancer will contact zookeeper to get location of PQS. In this case, 
> PQS needs to register to ZK itself once it comes online. Zookeeper location 
> is in hbase-site.xml. It will maintain a small cache of connection to the 
> PQS. When a request comes in, it will check for an open connection from the 
> cache. 
> How will load balancer know load on PQS ?
> To start with, it will pick a random open connection to PQS. This means that 
> load balancer does not know PQS load. Later , we can augment the code so that 
> thin client can receive load info from PQS and make intelligent decisions.  
> How will load balancer maintain sticky sessions ?
> While we still need to investigate how to implement sticky sessions. We can 
> look for some open source implementation for the same.
> How will PQS register itself to service locator ?
> PQS will have location of zookeeper in hbase-site.xml and it would register 
> itself to the zookeeper. Thin client will find out PQS location using 
> zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #236: PHOENIX-3654 Load Balancer for thin client

2017-06-20 Thread joshelser

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116460
  
--- Diff: 
phoenix-load-balancer/src/main/java/org/apache/phoenix/loadbalancer/service/LoadBalancer.java
 ---
@@ -0,0 +1,170 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.loadbalancer.service;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.net.HostAndPort;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.UnhandledErrorListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.curator.utils.CloseableUtils;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+
+import java.net.ConnectException;
+import java.util.ArrayList;
+import java.util.concurrent.ThreadLocalRandom;
+import java.io.Closeable;
+import java.util.List;
+
+
+public class LoadBalancer {
+
+private static final LoadBalanceZookeeperConf CONFIG = new 
LoadBalanceZookeeperConfImpl(HBaseConfiguration.create());
+private static CuratorFramework curaFramework = null;
+protected static final Log LOG = LogFactory.getLog(LoadBalancer.class);
+private static PathChildrenCache   cache = null;
+private static final LoadBalancer loadBalancer = new LoadBalancer();
+private ConnectionStateListener connectionStateListener = null;
+private UnhandledErrorListener unhandledErrorListener = null;
+private List closeAbles = Lists.newArrayList();
+
+private LoadBalancer()  {
+try  {
+start();
+}catch(Exception ex){
+LOG.error("Exception while creating a zookeeper clients and 
cache",ex);
+if ((curaFramework != null) && (connectionStateListener != 
null)){
+curaFramework.getConnectionStateListenable()
+.removeListener(connectionStateListener);
+}
+if ((curaFramework != null) && (unhandledErrorListener != 
null)){
+curaFramework.getUnhandledErrorListenable()
+.removeListener(unhandledErrorListener);
+}
+for (Closeable closeable : closeAbles) {
+CloseableUtils.closeQuietly(closeable);
+}
+}
+}
+
+
+
+/**
+ * Return Singleton Load Balancer every single time.
+ * @return LoadBalancer
+ */
+public static LoadBalancer getLoadBalancer() {
+return loadBalancer;
+}
+
+/**
+ * return the location of Phoenix Query Server
--- End diff --

Javadoc could use some updating to reflect the use of HostAndPort and alert 
users that they will see an exception if there are no registered PQS instances.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3654) Load Balancer for thin client


[ 
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056604#comment-16056604
 ] 

ASF GitHub Bot commented on PHOENIX-3654:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116460
  
--- Diff: 
phoenix-load-balancer/src/main/java/org/apache/phoenix/loadbalancer/service/LoadBalancer.java
 ---
@@ -0,0 +1,170 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.loadbalancer.service;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.net.HostAndPort;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.api.UnhandledErrorListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import org.apache.curator.framework.state.ConnectionState;
+import org.apache.curator.framework.state.ConnectionStateListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.curator.utils.CloseableUtils;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+
+import java.net.ConnectException;
+import java.util.ArrayList;
+import java.util.concurrent.ThreadLocalRandom;
+import java.io.Closeable;
+import java.util.List;
+
+
+public class LoadBalancer {
+
+private static final LoadBalanceZookeeperConf CONFIG = new 
LoadBalanceZookeeperConfImpl(HBaseConfiguration.create());
+private static CuratorFramework curaFramework = null;
+protected static final Log LOG = LogFactory.getLog(LoadBalancer.class);
+private static PathChildrenCache   cache = null;
+private static final LoadBalancer loadBalancer = new LoadBalancer();
+private ConnectionStateListener connectionStateListener = null;
+private UnhandledErrorListener unhandledErrorListener = null;
+private List closeAbles = Lists.newArrayList();
+
+private LoadBalancer()  {
+try  {
+start();
+}catch(Exception ex){
+LOG.error("Exception while creating a zookeeper clients and 
cache",ex);
+if ((curaFramework != null) && (connectionStateListener != 
null)){
+curaFramework.getConnectionStateListenable()
+.removeListener(connectionStateListener);
+}
+if ((curaFramework != null) && (unhandledErrorListener != 
null)){
+curaFramework.getUnhandledErrorListenable()
+.removeListener(unhandledErrorListener);
+}
+for (Closeable closeable : closeAbles) {
+CloseableUtils.closeQuietly(closeable);
+}
+}
+}
+
+
+
+/**
+ * Return Singleton Load Balancer every single time.
+ * @return LoadBalancer
+ */
+public static LoadBalancer getLoadBalancer() {
+return loadBalancer;
+}
+
+/**
+ * return the location of Phoenix Query Server
--- End diff --

Javadoc could use some updating to reflect the use of HostAndPort and alert 
users that they will see an exception if there are no registered PQS instances.


> Load Balancer for thin client
> -
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
>  Issue Type: New Feature
>Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
>Reporter: Rahul Shrivastava
>Assignee: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments:

[GitHub] phoenix pull request #236: PHOENIX-3654 Load Balancer for thin client

2017-06-20 Thread joshelser

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116500
  
--- Diff: phoenix-queryserver/src/it/resources/log4j.properties ---
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Define some default values that can be overridden by system properties
-hbase.root.logger=DEBUG,console
--- End diff --

Did you accidentally delete this file..?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix issue #236: PHOENIX-3654 Load Balancer for thin client

2017-06-20 Thread joshelser

Github user joshelser commented on the issue:

https://github.com/apache/phoenix/pull/236
  
Ok, thanks for the pointer, Rahul.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3654) Load Balancer for thin client


[ 
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056606#comment-16056606
 ] 

ASF GitHub Bot commented on PHOENIX-3654:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/236#discussion_r123116500
  
--- Diff: phoenix-queryserver/src/it/resources/log4j.properties ---
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Define some default values that can be overridden by system properties
-hbase.root.logger=DEBUG,console
--- End diff --

Did you accidentally delete this file..?


> Load Balancer for thin client
> -
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
>  Issue Type: New Feature
>Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
>Reporter: Rahul Shrivastava
>Assignee: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments: LoadBalancerDesign.pdf, Loadbalancer.patch
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> We have been having internal discussion on load balancer for thin client for 
> PQS. The general consensus we have is to have an embedded load balancer with 
> the thin client instead of using external load balancer such as haproxy. The 
> idea is to not to have another layer between client and PQS. This reduces 
> operational cost for system, which currently leads to delay in executing 
> projects.
> But this also comes with challenge of having an embedded load balancer which 
> can maintain sticky sessions, do fair load balancing knowing the load 
> downstream of PQS server. In addition, load balancer needs to know location 
> of multiple PQS server. Now, the thin client needs to keep track of PQS 
> servers via zookeeper ( or other means). 
> In the new design, the client ( PQS client) , it is proposed to  have an 
> embedded load balancer.
> Where will the load Balancer sit ?
> The load load balancer will embedded within the app server client.  
> How will the load balancer work ? 
> Load balancer will contact zookeeper to get location of PQS. In this case, 
> PQS needs to register to ZK itself once it comes online. Zookeeper location 
> is in hbase-site.xml. It will maintain a small cache of connection to the 
> PQS. When a request comes in, it will check for an open connection from the 
> cache. 
> How will load balancer know load on PQS ?
> To start with, it will pick a random open connection to PQS. This means that 
> load balancer does not know PQS load. Later , we can augment the code so that 
> thin client can receive load info from PQS and make intelligent decisions.  
> How will load balancer maintain sticky sessions ?
> While we still need to investigate how to implement sticky sessions. We can 
> look for some open source implementation for the same.
> How will PQS register itself to service locator ?
> PQS will have location of zookeeper in hbase-site.xml and it would register 
> itself to the zookeeper. Thin client will find out PQS location using 
> zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3390) Custom UDAF for HyperLogLogPlus


[ 
https://issues.apache.org/jira/browse/PHOENIX-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056597#comment-16056597
 ] 

Ethan Wang commented on PHOENIX-3390:
-

Had a discussion with [~swapna] today. Thanks to her time we clarified some 
items regarding this feature. Now I'd like to add some more detail to the 
description of this ticket.

First, HyperLogLog is an algorithm that is only good at one particular task: 
Count of distinct items in a multi-set. It does not help on generic 
user-defined aggregation function, such as APPROX_SUM, APPROX_MAX etc. (The 
APPROX_SUM example Swapna provided above is actually the count of the union of 
two multi-sets, but it's still count)

Secondly, HyperLogLog is optimizing on "Space usage", so that it is able to do 
the count on a huge data set (before it is impossible do so, because you have 
to remember each distinct item in memory). So HLL is a "can't-do" to "can-do" 
improvement, not a "slow" to "faster" improvement: It still has to full scan 
through every item, there is no "sampling" going on in the process!

Besides HLL, there do exist other algorithms and their implementations that 
will achieve other types of user defined approximate aggregation. Those works 
such as SuperLogLog, KLL(sketching algorithm by Zohar et al), the work by 
Manku, Rajagopalan et al (1), and the work by Munro and Paterson et al (2).  
Among which, Yahoo's library DataSketches(https://datasketches.github.io. 
thanks for recommending @Rahul G) provides the following Approximate 
aggregations that may be particularly interesting for Phoenix. Besides HLL 
based APPROX_COUNT, there are :
APPROX_MEDIAN
APPROX_PERCENTILE (like95 Percentile)
APPROX_MIN/MAX
APPROX_KMV (the famous Kth Minimum Value)


=
Now, everything above are space-optimized approaches. We could, have a 
different set of user defined functions for approximate aggregation that is 
time-optimized based.  i.e., run faster. 
Such as a use case below: "I'm able to do select count( * ) from A, but I don't 
want to, because full scan of A take too long. Can I just count on 10% sample 
and get a approximate that way?" 
Along this line, the similar approaches would be BlinkDB, PHOENIX-153 
tablesample.

Phoenix community please advice the direction we are going. 

[~jamestaylor][~gjacoby]





(1). Gurmeet Singh Manku, Sridhar Rajagopalan, and Bruce G. Lindsay. Random 
sampling tech-niques for space efficient online computation of order statistics 
of large datasets.
(2). J.I. Munro and M.S. Paterson. Selection and sorting with limited storage.

> Custom UDAF for HyperLogLogPlus
> ---
>
> Key: PHOENIX-3390
> URL: https://issues.apache.org/jira/browse/PHOENIX-3390
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Swapna Kasula
>Priority: Minor
>
> With ref # PHOENIX-2069
> Custome UDAF to aggregate/union of Hyperloglog's of a column and returns a 
> Hyperloglog.
> select hllUnion(col1) from table;  //returns a Hyperloglog, which is the 
> union of all hyperloglog's from all rows for column 'col1'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3654) Load Balancer for thin client


[ 
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056583#comment-16056583
 ] 

ASF GitHub Bot commented on PHOENIX-3654:
-

Github user rahulsIOT commented on the issue:

https://github.com/apache/phoenix/pull/236
  
Thin client can call " 
LoadBalancer.getLoadBalancer().getSingleServiceLocation() " and get hold of 
host and port for PQS location. It can user DriverManager.getConnection to get 
connection object. The 
LoadBalancer.getLoadBalancer().getSingleServiceLocation() throw exception that 
client needs to catch. Usually it shows that there is some issue with 
connecting to zookeeper or there is not PQS registered with the zookeeper. 


> Load Balancer for thin client
> -
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
>  Issue Type: New Feature
>Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
>Reporter: Rahul Shrivastava
>Assignee: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments: LoadBalancerDesign.pdf, Loadbalancer.patch
>
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> We have been having internal discussion on load balancer for thin client for 
> PQS. The general consensus we have is to have an embedded load balancer with 
> the thin client instead of using external load balancer such as haproxy. The 
> idea is to not to have another layer between client and PQS. This reduces 
> operational cost for system, which currently leads to delay in executing 
> projects.
> But this also comes with challenge of having an embedded load balancer which 
> can maintain sticky sessions, do fair load balancing knowing the load 
> downstream of PQS server. In addition, load balancer needs to know location 
> of multiple PQS server. Now, the thin client needs to keep track of PQS 
> servers via zookeeper ( or other means). 
> In the new design, the client ( PQS client) , it is proposed to  have an 
> embedded load balancer.
> Where will the load Balancer sit ?
> The load load balancer will embedded within the app server client.  
> How will the load balancer work ? 
> Load balancer will contact zookeeper to get location of PQS. In this case, 
> PQS needs to register to ZK itself once it comes online. Zookeeper location 
> is in hbase-site.xml. It will maintain a small cache of connection to the 
> PQS. When a request comes in, it will check for an open connection from the 
> cache. 
> How will load balancer know load on PQS ?
> To start with, it will pick a random open connection to PQS. This means that 
> load balancer does not know PQS load. Later , we can augment the code so that 
> thin client can receive load info from PQS and make intelligent decisions.  
> How will load balancer maintain sticky sessions ?
> While we still need to investigate how to implement sticky sessions. We can 
> look for some open source implementation for the same.
> How will PQS register itself to service locator ?
> PQS will have location of zookeeper in hbase-site.xml and it would register 
> itself to the zookeeper. Thin client will find out PQS location using 
> zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix issue #236: PHOENIX-3654 Load Balancer for thin client

2017-06-20 Thread rahulsIOT

Github user rahulsIOT commented on the issue:

https://github.com/apache/phoenix/pull/236
  
Thin client can call " 
LoadBalancer.getLoadBalancer().getSingleServiceLocation() " and get hold of 
host and port for PQS location. It can user DriverManager.getConnection to get 
connection object. The 
LoadBalancer.getLoadBalancer().getSingleServiceLocation() throw exception that 
client needs to catch. Usually it shows that there is some issue with 
connecting to zookeeper or there is not PQS registered with the zookeeper. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[RESULT][VOTE] Release of Apache Phoenix 4.11.0 RC1

2017-06-20 Thread James Taylor

The voting is now closed and passes with 5 +1 votes and no +0 or -1 votes.
Thanks to those who voted. We'll push the bits out shortly.

James

On Wed, Jun 21, 2017 at 6:35 AM James Taylor  wrote:

> +1
>
> On Wed, Jun 21, 2017 at 5:20 AM Samarth Jain  wrote:
>
>> +1
>>
>> On Mon, Jun 19, 2017 at 5:47 PM, Mujtaba Chohan 
>> wrote:
>>
>> > +1.
>> >
>> > Verified backward compat. and perf. with previous release.
>> >
>> > On Mon, Jun 19, 2017 at 12:16 PM Thomas D'Silva > >
>> > wrote:
>> >
>> > > +1
>> > >
>> > > Ran some manual tests.
>> > >
>> > > On Thu, Jun 15, 2017 at 5:18 PM, Josh Elser 
>> wrote:
>> > >
>> > > > +1 (binding)
>> > > >
>> > > > * xsums/sigs OK
>> > > > * Verified pom.xml made no changes (implying no required L
>> changes)
>> > > > * apache-rat:check on all src archives
>> > > > * Can build from source on all archives
>> > > > * Ran all UT/IT successfully on hbase-1.3 release
>> > > >
>> > > > - Josh
>> > > >
>> > > >
>> > > > On 6/14/17 4:57 PM, James Taylor wrote:
>> > > >
>> > > >> Hello Everyone,
>> > > >>
>> > > >> This is a call for a vote on Apache Phoenix 4.11.0 RC1. This is the
>> > next
>> > > >> minor release of Phoenix 4, compatible with Apache HBase 0.98, 1.1,
>> > > 1.2, &
>> > > >> 1.3. The release includes both a source-only release and a
>> convenience
>> > > >> binary release for each supported HBase version. The previous RC
>> was
>> > > sunk
>> > > >> due to PHOENIX-3890 which has now been fixed.
>> > > >>
>> > > >> This release has feature parity with supported HBase versions and
>> > > includes
>> > > >> the following improvements:
>> > > >> - Support for HBase 1.3.1+
>> > > >> - Atomic update of data and local index rows (HBase 1.3 only) [1]
>> > > >> - Local index hardening and performance improvements [2]
>> > > >> - Use HBase snapshots for MR-based queries and async index building
>> > > [3][4]
>> > > >> - Support for forward-only cursors [5]
>> > > >> - Ensure chunking of commit data uses byte-based and row-count
>> limits
>> > > [6]
>> > > >> - Reduce load on SYSTEM.CATALOG region server [7]
>> > > >> - Implement FIRST_VALUES and LAST_VALUES aggregate functions [8]
>> > > >> - 50+ bug fixes
>> > > >>
>> > > >> The source tarball, including signatures, digests, etc can be found
>> > at:
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-0.98-rc1/src/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.1-rc1/src/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.2-rc1/src/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.3-rc1/src/
>> > > >>
>> > > >> The binary artifacts can be found at:
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-0.98-rc1/bin/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.1-rc1/bin/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.2-rc1/bin/
>> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>> > > >> x-4.11.0-HBase-1.3-rc1/bin/
>> > > >>
>> > > >> For a complete list of changes, see:
>> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> > > >> ctId=12315120=12339764
>> > > >>
>> > > >> Release artifacts are signed with the following key:
>> > > >> https://people.apache.org/keys/committer/mujtaba.asc
>> > > >> https://dist.apache.org/repos/dist/release/phoenix/KEYS
>> > > >>
>> > > >> The hash and tag to be voted upon:
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
>> > > >> it;h=4bc09b8167c9a9f8af205eb19f25e0687527d3c4
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
>> > > >> h=refs/tags/v4.11.0-HBase-0.98-rc1
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
>> > > >> it;h=78a41484f583aa35285b5a8a8d82714458a9e8da
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
>> > > >> h=refs/tags/v4.11.0-HBase-1.1-rc1
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
>> > > >> it;h=d70dab9c134cdefb7c5771023d1f774ee601455a
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
>> > > >> h=refs/tags/v4.11.0-HBase-1.2-rc1
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
>> > > >> it;h=de6fbc4e2a13cdc482cbc1c91e51c4bc526aa12f
>> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
>> > > >> h=refs/tags/v4.11.0-HBase-1.3-rc1
>> > > >>
>> > > >> Vote will be open for at least 72 hours. Please vote:
>> > > >>
>> > > >> [ ] +1 approve
>> > > >> [ ] +0 no opinion
>> > > >> [ ] -1 disapprove (and reason why)
>> > > >>
>> > > >> Thanks,
>> > > >> The Apache Phoenix Team
>> > > >>
>> > > >> [1]

Re: [VOTE] Release of Apache Phoenix 4.11.0 RC1

2017-06-20 Thread James Taylor

+1

On Wed, Jun 21, 2017 at 5:20 AM Samarth Jain  wrote:

> +1
>
> On Mon, Jun 19, 2017 at 5:47 PM, Mujtaba Chohan 
> wrote:
>
> > +1.
> >
> > Verified backward compat. and perf. with previous release.
> >
> > On Mon, Jun 19, 2017 at 12:16 PM Thomas D'Silva 
> > wrote:
> >
> > > +1
> > >
> > > Ran some manual tests.
> > >
> > > On Thu, Jun 15, 2017 at 5:18 PM, Josh Elser  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > * xsums/sigs OK
> > > > * Verified pom.xml made no changes (implying no required L changes)
> > > > * apache-rat:check on all src archives
> > > > * Can build from source on all archives
> > > > * Ran all UT/IT successfully on hbase-1.3 release
> > > >
> > > > - Josh
> > > >
> > > >
> > > > On 6/14/17 4:57 PM, James Taylor wrote:
> > > >
> > > >> Hello Everyone,
> > > >>
> > > >> This is a call for a vote on Apache Phoenix 4.11.0 RC1. This is the
> > next
> > > >> minor release of Phoenix 4, compatible with Apache HBase 0.98, 1.1,
> > > 1.2, &
> > > >> 1.3. The release includes both a source-only release and a
> convenience
> > > >> binary release for each supported HBase version. The previous RC was
> > > sunk
> > > >> due to PHOENIX-3890 which has now been fixed.
> > > >>
> > > >> This release has feature parity with supported HBase versions and
> > > includes
> > > >> the following improvements:
> > > >> - Support for HBase 1.3.1+
> > > >> - Atomic update of data and local index rows (HBase 1.3 only) [1]
> > > >> - Local index hardening and performance improvements [2]
> > > >> - Use HBase snapshots for MR-based queries and async index building
> > > [3][4]
> > > >> - Support for forward-only cursors [5]
> > > >> - Ensure chunking of commit data uses byte-based and row-count
> limits
> > > [6]
> > > >> - Reduce load on SYSTEM.CATALOG region server [7]
> > > >> - Implement FIRST_VALUES and LAST_VALUES aggregate functions [8]
> > > >> - 50+ bug fixes
> > > >>
> > > >> The source tarball, including signatures, digests, etc can be found
> > at:
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-0.98-rc1/src/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.1-rc1/src/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.2-rc1/src/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.3-rc1/src/
> > > >>
> > > >> The binary artifacts can be found at:
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-0.98-rc1/bin/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.1-rc1/bin/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.2-rc1/bin/
> > > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > > >> x-4.11.0-HBase-1.3-rc1/bin/
> > > >>
> > > >> For a complete list of changes, see:
> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > > >> ctId=12315120=12339764
> > > >>
> > > >> Release artifacts are signed with the following key:
> > > >> https://people.apache.org/keys/committer/mujtaba.asc
> > > >> https://dist.apache.org/repos/dist/release/phoenix/KEYS
> > > >>
> > > >> The hash and tag to be voted upon:
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > > >> it;h=4bc09b8167c9a9f8af205eb19f25e0687527d3c4
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > > >> h=refs/tags/v4.11.0-HBase-0.98-rc1
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > > >> it;h=78a41484f583aa35285b5a8a8d82714458a9e8da
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > > >> h=refs/tags/v4.11.0-HBase-1.1-rc1
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > > >> it;h=d70dab9c134cdefb7c5771023d1f774ee601455a
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > > >> h=refs/tags/v4.11.0-HBase-1.2-rc1
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > > >> it;h=de6fbc4e2a13cdc482cbc1c91e51c4bc526aa12f
> > > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > > >> h=refs/tags/v4.11.0-HBase-1.3-rc1
> > > >>
> > > >> Vote will be open for at least 72 hours. Please vote:
> > > >>
> > > >> [ ] +1 approve
> > > >> [ ] +0 no opinion
> > > >> [ ] -1 disapprove (and reason why)
> > > >>
> > > >> Thanks,
> > > >> The Apache Phoenix Team
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/PHOENIX-3827
> > > >> [2]
> > > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%
> > > >> 20PHOENIX%20and%20fixVersion%3D4.11.0%20and%20labels%3DlocalIndex
> > > >> [3] https://issues.apache.org/jira/browse/PHOENIX-3744
> > > >> [4] https://issues.apache.org/jira/browse/PHOENIX-3812
> > > >> [5]

[jira] [Commented] (PHOENIX-3962) Not creating the table in custom schema

2017-06-20 Thread Biju Nair (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056411#comment-16056411
 ] 

Biju Nair commented on PHOENIX-3962:


Did you check/follow the instructions for {{namespace}} mapping provided 
[here|http://phoenix.apache.org/namspace_mapping.html]?

> Not creating the table in custom schema
> ---
>
> Key: PHOENIX-3962
> URL: https://issues.apache.org/jira/browse/PHOENIX-3962
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
> Environment: HBase : 1.2.6
> Phoenix : 4.10.0-Hbase-1.2.0
>Reporter: vishal
>
> I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
> But instead of creating the table in that namespace it is creating it in 
> default namespace with the table name  *MYTEST.MYTAB1*.
> This not my requirement.
> I have done like this:
> 1) hbase(main):059:0> create_namespace 'MYTEST'
> 2)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> result will be empty as i have not created any tables
> 3)createing table using phoenix using sql as below:
> {code:java}
> connection = DriverManager.getConnection("jdbc:phoenix:localhost:12181");
> statement = connection.createStatement();
> statement.executeUpdate("create table *MYTEST.MYTAB1* (employee_id integer 
> not null primary key, name varchar)");
> connection.commit();
> {code}
> 4)hbase(main):059:0> list_namespace_tables 'MYTEST'
>--> still it is returning empty.
> Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release of Apache Phoenix 4.11.0 RC1

2017-06-20 Thread Samarth Jain

+1

On Mon, Jun 19, 2017 at 5:47 PM, Mujtaba Chohan  wrote:

> +1.
>
> Verified backward compat. and perf. with previous release.
>
> On Mon, Jun 19, 2017 at 12:16 PM Thomas D'Silva 
> wrote:
>
> > +1
> >
> > Ran some manual tests.
> >
> > On Thu, Jun 15, 2017 at 5:18 PM, Josh Elser  wrote:
> >
> > > +1 (binding)
> > >
> > > * xsums/sigs OK
> > > * Verified pom.xml made no changes (implying no required L changes)
> > > * apache-rat:check on all src archives
> > > * Can build from source on all archives
> > > * Ran all UT/IT successfully on hbase-1.3 release
> > >
> > > - Josh
> > >
> > >
> > > On 6/14/17 4:57 PM, James Taylor wrote:
> > >
> > >> Hello Everyone,
> > >>
> > >> This is a call for a vote on Apache Phoenix 4.11.0 RC1. This is the
> next
> > >> minor release of Phoenix 4, compatible with Apache HBase 0.98, 1.1,
> > 1.2, &
> > >> 1.3. The release includes both a source-only release and a convenience
> > >> binary release for each supported HBase version. The previous RC was
> > sunk
> > >> due to PHOENIX-3890 which has now been fixed.
> > >>
> > >> This release has feature parity with supported HBase versions and
> > includes
> > >> the following improvements:
> > >> - Support for HBase 1.3.1+
> > >> - Atomic update of data and local index rows (HBase 1.3 only) [1]
> > >> - Local index hardening and performance improvements [2]
> > >> - Use HBase snapshots for MR-based queries and async index building
> > [3][4]
> > >> - Support for forward-only cursors [5]
> > >> - Ensure chunking of commit data uses byte-based and row-count limits
> > [6]
> > >> - Reduce load on SYSTEM.CATALOG region server [7]
> > >> - Implement FIRST_VALUES and LAST_VALUES aggregate functions [8]
> > >> - 50+ bug fixes
> > >>
> > >> The source tarball, including signatures, digests, etc can be found
> at:
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-0.98-rc1/src/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.1-rc1/src/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.2-rc1/src/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.3-rc1/src/
> > >>
> > >> The binary artifacts can be found at:
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-0.98-rc1/bin/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.1-rc1/bin/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.2-rc1/bin/
> > >> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
> > >> x-4.11.0-HBase-1.3-rc1/bin/
> > >>
> > >> For a complete list of changes, see:
> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > >> ctId=12315120=12339764
> > >>
> > >> Release artifacts are signed with the following key:
> > >> https://people.apache.org/keys/committer/mujtaba.asc
> > >> https://dist.apache.org/repos/dist/release/phoenix/KEYS
> > >>
> > >> The hash and tag to be voted upon:
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > >> it;h=4bc09b8167c9a9f8af205eb19f25e0687527d3c4
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > >> h=refs/tags/v4.11.0-HBase-0.98-rc1
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > >> it;h=78a41484f583aa35285b5a8a8d82714458a9e8da
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > >> h=refs/tags/v4.11.0-HBase-1.1-rc1
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > >> it;h=d70dab9c134cdefb7c5771023d1f774ee601455a
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > >> h=refs/tags/v4.11.0-HBase-1.2-rc1
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=comm
> > >> it;h=de6fbc4e2a13cdc482cbc1c91e51c4bc526aa12f
> > >> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=tag;
> > >> h=refs/tags/v4.11.0-HBase-1.3-rc1
> > >>
> > >> Vote will be open for at least 72 hours. Please vote:
> > >>
> > >> [ ] +1 approve
> > >> [ ] +0 no opinion
> > >> [ ] -1 disapprove (and reason why)
> > >>
> > >> Thanks,
> > >> The Apache Phoenix Team
> > >>
> > >> [1] https://issues.apache.org/jira/browse/PHOENIX-3827
> > >> [2]
> > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%
> > >> 20PHOENIX%20and%20fixVersion%3D4.11.0%20and%20labels%3DlocalIndex
> > >> [3] https://issues.apache.org/jira/browse/PHOENIX-3744
> > >> [4] https://issues.apache.org/jira/browse/PHOENIX-3812
> > >> [5] https://issues.apache.org/jira/browse/PHOENIX-3572
> > >> [6] https://issues.apache.org/jira/browse/PHOENIX-3784
> > >> [7] https://issues.apache.org/jira/browse/PHOENIX-3819
> > >> [8] https://issues.apache.org/jira/browse/PHOENIX-3773
> > >>
> > >>
> >
>

[jira] [Commented] (PHOENIX-3964) Index.preWALRestore should handle index write failure

2017-06-20 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056057#comment-16056057
 ] 

Hadoop QA commented on PHOENIX-3964:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12873695/PHOENIX-3964_v1.patch
  against master branch at commit fba5fa28a03279e3fc427de800774690d280edca.
  ATTACHMENT ID: 12873695

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
50 warning messages.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1115//testReport/
Javadoc warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1115//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1115//console

This message is automatically generated.

> Index.preWALRestore should handle index write failure
> -
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: chenglei
> Attachments: PHOENIX-3964_v1.patch
>
>
> When I restarted my hbase cluster a certain time, I noticed some regions are 
> in FAILED_OPEN state and the RegionServers have some error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
> handler.OpenRegionHandler: Failed open of 
> region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
>  starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
> Failed to write to multiple index tables
> at 
> org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at 
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at 
> org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at 
> org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at 
>

[jira] [Commented] (PHOENIX-3964) Index.preWALRestore should handle index write failure


[ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055896#comment-16055896
 ] 

chenglei commented on PHOENIX-3964:
---

upload first patch.

> Index.preWALRestore should handle index write failure
> -
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: chenglei
> Attachments: PHOENIX-3964_v1.patch
>
>
> When I restarted my hbase cluster a certain time, I noticed some regions are 
> in FAILED_OPEN state and the RegionServers have some error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
> handler.OpenRegionHandler: Failed open of 
> region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
>  starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
> Failed to write to multiple index tables
> at 
> org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at 
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at 
> org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at 
> org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When I look the code of Index.preWALRestore method, I find 
> RecoveryIndexWriter.write method is used to write the indexUpdates in 
> following line 543: 
>
> {code:java}
>  
> 526  public void preWALRestore(ObserverContext 
> env, HRegionInfo info,
> 527  HLogKey logKey, WALEdit logEdit) throws IOException {
> 528  if (this.disabled) {
> 529  super.preWALRestore(env, info, logKey, logEdit);
> 530  return;
> 531}
> 532// TODO check the regions in transition. If the server on which the 
> region lives is this one,
> 533// then we should rety that write later in postOpen.
> 534// we might be able to get even smarter here and pre-split the edits 
> that are server-local
> 535// into their own recovered.edits file. This then lets us do a 
> straightforward recovery of each
> 536// region (and more efficiently as we aren't writing quite as 
> hectically from this one place).
> 537
> 538/*
> 539 * Basically, we let the index regions recover for a little while long 
> before retrying in the
> 540 * hopes they come up before the primary table finishes.
> 541 */
> 542

[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle index write failure


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenglei updated PHOENIX-3964:
--
Attachment: PHOENIX-3964_v1.patch

> Index.preWALRestore should handle index write failure
> -
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: chenglei
> Attachments: PHOENIX-3964_v1.patch
>
>
> When I restarted my hbase cluster a certain time, I noticed some regions are 
> in FAILED_OPEN state and the RegionServers have some error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
> handler.OpenRegionHandler: Failed open of 
> region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
>  starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
> Failed to write to multiple index tables
> at 
> org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at 
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at 
> org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at 
> org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When I look the code of Index.preWALRestore method, I find 
> RecoveryIndexWriter.write method is used to write the indexUpdates in 
> following line 543: 
>
> {code:java}
>  
> 526  public void preWALRestore(ObserverContext 
> env, HRegionInfo info,
> 527  HLogKey logKey, WALEdit logEdit) throws IOException {
> 528  if (this.disabled) {
> 529  super.preWALRestore(env, info, logKey, logEdit);
> 530  return;
> 531}
> 532// TODO check the regions in transition. If the server on which the 
> region lives is this one,
> 533// then we should rety that write later in postOpen.
> 534// we might be able to get even smarter here and pre-split the edits 
> that are server-local
> 535// into their own recovered.edits file. This then lets us do a 
> straightforward recovery of each
> 536// region (and more efficiently as we aren't writing quite as 
> hectically from this one place).
> 537
> 538/*
> 539 * Basically, we let the index regions recover for a little while long 
> before retrying in the
> 540 * hopes they come up before the primary table finishes.
> 541 */
> 542Collection>

[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle index write failure


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenglei updated PHOENIX-3964:
--
Description: 
When I restarted my hbase cluster a certain time, I noticed some regions are in 
FAILED_OPEN state and the RegionServers have some error logs as following:

{code:java}
2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
handler.OpenRegionHandler: Failed open of 
region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
 starting to roll back the global memstore size.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
Failed to write to multiple index tables
at 
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
at 
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at 
org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
at 
org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}

When I look the code of Index.preWALRestore method, I find 
RecoveryIndexWriter.write method is used to write the indexUpdates in following 
line 543: 
   

{code:java}
 
526  public void preWALRestore(ObserverContext 
env, HRegionInfo info,
527  HLogKey logKey, WALEdit logEdit) throws IOException {
528  if (this.disabled) {
529  super.preWALRestore(env, info, logKey, logEdit);
530  return;
531}
532// TODO check the regions in transition. If the server on which the 
region lives is this one,
533// then we should rety that write later in postOpen.
534// we might be able to get even smarter here and pre-split the edits 
that are server-local
535// into their own recovered.edits file. This then lets us do a 
straightforward recovery of each
536// region (and more efficiently as we aren't writing quite as hectically 
from this one place).
537
538/*
539 * Basically, we let the index regions recover for a little while long 
before retrying in the
540 * hopes they come up before the primary table finishes.
541 */
542Collection> indexUpdates = 
extractIndexUpdate(logEdit);
543recoveryWriter.write(indexUpdates, true);
544  }
{code}

but the RecoveryIndexWriter.write method is as following, it directly throws 
Exception except non-existing tables, so RecoveryIndexWriter's 
failurePolicy(which is StoreFailuresInCachePolicy by default) even has no 
opportunity to be used,  and it leads to Index.failedIndexEdits which is filled 
by the StoreFailuresInCachePolicy is always empty.

{code:java}
 public void write(Collection> toWrite, boolean 
allowLocalUpdates) throws IOException {
try {

[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle index write failure


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenglei updated PHOENIX-3964:
--
Description: 
When I restarted my hbase cluster a certain time, I noticed some regions are in 
FAILED_OPEN state and the RegionServers have some error logs as following:

{code:java}
2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
handler.OpenRegionHandler: Failed open of 
region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
 starting to roll back the global memstore size.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
Failed to write to multiple index tables
at 
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
at 
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at 
org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
at 
org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}

When I look the code of Index.preWALRestore method, I find 
RecoveryIndexWriter.write method is used to write the indexUpdates in following 
line 534: 
   

{code:java}
 
526  public void preWALRestore(ObserverContext 
env, HRegionInfo info,
527  HLogKey logKey, WALEdit logEdit) throws IOException {
528  if (this.disabled) {
529  super.preWALRestore(env, info, logKey, logEdit);
530  return;
531}
532// TODO check the regions in transition. If the server on which the 
region lives is this one,
533// then we should rety that write later in postOpen.
534// we might be able to get even smarter here and pre-split the edits 
that are server-local
535// into their own recovered.edits file. This then lets us do a 
straightforward recovery of each
536// region (and more efficiently as we aren't writing quite as hectically 
from this one place).
537
538/*
539 * Basically, we let the index regions recover for a little while long 
before retrying in the
540 * hopes they come up before the primary table finishes.
541 */
542Collection> indexUpdates = 
extractIndexUpdate(logEdit);
543recoveryWriter.write(indexUpdates, true);
544  }
{code}

but the RecoveryIndexWriter.write method is as following, it directly throws 
Exception except non-existing tables, so RecoveryIndexWriter's 
failurePolicy(which is StoreFailuresInCachePolicy by default) even has no 
opportunity to be used,  and it leads to Index.failedIndexEdits which is filled 
by the StoreFailuresInCachePolicy is always empty.

{code:java}
 public void write(Collection> toWrite, boolean 
allowLocalUpdates) throws IOException {
try {

[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle index write failure


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenglei updated PHOENIX-3964:
--
Description: 
When I restarted my hbase cluster a certain time, I noticed some region is in 
FAILED_OPEN state and the RegionServer has error logs as following:

{code:java}
2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
handler.OpenRegionHandler: Failed open of 
region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
 starting to roll back the global memstore size.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
Failed to write to multiple index tables
at 
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
at 
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at 
org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
at 
org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}


   






 

> Index.preWALRestore should handle index write failure
> -
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: chenglei
>
> When I restarted my hbase cluster a certain time, I noticed some region is in 
> FAILED_OPEN state and the RegionServer has error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] 
> handler.OpenRegionHandler: Failed open of 
> region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5.,
>  starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
> Failed to write to multiple index tables
> at 
> org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at 
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at 
> org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at 
> org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at 
>

[jira] [Created] (PHOENIX-3964) Index.preWALRestore should handle index write failure

chenglei created PHOENIX-3964:
-

 Summary: Index.preWALRestore should handle index write failure
 Key: PHOENIX-3964
 URL: https://issues.apache.org/jira/browse/PHOENIX-3964
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.10.0
Reporter: chenglei






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3860) Implement TAL functionality for Omid

2017-06-20 Thread Ohad Shacham (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055391#comment-16055391
 ] 

Ohad Shacham commented on PHOENIX-3860:
---


Thanks [~giacomotay...@gmail.com]. 

Regarding the secondary index. As far as I understood, while creating an index 
to a non empty data table you update the index with the table data (original 
data) while using the timestamp taken at the fence and not using the original 
data timestamp. Entries of transaction that started after the fence are added 
to the index with the same timestamp as appear in the data table.

I assume that the timestamp of the fence should be generated during the fence.
However, taking a look at the code I see that the fence creation does not 
return a timestamp and a transaction is started right after creating the fence. 
Does the timestamp of this transaction is being used for the index entries of 
the original data? If so, can't we violate snapshot isolation if another 
transaction started right in between? this transaction timestamp is smaller 
than the index entries of the original data even though they were 
added/committed after the original data.

Thx,
Ohad

> Implement TAL functionality for Omid
> 
>
> Key: PHOENIX-3860
> URL: https://issues.apache.org/jira/browse/PHOENIX-3860
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Ohad Shacham
>
> Implement TAL functionality for Omid in order to be able to use Omid as 
> Phoenix's transaction processing engine. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Description: 
Hi There

I have a HBase Table with a column family which reduces the space requirements 
of HBase by using key value pairs in the form of (Timestamp, measurement). This 
is a wide column family and the column names can't be ascertained in advance- 
they are generated at the run time.

As per my investigation, phoenix necessitates that schema should be specified 
in advance. Is there any way we can enable phoenix to query data in such form? 
Thanks in Advance !

Sample data format is attached

  was:
Hi There

I have a HBase Table with a column family which reduces the space requirements 
of HBase by using key value pairs in the form of (Timestamp, measurement). This 
is a wide column family and the column names can't be ascertained in advance- 
they are generated at the run time.

As per my investigation, phoenix necessitates that schema should be specified 
in advance. Is there any way we can enable phoenix to query data in such form?

Sample data format is attached


> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: DataSample.jpg
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such 
> form? Thanks in Advance !
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Attachment: DataSample.jpg

> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: DataSample.jpg
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such form?
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Attachment: (was: Sample Data.png)

> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: DataSample.jpg
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such form?
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Attachment: (was: DataSample.jpg)

> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: Sample Data.png
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such form?
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Attachment: DataSample.jpg

> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: Sample Data.png
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such form?
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3963) Ability to query HBase wide column Family


 [ 
https://issues.apache.org/jira/browse/PHOENIX-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

CS Life updated PHOENIX-3963:
-
Attachment: Sample Data.png

> Ability to query HBase wide column Family
> -
>
> Key: PHOENIX-3963
> URL: https://issues.apache.org/jira/browse/PHOENIX-3963
> Project: Phoenix
>  Issue Type: Task
> Environment: HDP2.4 cluster, CentOS
>Reporter: CS Life
> Attachments: Sample Data.png
>
>
> Hi There
> I have a HBase Table with a column family which reduces the space 
> requirements of HBase by using key value pairs in the form of (Timestamp, 
> measurement). This is a wide column family and the column names can't be 
> ascertained in advance- they are generated at the run time.
> As per my investigation, phoenix necessitates that schema should be specified 
> in advance. Is there any way we can enable phoenix to query data in such form?
> Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3963) Ability to query HBase wide column Family

CS Life created PHOENIX-3963:


 Summary: Ability to query HBase wide column Family
 Key: PHOENIX-3963
 URL: https://issues.apache.org/jira/browse/PHOENIX-3963
 Project: Phoenix
  Issue Type: Task
 Environment: HDP2.4 cluster, CentOS
Reporter: CS Life


Hi There

I have a HBase Table with a column family which reduces the space requirements 
of HBase by using key value pairs in the form of (Timestamp, measurement). This 
is a wide column family and the column names can't be ascertained in advance- 
they are generated at the run time.

As per my investigation, phoenix necessitates that schema should be specified 
in advance. Is there any way we can enable phoenix to query data in such form?

Sample data format is attached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3962) Not creating the table in custom schema

vishal created PHOENIX-3962:
---

 Summary: Not creating the table in custom schema
 Key: PHOENIX-3962
 URL: https://issues.apache.org/jira/browse/PHOENIX-3962
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.10.0
 Environment: HBase : 1.2.6
Phoenix : 4.10.0-Hbase-1.2.0
Reporter: vishal


I am trying to create a table MYTAB1 in my schema/namespace MYTEST.
But instead of creating the table in that namespace it is creating it in 
default namespace with the table name  *MYTEST.MYTAB1*.
This not my requirement.

I have done like this:
1) hbase(main):059:0> create_namespace 'MYTEST'
2)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> result will be empty as i have not created any tables

3)createing table using phoenix using sql as below:

{code:java}
connection = DriverManager.getConnection("jdbc:phoenix:localhost:12181");
statement = connection.createStatement();
statement.executeUpdate("create table *MYTEST.MYTAB1* (employee_id integer not 
null primary key, name varchar)");
connection.commit();
{code}

4)hbase(main):059:0> list_namespace_tables 'MYTEST'
   --> still it is returning empty.

Please suggest me the right syntax to create table in my own schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #264: PHOENIX-3961 Should delete `tableOutputPath` in `...

2017-06-20 Thread asdf2014

GitHub user asdf2014 opened a pull request:

https://github.com/apache/phoenix/pull/264

PHOENIX-3961 Should delete `tableOutputPath` in `completebulkload`

For solving the situation that too many `.tmp` files in /tmp directory when 
processing a huge `bulkload` job, we should consider deleting `tableOutputPath` 
in `completebulkload`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/asdf2014/phoenix bulkload_del

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/phoenix/pull/264.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #264


commit 9c121a27315976fd16e1e919aa69592311e3e5de
Author: asdf2014 <1571805...@qq.com>
Date:   2017-06-20T06:14:37Z

PHOENIX-3961 Should delete `tableOutputPath` in `completebulkload`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3961) Should delete `tableOutputPath` in `completebulkload`