[jira] [Updated] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

2017-03-06 Thread Ankit Singhal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal updated PHOENIX-3649:
---
Attachment: PHOENIX-3649_v3.patch

> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on 
> immutable index creation with multiple regions on single RS
> --
>
> Key: PHOENIX-3649
> URL: https://issues.apache.org/jira/browse/PHOENIX-3649
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.9.1, 4.10.0
>
> Attachments: PHOENIX-3649.patch, PHOENIX-3649_v1.patch, 
> PHOENIX-3649_v2.patch, PHOENIX-3649_v3.patch
>
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
> cluster. 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
> FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
> BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
> FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create 
> ~4 regions on a single RS and then create index with data present
> {noformat}
> CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
> {noformat}
> From RS log
> {noformat}
> 2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] 
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
> occupancy alarm watermark (0.95)
> 2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: 
> Expiring session 0x15a00ad4f31, timeout of 1ms exceeded
> 2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: 
> Detected pause in JVM or host machine (eg GC): pause of approximately 10581ms
> GC pool 'ParNew' had collection(s): count=4 time=139ms
> 2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] 
> regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
> regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
> regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
> {noformat}
> Prior to the change index creation succeeds with as little as 2GB heap.
> [~an...@apache.org]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

2017-02-27 Thread Ankit Singhal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal updated PHOENIX-3649:
---
Attachment: PHOENIX-3649_v1.patch

v1 patch for test execution.

[~giacomotaylor], if the first index is created during Upsert Select is running 
on an immutable table, then at server side, we will not see a new index and 
updates to the index will be missed. For now, I have commented the optimisation 
of UPSERT SELECT on SERVER for immutable tables altogether. Let me know if you 
have any good idea to prevent this.

{code}
-&& !(table.isImmutableRows() && 
!table.getIndexes().isEmpty())
+//TODO UPSERT SELECT on Immutable tables 
without the indexes can also be optimized
+//but we need to handle case when index is 
created during UPSERT SELECT is already started
+//see 
ImmutableIndexIT#testCreateIndexDuringUpsertSelect
+&& !(table.isImmutableRows() /*&& 
!table.getIndexes().isEmpty()*/)
{code}

> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on 
> immutable index creation with multiple regions on single RS
> --
>
> Key: PHOENIX-3649
> URL: https://issues.apache.org/jira/browse/PHOENIX-3649
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.9.1, 4.10.0
>
> Attachments: PHOENIX-3649.patch, PHOENIX-3649_v1.patch
>
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
> cluster. 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
> FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
> BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
> FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create 
> ~4 regions on a single RS and then create index with data present
> {noformat}
> CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
> {noformat}
> From RS log
> {noformat}
> 2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] 
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
> occupancy alarm watermark (0.95)
> 2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: 
> Expiring session 0x15a00ad4f31, timeout of 1ms exceeded
> 2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: 
> Detected pause in JVM or host machine (eg GC): pause of approximately 10581ms
> GC pool 'ParNew' had collection(s): count=4 time=139ms
> 2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] 
> regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
> regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
> regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
> {noformat}
> Prior to the change index creation succeeds with as little as 2GB heap.
> [~an...@apache.org]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

2017-02-24 Thread James Taylor (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3649:
--
Priority: Blocker  (was: Major)

> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on 
> immutable index creation with multiple regions on single RS
> --
>
> Key: PHOENIX-3649
> URL: https://issues.apache.org/jira/browse/PHOENIX-3649
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.9.1, 4.10.0
>
> Attachments: PHOENIX-3649.patch
>
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
> cluster. 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
> FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
> BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
> FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create 
> ~4 regions on a single RS and then create index with data present
> {noformat}
> CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
> {noformat}
> From RS log
> {noformat}
> 2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] 
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
> occupancy alarm watermark (0.95)
> 2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: 
> Expiring session 0x15a00ad4f31, timeout of 1ms exceeded
> 2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: 
> Detected pause in JVM or host machine (eg GC): pause of approximately 10581ms
> GC pool 'ParNew' had collection(s): count=4 time=139ms
> 2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] 
> regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
> regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
> regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
> {noformat}
> Prior to the change index creation succeeds with as little as 2GB heap.
> [~an...@apache.org]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

2017-02-03 Thread Ankit Singhal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal updated PHOENIX-3649:
---
Attachment: PHOENIX-3649.patch

> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on 
> immutable index creation with multiple regions on single RS
> --
>
> Key: PHOENIX-3649
> URL: https://issues.apache.org/jira/browse/PHOENIX-3649
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: Mujtaba Chohan
> Fix For: 4.9.1, 4.10.0
>
> Attachments: PHOENIX-3649.patch
>
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
> cluster. 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
> FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
> BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
> FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create 
> ~4 regions on a single RS and then create index with data present
> {noformat}
> CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
> {noformat}
> From RS log
> {noformat}
> 2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] 
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
> occupancy alarm watermark (0.95)
> 2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: 
> Expiring session 0x15a00ad4f31, timeout of 1ms exceeded
> 2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: 
> Detected pause in JVM or host machine (eg GC): pause of approximately 10581ms
> GC pool 'ParNew' had collection(s): count=4 time=139ms
> 2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] 
> regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
> regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
> regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
> {noformat}
> Prior to the change index creation succeeds with as little as 2GB heap.
> [~an...@apache.org]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

2017-02-02 Thread Mujtaba Chohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan updated PHOENIX-3649:

Description: 
*Configuration*
hbase-0.98.23 standalone
Heap 5GB

*When*
Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
cluster. 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6

To repro
{noformat}
CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
{noformat}
Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create ~4 
regions on a single RS and then create index with data present
{noformat}
CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
{noformat}

>From RS log
{noformat}
2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] 
regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
occupancy alarm watermark (0.95)
2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: Expiring 
session 0x15a00ad4f31, timeout of 1ms exceeded
2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected 
pause in JVM or host machine (eg GC): pause of approximately 10581ms
GC pool 'ParNew' had collection(s): count=4 time=139ms
2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] 
regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
{noformat}

Prior to the change index creation succeeds with as little as 2GB heap.

[~an...@apache.org]

  was:
*Configuration*
hbase-0.98.23 standalone
Heap 5GB

*When*
Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
cluster. 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6

To repro
{noformat}
CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
{noformat}
Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create ~4 
regions on a single RS and then create index with data present
{noformat}
CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
{noformat}

>From RS log
{noformat}
2017-02-02 13:29:06,899 WARN  
[vampire8.internal.salesforce.com,51371,1486070044538-HeapMemoryChore] 
regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap 
occupancy alarm watermark (0.95)
2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: Expiring 
session 0x15a00ad4f31, timeout of 1ms exceeded
2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected 
pause in JVM or host machine (eg GC): pause of approximately 10581ms
GC pool 'ParNew' had collection(s): count=4 time=139ms
2017-02-02 13:29:19,669 FATAL [RS:0;vampire8:51371-EventThread] 
regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: 
regionserver:51371-0x15a00ad4f31, quorum=localhost:2181, baseZNode=/hbase 
regionserver:51371-0x15a00ad4f31 received expired from ZooKeeper, aborting
{noformat}

Prior to the change index creation succeeds with as little as 2GB heap.

[~an...@apache.org]


> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on 
> immutable index creation with multiple regions on single RS
> --
>
> Key: PHOENIX-3649
> URL: https://issues.apache.org/jira/browse/PHOENIX-3649
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Mujtaba Chohan
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across 
> cluster. 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, 
> FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB 
> BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, 
> FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create 
> ~4 regions on a single RS and then create index with