[jira] [Commented] (PHOENIX-3994) Index RPC priority still depends on the controller factory property in hbase-site.xml
[ https://issues.apache.org/jira/browse/PHOENIX-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101257#comment-16101257 ] William Yang commented on PHOENIX-3994: --- I've done the test that [PHOENIX-3360-V4.patch|https://issues.apache.org/jira/secure/attachment/12852511/PHOENIX-3360-v4.PATCH] works if HBASE-18359 is fixed. Never mind, the patch in current issue works well too. But I still want to know that whether it's OK to use an HBase internal class CoprocessorHConnection directly in Phoenix code. ping [~jamestaylor],[~samarthjain], thanks > Index RPC priority still depends on the controller factory property in > hbase-site.xml > - > > Key: PHOENIX-3994 > URL: https://issues.apache.org/jira/browse/PHOENIX-3994 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.11.0 >Reporter: Sergey Soldatov >Assignee: Samarth Jain >Priority: Critical > Fix For: 4.12.0, 4.11.1 > > Attachments: PHOENIX-3994_addendum.patch, PHOENIX-3994.patch, > PHOENIX-3994_v2.patch, PHOENIX-3994_v3.patch, PHOENIX-3994_v4.patch, > PHOENIX-3994_v5.patch > > > During PHOENIX-3360 we tried to remove dependency on > hbase.rpc.controllerfactory.class property in hbase-site.xml since it cause > problems on the client side (if client is using server side configuration, > all client request may go using index priority). Committed solution is using > setting the controller factory programmatically for coprocessor environment > in Indexer class, but it comes that this solution doesn't work because the > environment configuration is not used for the coprocessor connection > creation. We need to provide a better solution since this issue may cause > accidental locks and failures that hard to identify and avoid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3994) Index RPC priority still depends on the controller factory property in hbase-site.xml
[ https://issues.apache.org/jira/browse/PHOENIX-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088523#comment-16088523 ] William Yang commented on PHOENIX-3994: --- First, I'm deeply sorry for the committed patch not working in PHOENIX-3360. Please accept my apologies. What is being used in my production env is the [PHOENIX-3360-V3.patch|https://issues.apache.org/jira/secure/attachment/12851796/PHOENIX-3360-v3.PATCH] which is the simplest effective way to solve this problem. But that patch was rejected by using HBase internal class CoprocessorHConnection directly then I came up with the [PHOENIX-3360-V4.patch|https://issues.apache.org/jira/secure/attachment/12852511/PHOENIX-3360-v4.PATCH] which is unfortunately not working. As I saw in the final patch of this issue, CoprocessorHConnection is still created directly from phoenix' code. If so, why can't we just using the V3 patch instead? More over I think the only reason V4 is not working is because HBASE-18359, right? Besides, modify the env's configuration object is also not recommended according to what [~apurtell] said. If we have to clone the config then we cannot use the env's getTable() method as it cannot see the cloned/modified config, even though we get HBASE-18359 fixed. I'll test it this weekend. So, what say you, [~samarthjain], [~jamestaylor]? > Index RPC priority still depends on the controller factory property in > hbase-site.xml > - > > Key: PHOENIX-3994 > URL: https://issues.apache.org/jira/browse/PHOENIX-3994 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.11.0 >Reporter: Sergey Soldatov >Assignee: Samarth Jain >Priority: Critical > Fix For: 4.12.0, 4.11.1 > > Attachments: PHOENIX-3994_addendum.patch, PHOENIX-3994.patch, > PHOENIX-3994_v2.patch, PHOENIX-3994_v3.patch, PHOENIX-3994_v4.patch, > PHOENIX-3994_v5.patch > > > During PHOENIX-3360 we tried to remove dependency on > hbase.rpc.controllerfactory.class property in hbase-site.xml since it cause > problems on the client side (if client is using server side configuration, > all client request may go using index priority). Committed solution is using > setting the controller factory programmatically for coprocessor environment > in Indexer class, but it comes that this solution doesn't work because the > environment configuration is not used for the coprocessor connection > creation. We need to provide a better solution since this issue may cause > accidental locks and failures that hard to identify and avoid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-2388) Support pooling Phoenix connections
[ https://issues.apache.org/jira/browse/PHOENIX-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867327#comment-15867327 ] William Yang commented on PHOENIX-2388: --- Anyone have time to have a look at this? What else shall we implement? > Support pooling Phoenix connections > --- > > Key: PHOENIX-2388 > URL: https://issues.apache.org/jira/browse/PHOENIX-2388 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > Attachments: PHOENIX-2388.patch > > > Frequently user are plugging Phoenix into an ecosystem that pools > connections. It would be possible to implement a pooling mechanism for > Phoenix by creating a delegate Connection that instantiates a new Phoenix > connection when retrieved from the pool and then closes the connection when > returning it to the pool. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (PHOENIX-3360) Secondary index configuration is wrong
[
https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865236#comment-15865236
]
William Yang edited comment on PHOENIX-3360 at 2/14/17 8:28 AM:
bq. CompoundConfiguration treats the added configs as immutable, and has an
internal mutable config (see the code). This means that with the original
patch, the rest of region server (including replication) will not be affected.
I've done a simple test, see {{ConfCP.java}}. If we changed the RegionServer
level configuration in a region's coprocessor, then all the other Regions
opened afterwards on the same RS will see the change. It has nothing to do with
the implementation of Configuration class or any other internal classes, but is
determined by where a region's Configuration object comes from.
I checked the code in both hbase 1.1.2 and 0.94. See
{{RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema()}} for 1.1 and
{{RegionCoprocessorHost#loadTableCoprocessors()}} for 0.94.
Each region will have its own copy of Configuration, which are all copied from
the region server's configuration object. So it is safe to change the
configuration returned by {{CoprocessorEnvironment#getConfiguration()}} and
this change can be seen only within this Region. But we should never change the
Configurations return by
{{RegionCoprocessorEnvironment#getRegionServerServices().getConfiguration()}}
for this will change all the other Regions' conf (Only the regions being opened
afterwards).
How to use ConfCP.java
* create 'test1', 'cf'
* create 'test2', 'cf'
* make sure that all regions of the above two tables are hosted in the same
regionserver
* add coprocessor ConfCP for test1, check log, should see the print below:
{code}
YHYH1: [test1]conf hashCode = 2027310658
YHYH2: [test1]put conf (yh.special.key,XX)
YHYH3: [test1]get conf (yh.special.key,XX)
{code}
* add coprocessor ConfCP for test2, check the log again, should see the print
below
{code}
YHYH1: [test2]conf hashCode = 2027310658
YHYH3: [test2]get conf (yh.special.key,XX)
{code}
So we set a unique conf in coprocessor in test1, then test2 saw it.
Note that {{conf}} can be assigned in two ways. Currently
{code}
conf =
((RegionCoprocessorEnvironment)e).getRegionServerServices().getConfiguration();
{code}
is used, and this is what we do in V1 patch.
Change it to
{code}
conf = e.getConfiguration();
{code}
then table test2 will not see the change that test1 did.
Above all, we can use the v1 patch with a little modification that we just set
the conf returned by {{CoprocessorEnvironment#getConfiguration()}}. And for
PHOENIX-3271 that UPSART SELECT's write will still have higher priority.
WDYT? Ping [~jamestaylor], [~enis], [~rajeshbabu].
was (Author: yhxx511):
bq. CompoundConfiguration treats the added configs as immutable, and has an
internal mutable config (see the code). This means that with the original
patch, the rest of region server (including replication) will not be affected.
I've done a simple test, see {{ConfCP.java}}. If we change the RegionServer
level configuration in a coprocessor, then all the other Regions opened on the
same RS will see the change. It has nothing to do with the implementation of
Configuration class or any other internal classes, but is determined by where a
region's Configuration object comes from.
I checked the code in both hbase 1.1.2 and 0.94. See
{{RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema()}} for 1.1 and
{{RegionCoprocessorHost#loadTableCoprocessors()}} for 0.94.
Each region will have its own copy of Configuration, which are all copied from
the region server's configuration object. So it is safe to change the
configuration returned by {{CoprocessorEnvironment#getConfiguration()}} and
this change can be seen only within this Region. But we should never change the
Configurations return by
{{RegionCoprocessorEnvironment#getRegionServerServices().getConfiguration()}}
for this will change all the other Regions' conf.
How to use ConfCP.java
* create 'test1', 'cf'
* create 'test2', 'cf'
* make sure that all regions of the above two tables are hosted in the same
regionserver
* add coprocessor ConfCP for test1, check log, should see the print below:
{code}
YHYH1: [test1]conf hashCode = 2027310658
YHYH2: [test1]put conf (yh.special.key,XX)
YHYH3: [test1]get conf (yh.special.key,XX)
{code}
* add coprocessor ConfCP for test2, check the log again, should see the print
below
{code}
YHYH1: [test2]conf hashCode = 2027310658
YHYH3: [test2]get conf (yh.special.key,XX)
{code}
Note that {{conf}} can be assigned by two values. for
{code}
conf =
((RegionCoprocessorEnvironment)e).getRegionServerServices().getConfiguration();
{code}
is used now, and this is what we do in V1 patch.
Change it to
{code}
conf = e.getConfiguration();
{code}
then table test2 will not
[jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
[
https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865298#comment-15865298
]
William Yang commented on PHOENIX-3360:
---
New patch attached.
There is another reason we have to create a single connection used for index
updates. See {{CoprocessorHConnection#getConnectionForEnvironment()}}, it will
create a new connection at each call. Then the ctor of
{{HConnectionImplementation}} will be called. In this ctor, it will hit ZK to
read the cluster id by calling {{retrieveClusterId()}}. This is totally
unacceptable. Apart from the extra network operation, it will still generate
many CLOSE-WAIT tcp connections in ZK cluster. As ZK is always a critical
resource that we should try our best to not access it unless we have to. If we
haven't configured connection limit big enough in zoo.cfg ({{maxClientCnxns}}),
then index updates will fail at getting HTableInterface phase because ZK
connection requests are rejected for there are already too many.
Has anyone ever encountered this problem?
> Secondary index configuration is wrong
> --
>
> Key: PHOENIX-3360
> URL: https://issues.apache.org/jira/browse/PHOENIX-3360
> Project: Phoenix
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Fix For: 4.10.0
>
> Attachments: ConfCP.java, PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH,
> PHOENIX-3360-v3.PATCH, PHOENIX-3360-v4.PATCH
>
>
> IndexRpcScheduler allocates some handler threads and uses a higher priority
> for RPCs. The corresponding IndexRpcController is not used by default as it
> is, but used through ServerRpcControllerFactory that we configure from Ambari
> by default which sets the priority of the outgoing RPCs to either metadata
> priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it
> seems that the IndexRPCController DOES NOT look at whether the outgoing RPC
> is for an Index table or not. It just sets ALL rpc priorities to be the index
> priority. The intention seems to be the case that ONLY on servers, we
> configure ServerRpcControllerFactory, and with clients we NEVER configure
> ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We
> configure ServerRpcControllerFactory from Ambari, which in affect makes it so
> that ALL rpcs from Phoenix are only handled by the index handlers by default.
> It means all deadlock cases are still there.
> The documentation in https://phoenix.apache.org/secondary_indexing.html is
> also wrong in this sense. It does not talk about server side / client side.
> Plus this way of configuring different values is not how HBase configuration
> is deployed. We cannot have the configuration show the
> ServerRpcControllerFactory even only for server nodes, because the clients
> running on those nodes will also see the wrong values.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-3360: -- Attachment: PHOENIX-3360-v4.PATCH > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: ConfCP.java, PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH, > PHOENIX-3360-v3.PATCH, PHOENIX-3360-v4.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
[
https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865236#comment-15865236
]
William Yang commented on PHOENIX-3360:
---
bq. CompoundConfiguration treats the added configs as immutable, and has an
internal mutable config (see the code). This means that with the original
patch, the rest of region server (including replication) will not be affected.
I've done a simple test, see {{ConfCP.java}}. If we change the RegionServer
level configuration in a coprocessor, then all the other Regions opened on the
same RS will see the change. It has nothing to do with the implementation of
Configuration class or any other internal classes, but is determined by where a
region's Configuration object comes from.
I checked the code in both hbase 1.1.2 and 0.94. See
{{RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema()}} for 1.1 and
{{RegionCoprocessorHost#loadTableCoprocessors()}} for 0.94.
Each region will have its own copy of Configuration, which are all copied from
the region server's configuration object. So it is safe to change the
configuration returned by {{CoprocessorEnvironment#getConfiguration()}} and
this change can be seen only within this Region. But we should never change the
Configurations return by
{{RegionCoprocessorEnvironment#getRegionServerServices().getConfiguration()}}
for this will change all the other Regions' conf.
How to use ConfCP.java
* create 'test1', 'cf'
* create 'test2', 'cf'
* make sure that all regions of the above two tables are hosted in the same
regionserver
* add coprocessor ConfCP for test1, check log, should see the print below:
{code}
YHYH1: [test1]conf hashCode = 2027310658
YHYH2: [test1]put conf (yh.special.key,XX)
YHYH3: [test1]get conf (yh.special.key,XX)
{code}
* add coprocessor ConfCP for test2, check the log again, should see the print
below
{code}
YHYH1: [test2]conf hashCode = 2027310658
YHYH3: [test2]get conf (yh.special.key,XX)
{code}
Note that {{conf}} can be assigned by two values. for
{code}
conf =
((RegionCoprocessorEnvironment)e).getRegionServerServices().getConfiguration();
{code}
is used now, and this is what we do in V1 patch.
Change it to
{code}
conf = e.getConfiguration();
{code}
then table test2 will not see the change that test1 did.
Above all, we can use the v1 patch with a little modification that we just set
the conf returned by {{CoprocessorEnvironment#getConfiguration()}}. And for
PHOENIX-3271 that UPSART SELECT's write will still have higher priority.
WDYT? Ping [~jamestaylor], [~enis], [~rajeshbabu].
> Secondary index configuration is wrong
> --
>
> Key: PHOENIX-3360
> URL: https://issues.apache.org/jira/browse/PHOENIX-3360
> Project: Phoenix
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Fix For: 4.10.0
>
> Attachments: ConfCP.java, PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH,
> PHOENIX-3360-v3.PATCH
>
>
> IndexRpcScheduler allocates some handler threads and uses a higher priority
> for RPCs. The corresponding IndexRpcController is not used by default as it
> is, but used through ServerRpcControllerFactory that we configure from Ambari
> by default which sets the priority of the outgoing RPCs to either metadata
> priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it
> seems that the IndexRPCController DOES NOT look at whether the outgoing RPC
> is for an Index table or not. It just sets ALL rpc priorities to be the index
> priority. The intention seems to be the case that ONLY on servers, we
> configure ServerRpcControllerFactory, and with clients we NEVER configure
> ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We
> configure ServerRpcControllerFactory from Ambari, which in affect makes it so
> that ALL rpcs from Phoenix are only handled by the index handlers by default.
> It means all deadlock cases are still there.
> The documentation in https://phoenix.apache.org/secondary_indexing.html is
> also wrong in this sense. It does not talk about server side / client side.
> Plus this way of configuring different values is not how HBase configuration
> is deployed. We cannot have the configuration show the
> ServerRpcControllerFactory even only for server nodes, because the clients
> running on those nodes will also see the wrong values.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-3360: -- Attachment: ConfCP.java > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: ConfCP.java, PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH, > PHOENIX-3360-v3.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-3360: -- Attachment: PHOENIX-3360-v3.PATCH bq. I think Ankit Singhal was saying that the priorities are set correct for all RS to RS communication, but looking at the patch for this, it does not seem to be the case (the indexer changes the regionserver's configuration though). Before the v2 patch, it is true because ALL inter-RS requests are made with higher priority, no matter what they really are. But after not, as we don't change the region server's configurations. Only the requests executed by HTableInterface that created by CoprocessorHTableFactory will have higher priority. As PHOENIX-3271, see UngroupedAggregateRegionObserver#doPostScannerOpen()'s diff, an HTable object is created directly with the env's configuration. As this configuration no longer contains the RPC class config entry, then the writes will not have higher priority. In order to make it right, we have to use a properly configured table object. But I haven't figured out an elegant way to do this. bq. With William Yang's patch the short-circuit write optimization will not work because the connection created is not a coprocessor connection. This can be solved easily, see v3 patch. > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH, > PHOENIX-3360-v3.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858891#comment-15858891 ] William Yang commented on PHOENIX-3360: --- bq. What's the problem facing by setting the RPC class at RS level instead of region level? Considering the replication requests, when slave cluster is handling replication requests, slave RSs will send RPC to other RSs in the same cluster. If we set an RS level RPC config, then all replication requests will be handled by the index handlers in those RSs instead of the normal handlers. Usually, a slave cluster is not just a slave, it may also be active for user requests. For example, cluster A is master, B is the slave, and a table T1 is replicated from A to B. Clients' daily access will be handled by A. Then, we may create some phoenix tables on B without replication because the load of cluster B is relatively low. For users of these phoenix tables, they will access cluster B directly. This is not an active-active mode, but cluster B is both slave and 'master' here. Moreover, in an active-active mode, this is a problem that must be solved. Using the slave cluster as a normal cluster might not be standard, but leaving the slave cluster in a low load is also a waste. bq. With William Yang's patch the short-circuit write optimization will not work because the connection created is not a coprocessor connection. What is the 'short-circuit write optimization', by the way? > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-2388) Support pooling Phoenix connections
[ https://issues.apache.org/jira/browse/PHOENIX-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2388: -- Attachment: (was: PHOENIX-2388.patch) > Support pooling Phoenix connections > --- > > Key: PHOENIX-2388 > URL: https://issues.apache.org/jira/browse/PHOENIX-2388 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > Attachments: PHOENIX-2388.patch > > > Frequently user are plugging Phoenix into an ecosystem that pools > connections. It would be possible to implement a pooling mechanism for > Phoenix by creating a delegate Connection that instantiates a new Phoenix > connection when retrieved from the pool and then closes the connection when > returning it to the pool. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-2388) Support pooling Phoenix connections
[ https://issues.apache.org/jira/browse/PHOENIX-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2388: -- Attachment: PHOENIX-2388.patch > Support pooling Phoenix connections > --- > > Key: PHOENIX-2388 > URL: https://issues.apache.org/jira/browse/PHOENIX-2388 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > Attachments: PHOENIX-2388.patch > > > Frequently user are plugging Phoenix into an ecosystem that pools > connections. It would be possible to implement a pooling mechanism for > Phoenix by creating a delegate Connection that instantiates a new Phoenix > connection when retrieved from the pool and then closes the connection when > returning it to the pool. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-2388) Support pooling Phoenix connections
[ https://issues.apache.org/jira/browse/PHOENIX-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2388: -- Attachment: PHOENIX-2388.patch This is a very simple connection pool. Do u have time to have a look at it, [~jamestaylor]? As it is so simple that I thought it should have more features. > Support pooling Phoenix connections > --- > > Key: PHOENIX-2388 > URL: https://issues.apache.org/jira/browse/PHOENIX-2388 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor > Attachments: PHOENIX-2388.patch > > > Frequently user are plugging Phoenix into an ecosystem that pools > connections. It would be possible to implement a pooling mechanism for > Phoenix by creating a delegate Connection that instantiates a new Phoenix > connection when retrieved from the pool and then closes the connection when > returning it to the pool. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852818#comment-15852818 ] William Yang commented on PHOENIX-3360: --- PHOENIX-3360-v2.patch attached, [~jamestaylor]. In this version, only the mutations originated from the HTableInterface created by CoprocessorHTableFactory will have high priority. I've read PHOENIX-3271, unfortunately, this patch will break the assumption that all cross RS calls are made with higher priority, as we didn't change the coprocessor env's configuration, we just created a new hbase connection with specific configurations. The optimal way to make things right is to use separate handler queues for read and write. Sharing the same handler queue with index requests might not be the best idea, because if there are many UPSERT SELECTs, index updates might not have enough threads to be executed. And this will slow down the mutations of data table. On the other hand, separate RW handler queues are not always configured and we cannot force users to config this. In this case, we still have to make UPSERT SELECT's batch mutations have higher priority. Later I'll attach a V3 patch to fix this. WDYT, [~jamestaylor] > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-3360: -- Attachment: PHOENIX-3360-v2.PATCH > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch, PHOENIX-3360-v2.PATCH > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851023#comment-15851023 ] William Yang edited comment on PHOENIX-3360 at 2/3/17 4:08 AM: --- I don't think that current patch has solved the problem because it is still setting the RPC class in RegionServer level instead of region level. We've encountered this problem too, our resolution is simple. The only thing we need to do is to set specified priority ONLY for index updates that originated from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects to do index updates in current region server. If we can make these HTableInterface objects have this RPC config, then all index update requests will have the right priority and will not interfere with other RS-RS requests. Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table regions init, then each index region has its own CoprocessorHTableFactory object. We need to do : * create a global CoprocessorHTableFactory object in getDefaultDelegateHTableFactory(), thus all data table regions will share the same CoprocessorHTableFactory. * in CoprocessorHTableFactory's ctor, create an HBase connection configured with our RPC controller class. * in CoprocessorHTableFactory#getTable(), get the table from the connection created in ctor instead of calling CoprocessorEnvironment#getTable() The only target here is to make sure all HTableInterface objects for updating index tables are created from the specified connection. I've done this and it works fine. was (Author: yhxx511): I don't think that current patch has solved the problem because it is still setting the RPC class in RegionServer level instead of region level. We've encountered this problem too, our resolution is simple. The only thing we need to do is to set specified priority ONLY for index updates that originated from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects to do index updates in current region server. If we can make these HTableInterface objects have this RPC config, then all index update requests will have the right priority and will not interfere with other RS-RS requests. Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table regions init, then each index region has its own CoprocessorHTableFactory object. We need to do : * create a single HBase connection for current RS and this connection is configured with this RPC class * all data table regions get HTableInterface from this connection instead of calling CoprocessorEnvironment#getTable() before doing a batch mutations to their index regions. You can create this connection in getDefaultDelegateHTableFactory() or just let all regions share the same CoprocessorHTableFactory object. No matter which way you choose, the only target here is to make sure all HTableInterface objects for updating index tables are created from the specified connection. In this case, CoprocessorHTableFactory is no longer a suitable name as we don't get table from coprocessor env, but from an explicit connection object. I've done this and it works fine. > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by
[jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
[ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851023#comment-15851023 ] William Yang commented on PHOENIX-3360: --- I don't think that current patch has solved the problem because it is still setting the RPC class in RegionServer level instead of region level. We've encountered this problem too, our resolution is simple. The only thing we need to do is to set specified priority ONLY for index updates that originated from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects to do index updates in current region server. If we can make these HTableInterface objects have this RPC config, then all index update requests will have the right priority and will not interfere with other RS-RS requests. Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table regions init, then each index region has its own CoprocessorHTableFactory object. We need to do : * create a single HBase connection for current RS and this connection is configured with this RPC class * all data table regions get HTableInterface from this connection instead of calling CoprocessorEnvironment#getTable() before doing a batch mutations to their index regions. You can create this connection in getDefaultDelegateHTableFactory() or just let all regions share the same CoprocessorHTableFactory object. No matter which way you choose, the only target here is to make sure all HTableInterface objects for updating index tables are created from the specified connection. In this case, CoprocessorHTableFactory is no longer a suitable name as we don't get table from coprocessor env, but from an explicit connection object. I've done this and it works fine. > Secondary index configuration is wrong > -- > > Key: PHOENIX-3360 > URL: https://issues.apache.org/jira/browse/PHOENIX-3360 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Rajeshbabu Chintaguntla >Priority: Critical > Fix For: 4.10.0 > > Attachments: PHOENIX-3360.patch > > > IndexRpcScheduler allocates some handler threads and uses a higher priority > for RPCs. The corresponding IndexRpcController is not used by default as it > is, but used through ServerRpcControllerFactory that we configure from Ambari > by default which sets the priority of the outgoing RPCs to either metadata > priority, or the index priority. > However, after reading code of IndexRpcController / ServerRpcController it > seems that the IndexRPCController DOES NOT look at whether the outgoing RPC > is for an Index table or not. It just sets ALL rpc priorities to be the index > priority. The intention seems to be the case that ONLY on servers, we > configure ServerRpcControllerFactory, and with clients we NEVER configure > ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We > configure ServerRpcControllerFactory from Ambari, which in affect makes it so > that ALL rpcs from Phoenix are only handled by the index handlers by default. > It means all deadlock cases are still there. > The documentation in https://phoenix.apache.org/secondary_indexing.html is > also wrong in this sense. It does not talk about server side / client side. > Plus this way of configuring different values is not how HBase configuration > is deployed. We cannot have the configuration show the > ServerRpcControllerFactory even only for server nodes, because the clients > running on those nodes will also see the wrong values. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3498) Query with index failed when query back to data table with desc PK column
[
https://issues.apache.org/jira/browse/PHOENIX-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675675#comment-15675675
]
William Yang commented on PHOENIX-3498:
---
This should be a very old bug, I think the reason this bug had been hidden for
so long is: in most cases, we will use covered index so that 'query back to
data table' will never happen. In this case, even if DESC pk column is very
common, it will be hard to meet both of the conditions that will trigger this
problem.
> Query with index failed when query back to data table with desc PK column
> -
>
> Key: PHOENIX-3498
> URL: https://issues.apache.org/jira/browse/PHOENIX-3498
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Assignee: William Yang
> Attachments: PHOENIX-3498.patch
>
>
> This bug can be re-pro against two conditions:
> 1. select must hit index table and have to query back to data table
> 2. at least one of the data table's PK column is in DESC order
> see the following SQLs:
> {code}
> create table tt (p1 integer not null, p2 integer not null, a integer, b
> integer constraint pk primary key (p1,p2));
> create index tti on tt (a);
> upsert into tt values (0, 1, 2, 3);
> select /*+index(tt tti)*/ b from tt where a = 2; // will query back
> to data table
> //this SELECT works fine, will return b=3
> {code}
> if we declare ether p1/p2 as DESC, then the same SELECT will return nothing.
> {code}
> create table tt (p1 integer not null, p2 integer not null, a integer, b
> integer constraint pk primary key (p1 desc, p2));
> create index tti on tt (a);
> upsert into tt values (0, 1, 2, 3);
> select /*+index(tt tti)*/ b from tt where a = 2; // return nothing
> {code}
> if p1 is not DESC, but p2 is, SELECT will fail too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3498) Query with index failed when query back to data table with desc PK column
[
https://issues.apache.org/jira/browse/PHOENIX-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Yang updated PHOENIX-3498:
--
Attachment: PHOENIX-3498.patch
> Query with index failed when query back to data table with desc PK column
> -
>
> Key: PHOENIX-3498
> URL: https://issues.apache.org/jira/browse/PHOENIX-3498
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Assignee: William Yang
> Attachments: PHOENIX-3498.patch
>
>
> This bug can be re-pro against two conditions:
> 1. select must hit index table and have to query back to data table
> 2. at least one of the data table's PK column is in DESC order
> see the following SQLs:
> {code}
> create table tt (p1 integer not null, p2 integer not null, a integer, b
> integer constraint pk primary key (p1,p2));
> create index tti on tt (a);
> upsert into tt values (0, 1, 2, 3);
> select /*+index(tt tti)*/ b from tt where a = 2; // will query back
> to data table
> //this SELECT works fine, will return b=3
> {code}
> if we declare ether p1/p2 as DESC, then the same SELECT will return nothing.
> {code}
> create table tt (p1 integer not null, p2 integer not null, a integer, b
> integer constraint pk primary key (p1 desc, p2));
> create index tti on tt (a);
> upsert into tt values (0, 1, 2, 3);
> select /*+index(tt tti)*/ b from tt where a = 2; // return nothing
> {code}
> if p1 is not DESC, but p2 is, SELECT will fail too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3498) Query with index failed when query back to data table with desc PK column
William Yang created PHOENIX-3498:
-
Summary: Query with index failed when query back to data table
with desc PK column
Key: PHOENIX-3498
URL: https://issues.apache.org/jira/browse/PHOENIX-3498
Project: Phoenix
Issue Type: Bug
Reporter: William Yang
Assignee: William Yang
This bug can be re-pro against two conditions:
1. select must hit index table and have to query back to data table
2. at least one of the data table's PK column is in DESC order
see the following SQLs:
{code}
create table tt (p1 integer not null, p2 integer not null, a integer, b integer
constraint pk primary key (p1,p2));
create index tti on tt (a);
upsert into tt values (0, 1, 2, 3);
select /*+index(tt tti)*/ b from tt where a = 2; // will query back to
data table
//this SELECT works fine, will return b=3
{code}
if we declare ether p1/p2 as DESC, then the same SELECT will return nothing.
{code}
create table tt (p1 integer not null, p2 integer not null, a integer, b integer
constraint pk primary key (p1 desc, p2));
create index tti on tt (a);
upsert into tt values (0, 1, 2, 3);
select /*+index(tt tti)*/ b from tt where a = 2; // return nothing
{code}
if p1 is not DESC, but p2 is, SELECT will fail too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3335) Improve documention of unsigned_long type mapping
[ https://issues.apache.org/jira/browse/PHOENIX-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528692#comment-15528692 ] William Yang commented on PHOENIX-3335: --- Build.sh failed on my MAC and I'm still trying to make it work. Before this, you can have a glimpse of the changes of MD file. [~giacomotaylor] > Improve documention of unsigned_long type mapping > - > > Key: PHOENIX-3335 > URL: https://issues.apache.org/jira/browse/PHOENIX-3335 > Project: Phoenix > Issue Type: Task >Affects Versions: 4.8.0 >Reporter: Kui Xiang > Attachments: PHOENIX-3335.patch > > > when i use the function of increament in hbase 2.0.x > push a value like '\x00\x00\x00\x00\x00\x00\x00\x01' in hbase > then i create pheonix table like: > > create table click_pv(pk varchar primary key,"default"."min_time" > VARCHAR,"default"."pid" VARCHAR,"default"."pv" BIGINT); > > the pv column 's type use BIGINT will mapping the value to > -9223372036854775805 > and when i use UNSIGNED_LONG type ,it will works ok. > it looks a little strange.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3335) Improve documention of unsigned_long type mapping
[ https://issues.apache.org/jira/browse/PHOENIX-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-3335: -- Attachment: PHOENIX-3335.patch > Improve documention of unsigned_long type mapping > - > > Key: PHOENIX-3335 > URL: https://issues.apache.org/jira/browse/PHOENIX-3335 > Project: Phoenix > Issue Type: Task >Affects Versions: 4.8.0 >Reporter: Kui Xiang > Attachments: PHOENIX-3335.patch > > > when i use the function of increament in hbase 2.0.x > push a value like '\x00\x00\x00\x00\x00\x00\x00\x01' in hbase > then i create pheonix table like: > > create table click_pv(pk varchar primary key,"default"."min_time" > VARCHAR,"default"."pid" VARCHAR,"default"."pv" BIGINT); > > the pv column 's type use BIGINT will mapping the value to > -9223372036854775805 > and when i use UNSIGNED_LONG type ,it will works ok. > it looks a little strange.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3335) long type in hbase mapping pheonix type error
[ https://issues.apache.org/jira/browse/PHOENIX-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525803#comment-15525803 ] William Yang edited comment on PHOENIX-3335 at 9/27/16 11:17 AM: - This is exactly how it works. see http://phoenix.apache.org/language/datatypes.html BIGINT is a signed data type. As you know, the first bit of a negative value is 1, but positive value 0. So negative values will be 'greater than' positive values in dictionary order. In order to let all values sorted in the right way, PHOENIX flip the first bit for signed values, so that negative values will sort before positive values. But for unsigned types, this is not needed. So, this is not a bug. If you want to map hbase table to phoenix, make sure that your are using unsigned types for integers. As so many people have encountered this problem, I think we should add a explicit and detailed explanation here http://phoenix.apache.org/faq.html#How_I_map_Phoenix_table_to_an_existing_HBase_table [~giacomotaylor] was (Author: yhxx511): This is exactly how it works. see http://phoenix.apache.org/language/datatypes.html BIGINT is a signed data type. As you know, the first bit of a negative value is 1, but positive value 0. So negative values will be 'greater than' positive values in dictionary order. In order to let all values sorted in the right way, PHOENIX flip the first bit for signed values, so that negative values will sort before positive values. But for unsigned types, this is not needed. So, this is not a bug. If you want to map hbase table to phoenix, make sure that your are using unsigned types for integers. > long type in hbase mapping pheonix type error > - > > Key: PHOENIX-3335 > URL: https://issues.apache.org/jira/browse/PHOENIX-3335 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.8.0 >Reporter: Kui Xiang > > when i use the function of increament in hbase 2.0.x > push a value like '\x00\x00\x00\x00\x00\x00\x00\x01' in hbase > then i create pheonix table like: > > create table click_pv(pk varchar primary key,"default"."min_time" > VARCHAR,"default"."pid" VARCHAR,"default"."pv" BIGINT); > > the pv column 's type use BIGINT will mapping the value to > -9223372036854775805 > and when i use UNSIGNED_LONG type ,it will works ok. > it looks a little strange.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3335) long type in hbase mapping pheonix type error
[ https://issues.apache.org/jira/browse/PHOENIX-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525803#comment-15525803 ] William Yang commented on PHOENIX-3335: --- This is exactly how it works. see http://phoenix.apache.org/language/datatypes.html BIGINT is a signed data type. As you know, the first bit of a negative value is 1, but positive value 0. So negative values will be 'greater than' positive values in dictionary order. In order to let all values sorted in the right way, PHOENIX flip the first bit for signed values, so that negative values will sort before positive values. But for unsigned types, this is not needed. So, this is not a bug. If you want to map hbase table to phoenix, make sure that your are using unsigned types for integers. > long type in hbase mapping pheonix type error > - > > Key: PHOENIX-3335 > URL: https://issues.apache.org/jira/browse/PHOENIX-3335 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.8.0 >Reporter: Kui Xiang > > when i use the function of increament in hbase 2.0.x > push a value like '\x00\x00\x00\x00\x00\x00\x00\x01' in hbase > then i create pheonix table like: > > create table click_pv(pk varchar primary key,"default"."min_time" > VARCHAR,"default"."pid" VARCHAR,"default"."pv" BIGINT); > > the pv column 's type use BIGINT will mapping the value to > -9223372036854775805 > and when i use UNSIGNED_LONG type ,it will works ok. > it looks a little strange.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3328) Skip scan optimization failed for multi pk columns
[
https://issues.apache.org/jira/browse/PHOENIX-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519309#comment-15519309
]
William Yang commented on PHOENIX-3328:
---
The patch handles PK ranges OR operation only for the FIRST PK column. If there
are other PK columns in the WHERE clause, it cannot handle. For example:
{code:sql}
where (pk1 = 5 and pk2 > 10 and pk2 < 20) or (pk = 6 and pk2 > 60 and pk2 <
80);
{code}
this WHERE clause indicate four ranges:
5 10
5 20
6 60
6 80
which can be optimized to SkipScanFilter for 4 ranges [5,10] - [6, 80]
But this does not always happen, see the following case:
{code:sql}
where (pk1 = 5 and pk2 > 10 and pk2 < 20) or (pk1 = 1 and pk2 > 60 and pk2 <
80);
{code}
we have to do two separate skip scan range scans instead of one, because the
value of PK1 in the RHS of OR is smaller than the LHS', but range of PK2 is
greater than the left. I think it's OK for us to be able to handle the first
PK only.
> Skip scan optimization failed for multi pk columns
> --
>
> Key: PHOENIX-3328
> URL: https://issues.apache.org/jira/browse/PHOENIX-3328
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Priority: Minor
> Attachments: PHOENIX-3328.patch
>
>
> {code:sql}
> create table t1 (pk integer primary key, a integer);
> create table t2 (pk1 integer not null, pk2 integer not null, a integer
> constraint pk primary key (pk1, pk2));
> explain select * from t1 where (pk > 10 and pk < 20) or (pk > 30 and pk < 40);
> explain select * from t2 where (pk1 > 10 and pk1 < 20) or (pk1 > 30 and pk1 <
> 40);
> {code}
> The first SELECT statement will use skip scan filter for two ranges, so as
> the second one. But actually the WhereOptimizer failed doing so and using a
> full table scan instead. This happens for tables have multi PK columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3328) Skip scan optimization failed for multi pk columns
[
https://issues.apache.org/jira/browse/PHOENIX-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Yang updated PHOENIX-3328:
--
Attachment: PHOENIX-3328.patch
> Skip scan optimization failed for multi pk columns
> --
>
> Key: PHOENIX-3328
> URL: https://issues.apache.org/jira/browse/PHOENIX-3328
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Priority: Minor
> Attachments: PHOENIX-3328.patch
>
>
> {code:sql}
> create table t1 (pk integer primary key, a integer);
> create table t2 (pk1 integer not null, pk2 integer not null, a integer
> constraint pk primary key (pk1, pk2));
> explain select * from t1 where (pk > 10 and pk < 20) or (pk > 30 and pk < 40);
> explain select * from t2 where (pk1 > 10 and pk1 < 20) or (pk1 > 30 and pk1 <
> 40);
> {code}
> The first SELECT statement will use skip scan filter for two ranges, so as
> the second one. But actually the WhereOptimizer failed doing so and using a
> full table scan instead. This happens for tables have multi PK columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3328) Skip scan optimization failed for multi pk columns
William Yang created PHOENIX-3328:
-
Summary: Skip scan optimization failed for multi pk columns
Key: PHOENIX-3328
URL: https://issues.apache.org/jira/browse/PHOENIX-3328
Project: Phoenix
Issue Type: Bug
Reporter: William Yang
Priority: Minor
{code:sql}
create table t1 (pk integer primary key, a integer);
create table t2 (pk1 integer not null, pk2 integer not null, a integer
constraint pk primary key (pk1, pk2));
explain select * from t1 where (pk > 10 and pk < 20) or (pk > 30 and pk < 40);
explain select * from t2 where (pk1 > 10 and pk1 < 20) or (pk1 > 30 and pk1 <
40);
{code}
The first SELECT statement will use skip scan filter for two ranges, so as the
second one. But actually the WhereOptimizer failed doing so and using a full
table scan instead. This happens for tables have multi PK columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425796#comment-15425796 ] William Yang commented on PHOENIX-3185: --- I'm afraid we already has an issue and a applicable patch for this bug: (PHOENIX-930)[https://issues.apache.org/jira/browse/PHOENIX-930] > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Fix For: 4.8.1 > > Attachments: image1.png, image2.png, image3.png, > phoenix_duplicate_column_check.patch > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-930) duplicated columns cause query exception and drop table exception
[ https://issues.apache.org/jira/browse/PHOENIX-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309066#comment-15309066 ] William Yang commented on PHOENIX-930: -- [~junegunn], Why not move hasColumnWithSameNameAndFamily to CreateTableCompiler and check for duplicate columns at compilation time (CreateTableCompiler#compile())? I think this is more straightforward. > duplicated columns cause query exception and drop table exception > - > > Key: PHOENIX-930 > URL: https://issues.apache.org/jira/browse/PHOENIX-930 > Project: Phoenix > Issue Type: Bug >Reporter: wangkai >Assignee: Junegunn Choi > Fix For: 4.8.0 > > Attachments: PHOENIX-930, PHOENIX-930.patch > > > when I create table like this: "create table test (id varchar not null > primary key, f.name varchar, f.email varchar, f.email varchar)", this will > cause an org.apache.phoenix.schema.ColumnAlreadyExistsException, but the > table is successful created. > Then I run a query like "select * from test", an exception is threw: > Caused by: java.lang.ArrayIndexOutOfBoundsException: 3 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:283) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:216) > at > org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:209) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:443) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:254) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:1077) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1023) > ... 10 more > then I try to drop the table: "drop table test", an exception is also threw: > Caused by: java.lang.ArrayIndexOutOfBoundsException: 3 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:283) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:216) > at > org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:209) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:443) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:254) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:1077) > at > org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1023) > ... 10 more > So I have to drop SYSTEM.CATALOG, SYSTEM.SEQUENCE from hbase shell…… > The ArrayIndexOutOfBoundsException is threw out because the position of > f.email column in CATALOG table is not correct. I think it's better to check > columns before creating table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2936) Missing antlr runtime on server side after PHOENIX-2908
[
https://issues.apache.org/jira/browse/PHOENIX-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299268#comment-15299268
]
William Yang commented on PHOENIX-2936:
---
I am really sorry for this. I didn't mean to remove all antlr dependency on
server side, all i want is to remove antlr but keep antlr-runtime.
I mistook 'org.antlr:antlr*' for antlr only, but actually it contains both
antlr and antlr-runtime.
> Missing antlr runtime on server side after PHOENIX-2908
> ---
>
> Key: PHOENIX-2936
> URL: https://issues.apache.org/jira/browse/PHOENIX-2936
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: PHOENIX-2936.patch
>
>
> During PHOENIX-2908 antlr was completely removed from server jar. That's was
> a bad idea, since runtime is still required for indexes:
> {noformat}
> 2016-05-24 11:40:53,596 ERROR [StoreFileOpenerThread-L#0-1]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.hadoop.hbase.regionserver.IndexHalfStoreFileReaderGenerator threw
> java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
> java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1359)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1413)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2327)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2276)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2276)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:233)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:135)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:326)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:313)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:303)
> at
> org.apache.hadoop.hbase.regionserver.IndexHalfStoreFileReaderGenerator.preStoreFileReaderOpen(IndexHalfStoreFileReaderGenerator.java:145)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$64.call(RegionCoprocessorHost.java:1580)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1712)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preStoreFileReaderOpen(RegionCoprocessorHost.java:1575)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:246)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:399)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:504)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:494)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:653)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:118)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:520)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:517)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293120#comment-15293120 ] William Yang commented on PHOENIX-2908: --- I think the test result is OK. I checked some other builds, these 3 cases fail all the time. [~jamestaylor] what's your opinion? > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeBothAntlr_v3.patch, > PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: PHOENIX-2908_removeBothAntlr_v3.patch > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeBothAntlr_v3.patch, > PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: (was: PHOENIX-2908_removeBothAntlr_v2.patch) > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: PHOENIX-2908_removeBothAntlr_v2.patch > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292504#comment-15292504 ] William Yang commented on PHOENIX-2908: --- I have updated the patch for master branch, but I don't know how to trigger HUDSON integrated test. Can someone help me? > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: PHOENIX-2908_removeBothAntlr_v2.patch > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeBothAntlr_v2.patch, PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292495#comment-15292495 ] William Yang commented on PHOENIX-2908: --- This patch remove dependencies for antlr and limit us to antlr-runtime only. But for the master branch, the patch needs a little modification because antlr-runtime 3.5.2 'optionally' depends on stringtemplate so we don't have to exclude it explicitly. > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288170#comment-15288170 ] William Yang commented on PHOENIX-2908: --- Yes, the master branch doesn't have this issue. Because master uses antlr-runtime 3.5.2, not 3.5 in phoenix-4.6.0. In the pom.xml of antlr-runtime 3.5.2, dependency of stringtemplate is 'optional', which is not in 3.5. As phoenix does not use stringtemplate, so it is not included any more. But stringtemplate (3.2.1) still depends on antlr 2.7.7, both in antlr-runtime 3.5.2 and 3.5. So if some one uses stringtemplate in the future, he will still depend both antlr versions. Never mind, it won't happen. Up to now, there is only one thing that matters: can we only depend on antlr-runtime (exclude antlr 3.5.2) ? > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Fix For: 4.8.0 > > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: PHOENIX-2908_removeBothAntlr.patch There's also an option 2: remove both antlr 3.5 and 2.7.7. Even though i have run the tests, but I'm not definitely sure that it'll be safe to remove so many things. Especially the query-server mode. So, [~jamestaylor], choose one. > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Attachments: PHOENIX-2908_removeBothAntlr.patch, > PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
[ https://issues.apache.org/jira/browse/PHOENIX-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Yang updated PHOENIX-2908: -- Attachment: PHOENIX-2908_removeOnlyAntlr2.7.7.patch option 1: The safest way, only remove antlr2.7.7 and keep antlr 3.5, also the minimum modification > phoenix-core depends on both antlr 3.5 and antlr 2.7.7 > -- > > Key: PHOENIX-2908 > URL: https://issues.apache.org/jira/browse/PHOENIX-2908 > Project: Phoenix > Issue Type: Bug >Reporter: William Yang >Priority: Minor > Labels: build > Attachments: PHOENIX-2908_removeOnlyAntlr2.7.7.patch > > > when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we > depends on both antlr-3.5.jar and antlr-2.7.7.jar. > [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT > [INFO] +- org.antlr:antlr:jar:3.5:compile > [INFO] | \- org.antlr:ST4:jar:4.0.7:compile > [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile > [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile > [INFO] | \- antlr:antlr:jar:2.7.7:compile > Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it > contains both antlr 3.5 and 2.7.7 too. Fortunately they have different > package name and will not cause unexpected problems. So it does not have any > negative effects, but it is a existed problem. > https://github.com/antlr/stringtemplate4/issues/51 > https://github.com/antlr/antlr3/issues/75 > It seems that it is a known issue and has not been solved and never will be > for antlr 3. I don't know whether it still exists in antlr 4. > I searched the source code and generated source code and i cannot find > explicit import for classes in antlr-*.jar, but only antlr-runtime instead. > Is antlr 3.5 necessary for phoenix? > I did the following things: > 1. remove antlr 3.5 dependency from pom > 2. add exclusion of stringtemplate for antlr-runtime (so it will not have > the transit dependency of antlr 2.7.7) > > Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs > passed. Everything seems to be fine. > Moreover, I added java parameter '-verbose:class' for sqlline.py and see > which antlr class is loaded. The result shows that only classes in > antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. > So, I think it might be safe to remove dependency of antlr 3.5 and exclude > stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2908) phoenix-core depends on both antlr 3.5 and antlr 2.7.7
William Yang created PHOENIX-2908: - Summary: phoenix-core depends on both antlr 3.5 and antlr 2.7.7 Key: PHOENIX-2908 URL: https://issues.apache.org/jira/browse/PHOENIX-2908 Project: Phoenix Issue Type: Bug Reporter: William Yang Priority: Minor when executing 'mvn dependency:tree -pl=phoenix-core'. It shows that we depends on both antlr-3.5.jar and antlr-2.7.7.jar. [INFO] org.apache.phoenix:phoenix-core:jar:4.6-adh6u-SNAPSHOT [INFO] +- org.antlr:antlr:jar:3.5:compile [INFO] | \- org.antlr:ST4:jar:4.0.7:compile [INFO] +- org.antlr:antlr-runtime:jar:3.5:compile [INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile [INFO] | \- antlr:antlr:jar:2.7.7:compile Moreover, in the final jar file phoenix-4.6.0-HBase-1.1-client.jar, it contains both antlr 3.5 and 2.7.7 too. Fortunately they have different package name and will not cause unexpected problems. So it does not have any negative effects, but it is a existed problem. https://github.com/antlr/stringtemplate4/issues/51 https://github.com/antlr/antlr3/issues/75 It seems that it is a known issue and has not been solved and never will be for antlr 3. I don't know whether it still exists in antlr 4. I searched the source code and generated source code and i cannot find explicit import for classes in antlr-*.jar, but only antlr-runtime instead. Is antlr 3.5 necessary for phoenix? I did the following things: 1. remove antlr 3.5 dependency from pom 2. add exclusion of stringtemplate for antlr-runtime (so it will not have the transit dependency of antlr 2.7.7) Then, both antlr 3.5 and antlr 2.7.7 disappeared. After that, all IT/UTs passed. Everything seems to be fine. Moreover, I added java parameter '-verbose:class' for sqlline.py and see which antlr class is loaded. The result shows that only classes in antlr-runtime had been loaded, not any in antlr 3.5/2.7.7. So, I think it might be safe to remove dependency of antlr 3.5 and exclude stringtemplate from antlr-runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2520) Create DDL property for metadata update frequency
[
https://issues.apache.org/jira/browse/PHOENIX-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088634#comment-15088634
]
William Yang commented on PHOENIX-2520:
---
Updating the meta data periodically has a problem that if some connection
changes the table meta in the middle of a period, then current connection will
have to wait for the rest of the period before it can update the meta. Even if
it knows that there's a new version of meta it cannot pick it up until current
meta is 'old' enough. So I propose the following solution:
update meta data once(at the first time creating the TableRef object) and then
update on demand.
1. table meta change can be categroized into 3 types: 'ALWAYS' (current
behaviour), 'NEVER', 'RARELY'. For the last two types, we should only update
table meta at the first time we access the table and then update by demand.
2. If user add a new column, and execute a SQL contains the new column with an
old connection, SQL compilation will fail for MetaDataEntityNotFoundException,
so we know that it is time to update table meta explicitly and retry compiling.
3. The defect of this solution is that it cannot handle column deletion. If
some connections remove a column, the old connections can still access the
deleted column until it get re-opened. If user cannot accept this behaviour he
should choose the 'ALWAYS' type. So a switch should be introduced.
new configurations in hbase-site.xml:
phoenix.functions.preferMetaCache.enabled
true
phoenix.functions.preferMetaCache.enabled.your_table_name
false
I introduce two level configs here: global config and table-level config. If
you enable the 'preferMetaCache' property, table meta will be updated once and
then update by demand. Otherwise, it will update table meta every time (current
behaviour). Users are responsible to decide which mode to use.
Since 'NEVER' and 'RARELY' and 'add column' are most common cases, this
solution will work well for most scenarios. And for those minority cases you
can choose 'ALWAYS'.
More details, see patch 'preferMetaCache.patch'.
> Create DDL property for metadata update frequency
> -
>
> Key: PHOENIX-2520
> URL: https://issues.apache.org/jira/browse/PHOENIX-2520
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: preferMetaCache.patch
>
>
> On the client-side, Phoenix pings the server when a query is compiled to
> confirm that the client has the most up-to-date metadata for the table being
> queried. For some tables that are known to not change, this RPC is wasteful.
> We can allow a property such as {{UPDATE_METADATA_CACHE_FREQUENCY_MS}} to be
> specified a time to wait before checking with the server to see if the
> metadata has changed. This could be specified in the CREATE TABLE call and
> stored in the SYSTEM.CATALOG table header row. By default the value could be
> 0 which would keep the current behavior. Tables that never change could use
> Long.MAX_VALUE. Potentially we could allow 'ALWAYS' and 'NEVER' values for
> convenience.
> Proposed implementation:
> - add {{public long getAge()}} method to {{PTableRef}}.
> - when setting lastAccessTime, also store System.currentMillis() to new
> {{setAccessTime}} private member variable
> - the getAge() would return {{System.currentMillis() - setAccessTime}}
> - code in MetaDataClient would prevent call to server if age <
> {{UPDATE_METADATA_CACHE_FREQUENCY_MS}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-2520) Create DDL property for metadata update frequency
[
https://issues.apache.org/jira/browse/PHOENIX-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088634#comment-15088634
]
William Yang edited comment on PHOENIX-2520 at 1/8/16 3:25 AM:
---
Updating the meta data periodically has a problem that if some connection
changes the table meta in the middle of a period, then current connection will
have to wait for the rest of the period before it can update the meta. Even if
it knows that there's a new version of meta it cannot pick it up until current
meta is 'old' enough. So I propose the following solution:
update meta data once(at the first time creating the TableRef object) and then
update on demand.
1. table meta change can be categroized into 3 types: 'ALWAYS' (current
behaviour), 'NEVER', 'RARELY'. For the last two types, we should only update
table meta at the first time we access the table and then update by demand.
2. If user add a new column, and execute a SQL contains the new column with an
old connection, SQL compilation will fail for MetaDataEntityNotFoundException,
so we know that it is time to update table meta explicitly and retry compiling.
3. The defect of this solution is that it cannot handle column deletion. If
some connections remove a column, the old connections can still access the
deleted column until it get re-opened. If user cannot accept this behaviour he
should choose the 'ALWAYS' type. So a switch should be introduced.
new configurations in hbase-site.xml:
phoenix.functions.preferMetaCache.enabled
true
phoenix.functions.preferMetaCache.enabled.your_table_name
false
I introduce two level configs here: global config and table-level config. If
you enable the 'preferMetaCache' property, table meta will be updated once and
then update by demand. Otherwise, it will update table meta every time (current
behaviour). Users are responsible to decide which mode to use.
Since 'NEVER' and 'RARELY' and 'add column' are most common cases, this
solution will work well for most scenarios. And for those minority cases you
can choose 'ALWAYS'.
More details, see patch 'preferMetaCache.patch'.
Please review, [~jamestaylor]
was (Author: yhxx511):
Updating the meta data periodically has a problem that if some connection
changes the table meta in the middle of a period, then current connection will
have to wait for the rest of the period before it can update the meta. Even if
it knows that there's a new version of meta it cannot pick it up until current
meta is 'old' enough. So I propose the following solution:
update meta data once(at the first time creating the TableRef object) and then
update on demand.
1. table meta change can be categroized into 3 types: 'ALWAYS' (current
behaviour), 'NEVER', 'RARELY'. For the last two types, we should only update
table meta at the first time we access the table and then update by demand.
2. If user add a new column, and execute a SQL contains the new column with an
old connection, SQL compilation will fail for MetaDataEntityNotFoundException,
so we know that it is time to update table meta explicitly and retry compiling.
3. The defect of this solution is that it cannot handle column deletion. If
some connections remove a column, the old connections can still access the
deleted column until it get re-opened. If user cannot accept this behaviour he
should choose the 'ALWAYS' type. So a switch should be introduced.
new configurations in hbase-site.xml:
phoenix.functions.preferMetaCache.enabled
true
phoenix.functions.preferMetaCache.enabled.your_table_name
false
I introduce two level configs here: global config and table-level config. If
you enable the 'preferMetaCache' property, table meta will be updated once and
then update by demand. Otherwise, it will update table meta every time (current
behaviour). Users are responsible to decide which mode to use.
Since 'NEVER' and 'RARELY' and 'add column' are most common cases, this
solution will work well for most scenarios. And for those minority cases you
can choose 'ALWAYS'.
More details, see patch 'preferMetaCache.patch'.
> Create DDL property for metadata update frequency
> -
>
> Key: PHOENIX-2520
> URL: https://issues.apache.org/jira/browse/PHOENIX-2520
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: preferMetaCache.patch
>
>
> On the client-side, Phoenix pings the server when a query is compiled to
> confirm that the client has the most up-to-date metadata for the table being
> queried. For some tables that are known to not change, this RPC is wasteful.
> We can allow a property such as {{UPDATE_METADATA_CACHE_FREQUENCY_MS}} to be
> specified a time to wait before checking with the server to see if the
> metadata has changed. This
[jira] [Updated] (PHOENIX-2520) Create DDL property for metadata update frequency
[
https://issues.apache.org/jira/browse/PHOENIX-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Yang updated PHOENIX-2520:
--
Attachment: preferMetaCache.patch
> Create DDL property for metadata update frequency
> -
>
> Key: PHOENIX-2520
> URL: https://issues.apache.org/jira/browse/PHOENIX-2520
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: preferMetaCache.patch
>
>
> On the client-side, Phoenix pings the server when a query is compiled to
> confirm that the client has the most up-to-date metadata for the table being
> queried. For some tables that are known to not change, this RPC is wasteful.
> We can allow a property such as {{UPDATE_METADATA_CACHE_FREQUENCY_MS}} to be
> specified a time to wait before checking with the server to see if the
> metadata has changed. This could be specified in the CREATE TABLE call and
> stored in the SYSTEM.CATALOG table header row. By default the value could be
> 0 which would keep the current behavior. Tables that never change could use
> Long.MAX_VALUE. Potentially we could allow 'ALWAYS' and 'NEVER' values for
> convenience.
> Proposed implementation:
> - add {{public long getAge()}} method to {{PTableRef}}.
> - when setting lastAccessTime, also store System.currentMillis() to new
> {{setAccessTime}} private member variable
> - the getAge() would return {{System.currentMillis() - setAccessTime}}
> - code in MetaDataClient would prevent call to server if age <
> {{UPDATE_METADATA_CACHE_FREQUENCY_MS}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
