Re: Ignite thick client triggering PME in 2.8.0 ?

[Hemambara]

Hi, can anyone please check and respond on this..appreciate your help in
advance



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite thick client triggering PME in 2.8.0 ?

[Hemambara]

https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood

As per the documentation, if we are using apache ignite thick client 2.8 and
above, it should not trigger PME in server nodes. But it does not look like
it is skipping, I see below logs

INFO  2021-02-01 18:56:20,362 [sys-#45%TRCAP%]
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture:
Received full message, will finish exchange
[node=7b01dd15-f5b3-48c4-8d7a-2765d9de13e8, resVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0]]
INFO  2021-02-01 18:56:20,877 [sys-#45%TRCAP%]
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture:
Finish exchange future [startVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
err=null, rebalanced=true, wasRebalanced=false]

INFO  2021-02-01 18:56:21,424 [exchange-worker-#38%TRCAP%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager:
Skipping rebalancing (no affinity
changes)org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture:
Completed partition exchange


I see "Skipping rebalancing" on GridCachePartition but it says "Completed
partition exchange" in GridDhtPartitions.

What is the difference between GridDhtPartitionsExchangeFuture and
GridCachePartitionExchangeManager. At present our thick clients are taking
more than 2 minutes to connect to servers nodes (8 jvms) and most of the
time has been taken here. Wondering if we skip this
GridDhtPartitionsExchange then it will boost up the connectivity time. Is
there a way to skip it ? and also is it valid to skip it ?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.9 one way client to server communication

[Hemambara]

Can you please check above questions and help me out



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.9 one way client to server communication

[Hemambara]

Thank you for response. If I understand correctly, so thick client
connectivity time is not depending on # of server nodes, it all depends on
PME length ? Can you please elaborate or point me to the resources where I
can understand PME length ? I got few links on how PME works. But sorry I
did not what is PME length. Also as per below reference, ignite thick client
2.8.0 does not trigger any PME. Is this right?

https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.9 one way client to server communication

[Hemambara]

Thank you Stephen and Ilya for your response.

Please find my server and client config and let me know if I a missing
anything which is causing these delays. We have 8 jvm server nodes running
on Ignite 2.8.0 with 4gig each and each node is on separate host. We have 60
client nodes (Ignite 2.8.0) using this same client configuration. We are not
using persistence, it is purely in-memory. Both server and clients are in
same data center. Initially first few clients are able to connect in 1-2
minutes, but as it grows, last clients are taking 5-6 minutes. We really
want to reduce the connectivity time so that we can use map listener. Right
now, it became a road blocker to use any other thick client functionalities. 

Can you please clarify below queries:
1) If client is not part of ring, any reason why it has to trigger PME ?
2) If it just connecting to one server node like thin client, does it
transfer/wait for any additional events before it successfully establish
connection, which is causing delays ?
3) Does a client node wait until all the other server and client
nodes get notified that it joined ? 
4) I am defining cache on client config as well? I think it is not
required. Will it cause any delays during connectivity ?
5) Any plan / any other way to get map listener functionality in thin
client (other than continuous query) ? 


Server config:
---





"/>






























"/>





"/>




localhost
host1:port# 
host2:port#
host3:port#
host4:port#
host5:port#
host6:port#
host7:port#
host8:port#


























Client config
-





























"/>





"/>




localhost
host1:port# 
host2:port#
host3:port#
host4:port#
host5:port#
host6:port#
host7:port#
host8:port#






























--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite 2.9 one way client to server communication

[Hemambara]

I see that ignite 2.9 has added support for one-way thick-client to server
connections. Does it reduce the time taken to connect thick client to server
and provides all thick client functionalities? Does client still be in ring?
Right now we r facing issues with thick client where it is taking more time
to connect especially when we have 60 clients. Switched to thin clients for
now. But we need map listeners. Does upgrading to 2.9 helps reducing long
connection times? Also is there any plan to provide map listeners on thin
clients?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Planning to pick Ignite 13006

[Hemambara]

Hi, I am interested to pick
https://issues.apache.org/jira/browse/IGNITE-13006 and work on it. Please
let me know for any constraints to pick it up. I see fix version on this as
2.10, but I do not see any branch created. Could you please let me know if
we are planning to start 2.10 and is it the right time to pick this jira. If
so which branch I should take it as a base to fork to my local and start
working



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite thin client add column and index dynamically

[Hemambara]

I have seen that if I provide query entities in cache configuration I am able
to query by column. But when I execute ALTER TABLE ADD COLUMN
programmatically its not working with java thin clients where its working
fine with thick clients. Java  Thin clients supports sqlfieldquery then why
not alter table ? DDLs done work with Java thin clients??



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache ignite evolvable object

[Hemambara]

Does it save additional bytes by default or do we have to implement
binarylable, if so do you have any example. 

If it does by default, does that mean, let's say if I send data from new
version node to old version node and send the same data back to new version
node will it preserve thos new fields??



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Apache ignite evolvable object

[Hemambara]

I am using apache ignite 2.8.0. Looking for an option to have evolvable data
model similar to coherence (com.tangosol.io.Evolvable). Do we have any ?
Idea is to save future data if the domain model version is backward
compatible and when the same model is transferred to new version, we can
retrieve fields from future data, with out any data loss



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Custom IgniteConfiguration does not work

[Hemambara]

I am trying to extend IgniteConfiguration class and try to enhance its
properties as I needed them for some custom checks. I have created config
like below and started using Ignition.start()

But when I call GridKernalContext.config() I am not getting
MyCustomIgniteConfiguration instance. I am still getting IgniteConfiguration
reference.






---


This is happening because IgnitionEx has below code in
initializeConfiguration() method. So even if I extend this configuration it
is creating new instance and using that. 

Question is : Is there any reason why it is not honoring what client provide
in config xml ? Seems like a bug to me.

 private IgniteConfiguration initializeConfiguration(IgniteConfiguration
cfg) throws IgniteCheckedException {
IgniteConfiguration myCfg = new IgniteConfiguration(cfg);



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Pause ignite grid communication

[Hemambara]

Once the grid started, I am performing some customized post process setup. It
takes about 10 milliseconds. During this 10 millis I want to pause any grid
communication from other nodes (server/client) like adding any new fields or
queries. Is there any way to pause grid communication once it started for
sometime?? If so once grid state is unpaused will the queries that are on
hold will execute one after another



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to start near cache (a cache with the same name without near cache is already started)

[Hemambara]

I have disabled copyonread and the time taken to getAll() reduced to 1/3.
Earlier it was 300 ms on 10k entries and now it is 100ms. Is disabling
copyonread is fine ???



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How sql works for near cache

[Hemambara]

Is there any way we can get complete keyset or values from a near cache.
Something like cache.keyset()



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

How sql works for near cache

[Hemambara]

If I run sql with out where condition on near caches, query will hit server
node correct ? It wont fetch from near cache in local node. Is this a true
statement?

Ex: select _key from cachename;



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to start near cache (a cache with the same name without near cache is already started)

[Hemambara]

Thank you. I am not getting this error now. But not sure if something wrong
with config. When I call getAll(all keys) on near cache, then the
performance is worse than regular cache. If regular cache getAll() is taking
100ms then near cache getAll() is taking 330 ms. Its almost 3 times. I am
expecting better performance as the near cache is local



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Failed to start near cache (a cache with the same name without near cache is already started)

[Hemambara]

I followed the below post
http://apache-ignite-users.70518.x6.nabble.com/Failed-to-start-near-cache-a-cache-with-the-same-name-without-near-cache-is-already-started-td18322.html
but it did not solve my problem, hence requesting for help. 

I tried two options
a) Start ignite with below xml config and then call
this.ignite.getOrCreateNearCache("MyCache", nearCfg);
b) Start ignite with below xml config and then call
this.ignite.getOrCreateCache("MyNearCache"); and then
this.ignite.getOrCreateNearCache("MyNearCache", nearCfg);

Both are giving error like "Failed to start near cache (a cache with the
same name without near cache is already started)". AS per doc, I just need
to pass existing cache name and pass near cfg to get near cache for client
alone. But either way I am getting exception. Can you provide how can I
create a near cache ONLY for client, but not on server






























--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Can anyone please check above comment and update. Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Hello, I tried to go to internals and override GridQueryProcessor but somehow
I do not feel like clean solution. Just had another thought...whatever code
I have written and created a pull request for, shall is surround it with -D
arg like nested_field_experimental_feature or with better system property
name. So that my logic will be enabled for the guys that need this feature,
else default to existing logic. Any way as it is mentioned as experimental
feature it leaves the testing part to individuals for thorough testing and
whenever you have time you can get back on this.. if this is fine I will
change code and raise pull request



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Benefits of near cache other than get() ?

[Hemambara]

What are the operations,  out of below, that are purely benefited by near
cache. When I tested it other than get() I do not see any benefit of near
cache. All the other calls are going on network. I thought getAll(),
keySet() and entrySet() will be coming from local node, but they are going
to servers from client

put()
get() - local
keySet()
entrySet()
values()
getAll()
putAll()
equalsFilter




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Overriding GridQueryProcessor

[Hemambara]

Hi 

In order to fix the issue that we are discussing @
http://apache-ignite-users.70518.x6.nabble.com/Issue-with-adding-nested-index-dynamically-tc29571.html

I found a workaround to override GridQueryProcessor and use
CustomGridQueryProcessor which extends GridQueryProcessor and override few
functionality. To do that I am getting hold of KernalContext and stopping
existing GridQueryProcessor that has been already started during startup and
adding my CustomGridQueryProcessor. But my worry is when Ignite instance is
created it is open for discovery, traffic will be started and existing
GridQueryProcessor might get traffic. But if stop and start meanwhile if we
get any message, I might loose it. 

My question is :
1) Is there any other better way to override GridQueryProcessor ? (or)
2) Is there any way we can stop traffic in local node for that moment until
new CustomGridQueryProcessor has been started and restart it ?


List igniteLocalGrids = Ignition.allGrids();
for(Ignite igniteLocalGrid : igniteLocalGrids) {
GridKernalContextImpl kernalContext =
((GridKernalContextImpl)((IgniteKernal)igniteLocalGrid).context());
kernalContext.query().stop(true);
CustomGridQueryProcessor processor = new
CustomGridQueryProcessor(kernalContext);
processor.start();

GridCacheProcessor gridCacheProcessor = kernalContext.cache();
for (final IgniteInternalCache cache :
gridCacheProcessor.caches()) {
GridCacheContext cctx = cache.context();
DynamicCacheDescriptor desc =
gridCacheProcessor.cacheDescriptor(cctx.name());
processor.onCacheStart0(cctx, desc.schema());
}

processor.onCacheKernalStart();
kernalContext.add(processor, true);
}






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Okay, so the issue you are facing with is incorrect data type which is valid,
so its not an issue then.

Yes agreed that it requires more testing, but I feel the fix that is going
in, is safe and good to do. This fix is really important for us to proceed
further. I have tested few other scnearios and its working fine for me. Is
there any way we can plan to merge ?  or you do not want to do it now ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

I did not face any issue. Its working fine for me. Can you share your code
and exception that you are getting

I tried like below and it worked for me.
((Person)cache.get(1)).address.community)





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Hello Ivan, please let me know if you get a chance to check the code and
merge it..



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Sure. Have created one https://github.com/apache/ignite/pull/7016

Can you please check this. It has all the changes at one place. please let
me know for any issues. Thank you.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Ohh yes you are right. Without the fix, it is failing to query cache. With
the other approach that I have attached earlier it failed to join node,
there I have CacheConfiguration in xml file for both server and client node.
Not sure if that does matters. Either way it is failing

Also this is my bad, my apologies, that I forget to check in one change in
GridQueryProcessor.java where it is using QueryBinaryProperty constructor,
instead I am using QUeryUtils.buildBinaryProperty to build it. Now I have
checked in all the changes along with test cases. 

Can you please review pull requests:
https://github.com/apache/ignite/pull/7008 - Change in QueryUtils
https://github.com/apache/ignite/pull/7009 - Test case for QueryUtilsTest
https://github.com/apache/ignite/pull/7010 - Change in
GridQueryProcessor.java
https://github.com/apache/ignite/pull/7011 - Test case to make sure it is
adding nested field

I could not merge pull request as it is saying that only users with "Write
access" can merge pull requests. Can you please merge these pull requests
and review. Please let me know if you see any issues. 

Appreciate your help on this.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Result should not be same, could you please share your logs. Couple of issues
here.

1) *With out fix *- I got null instead of "Address.street" value
[[1, Person{name='john', address=Address{street='baker', number=221}}, john,
Address{street='baker', number=221}, *null*]]

   *With fix* 
   [[1, Person{name='john', address=Address{street='baker', number=221}},
john, Address{street='baker', number=221},* baker*]]

2) Also you are trying with only one node. Could you please try to *bring
another node*. Using below code. This one will just start another node and
try to join existing. *With out fix - it cannot join and with fix - it will
join*


@Test
public void test(){
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi()
.setIpFinder(new
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500..47509")));

IgniteConfiguration igniteConfig = new
IgniteConfiguration().setDiscoverySpi(discoverySpi);

try(Ignite ignite = Ignition.start(igniteConfig)){
IgniteCache cache = ignite.getOrCreateCache(new
CacheConfiguration<>("cache")
.setIndexedTypes(Integer.class, Person.class));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());

}

}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

HI Ivan, thanks for the quick reply.

Yes it perfectly works as needed with the fix. 

Person.Address.Street will not work because
person.getPerson().getAddress().getStreet() does not exist. It has to be
person.getAddress().getStreet(). So column name should be "Address.street"

Any other name does the trick as long as there is alias option provided in
SQL, but in SQLFieldQuery I cannot provide alias option so the only option
that we are left with is use field name as alias name by default. That way
we will be able to rejoin the node. 

Also just creating field name as "street" will not work as with the same
name there could be another column

Ex: Person.Address.Street  and Person.College.Address.Street. If we use
street here then which street we we are referring to ? This fix will take
the whole name and use it query to avoid complete ambiguity


Please follow the instructions that I have mentioned in my previous email.
If you execute those two programs with and with out fix, you can get better
idea.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

I see there are couple of issues with the test

1) cache.query(new SqlFieldsQuery("alter table Person add column
\"Address.street\"  as street varchar"));
   - "as" is not supported in sql. So we should remove "as street" and it
should be like
cache.query(new SqlFieldsQuery("alter table Person add column
\"Address.street\"  varchar"));

2) After making above change, it will work with / with out fix. But the
problem is with out the fix, if you have any other node joining this node,
it will not be able to join / query

I just tweaked your test code. PLease find the below one. main() is your
test class which will start one node. 
  a) Take this example and execute main() - this will start one node
 b) now execute @Test method, this will start another node. You will get
attached exception ( exception.txt
  )
with out fix as by default it take alias as "street". But with fix it will
pass through, as it will create alias as "Address.street"

Please let me know if you need more clarification. Any quick help would be
appreciated. Thank you!


package my.ignite.cache.poc.index;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.query.SqlFieldsQuery;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import org.junit.Test;

import java.util.Collections;

public class NestedIndexFieldTest {

public static void main(String args[]){
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi()
.setIpFinder(new
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500..47509")));

IgniteConfiguration igniteConfig = new
IgniteConfiguration().setDiscoverySpi(discoverySpi);
Ignite ignite = Ignition.start(igniteConfig);
try{
IgniteCache cache = ignite.createCache(new
CacheConfiguration<>("cache")
.setIndexedTypes(Integer.class, Person.class));

cache.put(1, new Person("john", new Address("baker", 221)));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());

cache.query(new SqlFieldsQuery("alter table Person add column
name varchar"));

cache.query(new SqlFieldsQuery("alter table Person add column
address other"));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());

cache.query(new SqlFieldsQuery("alter table Person add column
\"Address.street\"  varchar"));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());

cache.put(2, new Person("bill", new Address("green", 223)));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());
} catch (Exception anyEx){

}

}

@Test
public void test(){
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi()
.setIpFinder(new
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500..47509")));

IgniteConfiguration igniteConfig = new
IgniteConfiguration().setDiscoverySpi(discoverySpi);

try(Ignite ignite = Ignition.start(igniteConfig)){
IgniteCache cache = ignite.getOrCreateCache(new
CacheConfiguration<>("cache")
.setIndexedTypes(Integer.class, Person.class));

System.err.println(cache.query(new SqlFieldsQuery("Select * from
Person")).getAll());

}

}
}

class Person {
String name;
Address address;

Person(String name, Address address){
this.name = name;
this.address = address;
}

@Override
public String toString() {
return "Person{" +
"name='" + name + '\'' +
", address=" + address +
'}';
}
}

class Address {
String street;
int number;

public Address(String street, int number){
this.street = street;
this.number = number;
}

@Override
public String toString() {
return "Address{" +
"street='" + street + '\'' +
", number=" + number +
'}';
}
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Does apache ignite support Spring 5.x

[Hemambara]

Please ignore, its my bad that there is maven dependency issue in my local.
Sorry about that



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Can you please provide an update



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Does apache ignite support Spring 5.x

[Hemambara]

I see ignite-spring 2.7.6 comes with 4.3.18.RELEASE. If I exclude these jars
and maintain 5.x.RELEASE, it is not working. So does ignite supports only
till 4.x release?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

ignite_bug_share.zip

  

Thanks for the reply. Please find the attached.

1) IndextestIgniteServer1 - starts server
2) IndexTestIgniteClient1 - starts the client
3) I did not add QueryEntities - we have usecase where we should be able to
add fields and indexes on the fly and ignite does support it.

This program works when we run for the first time. But if we restart client,
it is not able to join the server again because when we use "Users.userName"
field added dynamically, alias is being added as username and when we
restart due to mismatch in configuration it is not able to join

- With this fix, both field and index name will be "Users.userName" and
there wont be any mismatch. I built in local and it looks good. 

January we are going with first prod release, if we can have interim patch
for this, that should be fine. Please let us know if that works.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Hello Denis/Ivan, can you please check this and if there are no issues can
you merge the code. Sorry for the rush. We are getting ready for prod in
next 3 months and this is really important for us and we need to test all
our functionality as well. It would be a great help if we have this fixed



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Hello Ivan, can you please check and provide updates on this 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Can you please provide any update in this...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Hello Ivan, to be more specific, if I want add field 

"Users.userName" - current ignite version 2.7.6 is setting the field name as
"Users.userName" and default alias name as "userName" with which makes it
non-queryable, non -indexable and issues during restart due to mismatch in
configuration . But with this fix for fieldname "Users.userName" it will set
alias as "Users.userName" and the same can be used to add index and
everywhere and there wont be any issues



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Thank you. One more thing to add - it is able to add index dynamically only
thing is, when I restart the client it is not able join the cluster due to
incorrect alias



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Sorry for the push. This is a major blocker for us. We have coherence clients
where they can add indexes dynamically and we want to move them to ignite in
next 3 months. Coherence has a way to add indexes dynamically and ignite
also does supporting it. This issue exists event with QuerySqlField. If this
fix cannot be merged, I am not sure how to proceed further. Do you see any
issue putting this fix ? or it is just that we are not ready yet ? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

My apologies for multiple replies. Please consider latest reply



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

I have tested with adding QuerySqlFields as well, it is enabling to add
dynamic field and indexes on nested objects but if I restart the client
again it is not able to join the grid because it is not able to find this
column. But with the fix and maintaining proper aliases we are able to add
fields and indexes dynamically on direct/ nested attribute. This
functionality will be out of the box with this fix. Please let me know if
you still see any issue. 

Ignite does provide dynamic addition of columns and indexes but it is just
that there is a bug on adding indexes dynamically on QuerySqlFields nested
objects. With this small fix it will run with out any issues.

SEVERE: Failed to reinitialize local partitions (rebalancing will be
stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=3, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7af3d442-946f-4572-9fb7-08971b8d82cc, addrs=[0:0:0:0:0:0:0:1,
114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0,
DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=3,
intOrder=0, lastExchangeTime=1570647484111, loc=true,
ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=3,
nodeId8=7af3d442, msg=null, type=NODE_JOINED, tstamp=1570647484265],
nodeId=7af3d442, evt=NODE_JOINED]
class org.apache.ignite.IgniteCheckedException: Field not found:
Users.userName
at
org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124)
at
org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735)
at
org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752)
at
org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688)
at
org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612)
at
org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

ignite_bug_share.zip

  

I have tested by adding QuerySqlFields as well, with or with out it, it is
working when I add it, but if I restart the client it is not able to
identify nested columns as its alias is null or not equal to whole path.
Please see below exception. If we can make alias with whole nested object
path we will be able to add indexes and fields dynamically with out any
issues. Ignite, out of the box provides dynamic field and index alterations.
But it has a bug when we add nested field / index dynamically. This small
fix will fix this bug.

PFA for the code.


SEVERE: Failed to reinitialize local partitions (rebalancing will be
stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=5, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=a5379438-868b-43e4-b402-02edf2913795, addrs=[0:0:0:0:0:0:0:1,
114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0,
DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=5,
intOrder=0, lastExchangeTime=1570648315703, loc=true,
ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=5,
nodeId8=a5379438, msg=null, type=NODE_JOINED, tstamp=1570648318843],
nodeId=a5379438, evt=NODE_JOINED]
class org.apache.ignite.IgniteCheckedException: Field not found:
Users.userName
at
org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124)
at
org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735)
at
org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752)
at
org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688)
at
org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612)
at
org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

ignite_bug_share.zip

 
I have tested by adding QuerySqlFields as well, with or with out it, it is
working when I add it, but if I restart the client it is not able to
identify nested columns as its alias is null or not equal to whole path.
Please see below exception. If we can make alias with whole nested object
path we will be able to add indexes and fields dynamically with out any
issues. Ignite, out of the box provides dynamic field and index alterations.
But it has a bug when we add nested field / index dynamically. This small
fix will fix this bug. PFA for the code. SEVERE: Failed to reinitialize
local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0],
discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=a5379438-868b-43e4-b402-02edf2913795, addrs=[0:0:0:0:0:0:0:1,
114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0,
DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=5,
intOrder=0, lastExchangeTime=1570648315703, loc=true,
ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=5,
nodeId8=a5379438, msg=null, type=NODE_JOINED, tstamp=1570648318843],
nodeId=a5379438, evt=NODE_JOINED] class
org.apache.ignite.IgniteCheckedException: Field not found: Users.userName   
 
at
org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124)

at
org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735)

at
org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752)

at
org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688)

at
org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612)

at
org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480)

at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706)

at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)

at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337)

at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172)

at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030)

at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927)

at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769)

at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)

at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)

at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)  
  
at java.lang.Thread.run(Thread.java:748) 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Added test case and created pull request. Please check both the code changes
under pull request #6949 and test case pull request #6955 and let me know
for any suggestions. Thank you.

IGNITE-12261 - Issue with adding nested index dynamically - Adding tests
#6955



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

Sure, will add test case. 

My domain model is existing domain model and I cannot change or add
@QuerySqlField to my domain model. I am trying to achieve dynamic index
creation on any domain model object (pojo) irrespective @QuerySqlField's



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

https://github.com/apache/ignite/pull/6949

Created pull request with suggestion. Can you please check and close



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

I have made changes and created pull request. Can you please check and let me
know if there are any issues with the commit. Otherwise can you please let
me know when it can it be available in maven repo so that we can start
pulling it in our organization

https://github.com/apache/ignite/pull/6947




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with adding nested index dynamically

[Hemambara]

https://issues.apache.org/jira/browse/IGNITE-12261

I have created JIRA. Please let me know how I can assign to myself. My
username on jira board is "kotari"



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issue with adding nested index dynamically

[Hemambara]

We have to add indexes on cache dynamically on java pojo with nested objects.
In the below example we do not have @QuerySqlField. In this case if I try to
add index on "username" dynamically using "CREATE INDEX" it worked. But if I
want to add index on "Address.zipcode" - this is not working as we are NOT
creating aliases for these fields which are getting added dynamically. I see
there is a bug in this implementation. I fixed the bug (please see below)
and rebuilt the jar in my local and it worked fine. Proposing this bug to
this open source community so that it can be fixed in ignite and we can
start using the jar from central repo. We have just started using ignite and
planning to take this to production. So it would be helpful if we can
implement this bug fix.

User{
String userName;
Address address;

}

Address{
String streetName;
String zipcode;
}

Bug Fix: QueryUtils.class
--
Method name :
-- 
QueryBinaryProperty buildBinaryProperty(GridKernalContext ctx, String
pathStr,
Class resType, Map aliases, @Nullable Boolean
isKeyField, boolean notNull, Object dlftVal,
int precision, int scale)



Change from :
-
String alias = (String)aliases.get(fullName.toString());

to :
-
String alias = aliases.get(fullName.toString()) == null ?
fullName.toString() : aliases.get(fullName.toString());

With this way we will always make sure we have default alias (if not
provided), otherwise aliases on dynamic query fields and dynamic indexes are
not working properly





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/