Re: Schema Questions
Hi, The main question here is how you want to use this data. Do you use SQL? 1) It depends on the use case. If you plan to access only a person object without any filtering by addresses and you will always need the entire object, it makes sense to have one big object. But in this case, you won't be able to filter persons by addresses, since SQL doesn't work with collections. So, if you want to use SQL, it definitely makes sense to use the second approach. 2) Of course, if you already have unique ID for object, it makes sense to use it as a key, there is no need to generate an additional field for this. Evgenii пн, 11 мая 2020 г. в 09:20, narges saleh : > Hi All, > > I would appreciate your feedback, for the following, in terms of > performance for both inserts and queries. > > 1) Which one of these patterns is preferable for the table design? > A- Have a fat table/cache with nested objects, e.g. person table with a > hashmap of addresses. > B- Have person and address tables separate and just link them via foreign > keys. > > 2) Which one of these patterns is preferable for primary keys? > A- Have a UUID + affinity key as the primary key > B- Have the keys spelled out + affinity key. For example, assume person > table, combination of age and name uniquely identifies a person, so the key > will be person-name, person-age, and org-id. > If I have a associative table joining persons and addresses (if address is > a separate object), then in case B, I will have to include three fields > from person and the id from the address table, as opposed to case A, where > I will have UUID + orgid + address id. Would having one less field buy me > much, as opposed to having the overhead of creating UUIDs? > > thanks > >
Re: Apache Ignite Persistence Issues/Warnings
Hi, You need to be careful w/disabling WAL archiving. It is best to put the WAL on a separate (quicker) disk: https://apacheignite.readme.io/docs/durable- memory-tuning#separate-disk-device-for-wal This section will be of benefit to you when tuning persistence: https://apacheignite.readme.io/docs/durable-memory-tuning#general-tuning Tips to follow: https://apacheignite.readme.io/docs/performance-tips Try to achieve the best performance you can w/out changing wal archiving. If that doesn't help, the following section will guide you: WAL archive tuning: https://apacheignite.readme.io/docs/write-ahead-log#tuning-wal-archive Thanks, Alex -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Suggest a better way to access a particular K-V store
The K is key with type String.
The V is the value with type JSONObject/HashMap.
V is having the following structure:
{"agentId":,"score":,"url":}
Now the query on this will be in such a way that -> Get top 50 values of "K"
where values in that K-V store are sorted based on "score". Provided that
K-V store has 10-50 million K-V pairs.
Can someone please suggest which data structure I need to use to solve this
particular case?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Apache Ignite Persistence Issues/Warnings
Hi Alex, thanks for the reply. But as our application is write/update intensive, we want a good performance/high availability application solution along with disk persistence. In that case, what configuration would you suggest for the following case, 1. Disk storing functionality can run periodically and while that(checkpointing) is running it can be delayed to some extent but not too much delay/starvations. 2. Faster access to data is needed. Queries should run quicker. (which is actually happening now when checkpointing is not present). 3. All logical operations should be updated in WAL at any cost and even if we configure checkpointing to be done at an interval of 1hour per 1 checkpoint, WAL should have all those operations logged. 4. We are disabling WAL archiving. Is it needed if point 3 is the case? Thanks, Aditya. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Event for an update that should have been filtered is received in Local Listener of Continuous Query when a 1000 row insert is triggered
Hi,
handleEntry() performs internal housekeeping chores.
You are correct a notification is sent to localListeners whether an entry
was filtered or not.
your actual local listener will be called only if the entry has passed
your filter criteria.
If you place a breakpoint inside
CacheContinuousQueryHandler#notifyLocalListener you will see that
this message gets rejected by the caller.
here:if (F.isEmpty(evts)) return;
In general you shouldn't need to go so far deeply into the internals:
Try the following:
server:
IgniteCache cache =
ignite.getOrCreateCache(CACHE_NAME);
int i = 0;
while (true) {
cache.put(i++, Integer.toString(i));
System.out.println("added entry: " + i);
Thread.sleep(100);
}
client:
IgniteCache cache =
ignite.getOrCreateCache(CACHE_NAME);
ContinuousQuery < Integer, String > qry = new
ContinuousQuery<>();
qry.setLocalListener((evts) -> evts.forEach(e ->
System.out.println("key=" + e.getKey() + ", val=" + e.getValue(;
qry.setRemoteFilterFactory((Factory>) () -> (CacheEntryEventFilter) e -> e.getKey() % 2
== 0);
cache.query(qry);
Only the filtered events should arrive.
Thanks, Alex
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cache was inconsistent state
John,
Yes, client nodes should have this parameter too.
Evgenii
пн, 11 мая 2020 г. в 07:54, John Smith :
> I mean both the prefer IPV4 and the Zookeeper discovery should be on the
> "central" cluster as well as all nodes specifically marked as client = true?
>
> On Mon, 11 May 2020 at 09:59, John Smith wrote:
>
>> Should be on client nodes as well that are specifically setClient = true?
>>
>> On Fri, 8 May 2020 at 22:26, Evgenii Zhuravlev
>> wrote:
>>
>>> John,
>>>
>>> It looks like a split-brain. They were in one cluster at first. I'm not
>>> sure what was the reason for this, it could be a network problem or
>>> something else.
>>>
>>> I saw in logs that you use both ipv4 and ipv6, I would recommend using
>>> only one of them to avoid problems - just add
>>> -Djava.net.preferIPv4Stack=true
>>> to all nodes in the cluster.
>>>
>>> Also, to avoid split-brain situations, you can use Zookeeper Discovery:
>>> https://apacheignite.readme.io/docs/zookeeper-discovery#failures-and-split-brain-handling
>>> or
>>> implement Segmentation resolver. More information about the second can be
>>> found on the forum, for example, here:
>>> http://apache-ignite-users.70518.x6.nabble.com/split-brain-problem-and-GridSegmentationProcessor-td14590.html
>>>
>>> Evgenii
>>>
>>> пт, 8 мая 2020 г. в 14:30, John Smith :
>>>
How though? It's the same cluster! We haven't changed anything
this happened on it's own...
All I did was reboot the node and the cluster fixed itself.
On Fri, 8 May 2020 at 15:32, Evgenii Zhuravlev <
e.zhuravlev...@gmail.com> wrote:
> Hi John,
>
> *Yes, it looks like they are in a different clusters:*
> *Metrics from the node with a problem:*
> [15:17:28,668][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
>
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=5bbf262e, name=xx, uptime=93 days, 19:36:10.921]
> ^-- H/N/C [hosts=3, nodes=4, CPUs=10]
>
> *Metrics from another node:*
> [15:17:05,635][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
>
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=dddefdcd, name=xx, uptime=19 days, 16:49:48.381]
> ^-- H/N/C [hosts=6, nodes=7, CPUs=21]
>
> *The same topology versions for 2 nodes has different nodes:*
> [03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
> Topology snapshot [ver=1036, locNode=5bbf262e, servers=1, clients=3,
> state=ACTIVE, CPUs=10, offheap=10.0GB, heap=13.0GB]
> [03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
> ^-- Baseline [id=0, size=3, online=1, offline=2]
>
> *And*
>
> [03:56:43,388][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
> Topology snapshot [ver=1036, locNode=4394fdd4, servers=2, clients=2,
> state=ACTIVE, CPUs=15, offheap=20.0GB, heap=19.0GB]
> [03:56:43,389][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
> ^-- Baseline [id=0, size=3, online=2, offline=1]
>
> So, it's just 2 different clusters.
>
> Best Regards,
> Evgenii
>
> пт, 8 мая 2020 г. в 08:50, John Smith :
>
>> Hi Evgenii, here the logs.
>>
>> https://www.dropbox.com/s/ke71qsoqg588kc8/ignite-logs.zip?dl=0
>>
>> On Fri, 8 May 2020 at 09:21, John Smith
>> wrote:
>>
>>> Ok let me try get them...
>>>
>>> On Thu., May 7, 2020, 1:14 p.m. Evgenii Zhuravlev, <
>>> e.zhuravlev...@gmail.com> wrote:
>>>
Hi,
It looks like the third server node was not a part of this cluster
before restart. Can you share full logs from all server nodes?
Evgenii
чт, 7 мая 2020 г. в 09:11, John Smith :
> Hi, running 2.7.0 on 3 deployed on VMs running Ubuntu.
>
> I checked the state of the cluster by going
> to: /ignite?cmd=currentState
> And the response was:
> {"successStatus":0,"error":null,"sessionToken":null,"response":true}
> I also checked: /ignite?cmd=size=
>
> 2 nodes where reporting 3 million records
> 1 node was reporting 2 million records.
>
> When I connected to visor and ran the node command... The details
> where wrong as it only showed 2 server nodes and only 1 client, but 3
> server nodes actually exist and more clients are connected.
>
> So I rebooted the node that was claiming 2 million records instead
> of 3 and when I re-ran the node command displayed all the proper
> nodes.
> Also after the reboot all the nodes started reporting 2 million
> records instead of 3 million so there some sort of rebalancing or
> correction (the cache has a 90 day TTL)?
>
>
>
> Before
Schema Questions
Hi All, I would appreciate your feedback, for the following, in terms of performance for both inserts and queries. 1) Which one of these patterns is preferable for the table design? A- Have a fat table/cache with nested objects, e.g. person table with a hashmap of addresses. B- Have person and address tables separate and just link them via foreign keys. 2) Which one of these patterns is preferable for primary keys? A- Have a UUID + affinity key as the primary key B- Have the keys spelled out + affinity key. For example, assume person table, combination of age and name uniquely identifies a person, so the key will be person-name, person-age, and org-id. If I have a associative table joining persons and addresses (if address is a separate object), then in case B, I will have to include three fields from person and the id from the address table, as opposed to case A, where I will have UUID + orgid + address id. Would having one less field buy me much, as opposed to having the overhead of creating UUIDs? thanks
Re: How to list caches and the number of key-values inside them
Hi,
Ignite binaries contain visor cmd tool, it can do what you want:
https://apacheignite-tools.readme.io/docs/command-line-interface
Evgenii
пн, 11 мая 2020 г. в 00:05, scriptnull :
> I would like to know if there is a command line tool that will help us list
> down the caches and the respective number of keys present in each cache.
>
> The closest I have got is using `control.sh` to list down the names of the
> caches.
>
> Also since we are currently only using the redis layer to create the dbs, I
> wrote this little shell script to do it.
>
> ```
> dbsize() {
> /usr/share/apache-ignite/bin/control.sh --cache list redis-* | grep -o
> "redis-ignite-internal-cache-\w*" | sed -r
> 's/redis-ignite-internal-cache-//g' | xargs -n 1 --replace="{}" sh -c
> 'printf "Database Number = {} | DBSIZE = "; redis-cli -p 11211 -n {}
> dbsize'
> }
> ```
>
> But I really wonder, if there is a clean of doing this!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Cache was inconsistent state
I mean both the prefer IPV4 and the Zookeeper discovery should be on the
"central" cluster as well as all nodes specifically marked as client = true?
On Mon, 11 May 2020 at 09:59, John Smith wrote:
> Should be on client nodes as well that are specifically setClient = true?
>
> On Fri, 8 May 2020 at 22:26, Evgenii Zhuravlev
> wrote:
>
>> John,
>>
>> It looks like a split-brain. They were in one cluster at first. I'm not
>> sure what was the reason for this, it could be a network problem or
>> something else.
>>
>> I saw in logs that you use both ipv4 and ipv6, I would recommend using
>> only one of them to avoid problems - just add -Djava.net.preferIPv4Stack=true
>> to all nodes in the cluster.
>>
>> Also, to avoid split-brain situations, you can use Zookeeper Discovery:
>> https://apacheignite.readme.io/docs/zookeeper-discovery#failures-and-split-brain-handling
>> or
>> implement Segmentation resolver. More information about the second can be
>> found on the forum, for example, here:
>> http://apache-ignite-users.70518.x6.nabble.com/split-brain-problem-and-GridSegmentationProcessor-td14590.html
>>
>> Evgenii
>>
>> пт, 8 мая 2020 г. в 14:30, John Smith :
>>
>>> How though? It's the same cluster! We haven't changed anything
>>> this happened on it's own...
>>>
>>> All I did was reboot the node and the cluster fixed itself.
>>>
>>> On Fri, 8 May 2020 at 15:32, Evgenii Zhuravlev
>>> wrote:
>>>
Hi John,
*Yes, it looks like they are in a different clusters:*
*Metrics from the node with a problem:*
[15:17:28,668][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=5bbf262e, name=xx, uptime=93 days, 19:36:10.921]
^-- H/N/C [hosts=3, nodes=4, CPUs=10]
*Metrics from another node:*
[15:17:05,635][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=dddefdcd, name=xx, uptime=19 days, 16:49:48.381]
^-- H/N/C [hosts=6, nodes=7, CPUs=21]
*The same topology versions for 2 nodes has different nodes:*
[03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
Topology snapshot [ver=1036, locNode=5bbf262e, servers=1, clients=3,
state=ACTIVE, CPUs=10, offheap=10.0GB, heap=13.0GB]
[03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
^-- Baseline [id=0, size=3, online=1, offline=2]
*And*
[03:56:43,388][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
Topology snapshot [ver=1036, locNode=4394fdd4, servers=2, clients=2,
state=ACTIVE, CPUs=15, offheap=20.0GB, heap=19.0GB]
[03:56:43,389][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
^-- Baseline [id=0, size=3, online=2, offline=1]
So, it's just 2 different clusters.
Best Regards,
Evgenii
пт, 8 мая 2020 г. в 08:50, John Smith :
> Hi Evgenii, here the logs.
>
> https://www.dropbox.com/s/ke71qsoqg588kc8/ignite-logs.zip?dl=0
>
> On Fri, 8 May 2020 at 09:21, John Smith
> wrote:
>
>> Ok let me try get them...
>>
>> On Thu., May 7, 2020, 1:14 p.m. Evgenii Zhuravlev, <
>> e.zhuravlev...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> It looks like the third server node was not a part of this cluster
>>> before restart. Can you share full logs from all server nodes?
>>>
>>> Evgenii
>>>
>>> чт, 7 мая 2020 г. в 09:11, John Smith :
>>>
Hi, running 2.7.0 on 3 deployed on VMs running Ubuntu.
I checked the state of the cluster by going
to: /ignite?cmd=currentState
And the response was:
{"successStatus":0,"error":null,"sessionToken":null,"response":true}
I also checked: /ignite?cmd=size=
2 nodes where reporting 3 million records
1 node was reporting 2 million records.
When I connected to visor and ran the node command... The details
where wrong as it only showed 2 server nodes and only 1 client, but 3
server nodes actually exist and more clients are connected.
So I rebooted the node that was claiming 2 million records instead
of 3 and when I re-ran the node command displayed all the proper nodes.
Also after the reboot all the nodes started reporting 2 million
records instead of 3 million so there some sort of rebalancing or
correction (the cache has a 90 day TTL)?
Before reboot
+=+
| # | Node ID8(@), IP |Consistent ID
| Node Type | Up Time | CPUs | CPU Load |
Re: Cache was inconsistent state
Should be on client nodes as well that are specifically setClient = true?
On Fri, 8 May 2020 at 22:26, Evgenii Zhuravlev
wrote:
> John,
>
> It looks like a split-brain. They were in one cluster at first. I'm not
> sure what was the reason for this, it could be a network problem or
> something else.
>
> I saw in logs that you use both ipv4 and ipv6, I would recommend using
> only one of them to avoid problems - just add -Djava.net.preferIPv4Stack=true
> to all nodes in the cluster.
>
> Also, to avoid split-brain situations, you can use Zookeeper Discovery:
> https://apacheignite.readme.io/docs/zookeeper-discovery#failures-and-split-brain-handling
> or
> implement Segmentation resolver. More information about the second can be
> found on the forum, for example, here:
> http://apache-ignite-users.70518.x6.nabble.com/split-brain-problem-and-GridSegmentationProcessor-td14590.html
>
> Evgenii
>
> пт, 8 мая 2020 г. в 14:30, John Smith :
>
>> How though? It's the same cluster! We haven't changed anything
>> this happened on it's own...
>>
>> All I did was reboot the node and the cluster fixed itself.
>>
>> On Fri, 8 May 2020 at 15:32, Evgenii Zhuravlev
>> wrote:
>>
>>> Hi John,
>>>
>>> *Yes, it looks like they are in a different clusters:*
>>> *Metrics from the node with a problem:*
>>> [15:17:28,668][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
>>>
>>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>>> ^-- Node [id=5bbf262e, name=xx, uptime=93 days, 19:36:10.921]
>>> ^-- H/N/C [hosts=3, nodes=4, CPUs=10]
>>>
>>> *Metrics from another node:*
>>> [15:17:05,635][INFO][grid-timeout-worker-#23%xx%][IgniteKernal%xx]
>>>
>>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>>> ^-- Node [id=dddefdcd, name=xx, uptime=19 days, 16:49:48.381]
>>> ^-- H/N/C [hosts=6, nodes=7, CPUs=21]
>>>
>>> *The same topology versions for 2 nodes has different nodes:*
>>> [03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
>>> Topology snapshot [ver=1036, locNode=5bbf262e, servers=1, clients=3,
>>> state=ACTIVE, CPUs=10, offheap=10.0GB, heap=13.0GB]
>>> [03:56:17,643][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
>>> ^-- Baseline [id=0, size=3, online=1, offline=2]
>>>
>>> *And*
>>>
>>> [03:56:43,388][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
>>> Topology snapshot [ver=1036, locNode=4394fdd4, servers=2, clients=2,
>>> state=ACTIVE, CPUs=15, offheap=20.0GB, heap=19.0GB]
>>> [03:56:43,389][INFO][disco-event-worker-#42%xx%][GridDiscoveryManager]
>>> ^-- Baseline [id=0, size=3, online=2, offline=1]
>>>
>>> So, it's just 2 different clusters.
>>>
>>> Best Regards,
>>> Evgenii
>>>
>>> пт, 8 мая 2020 г. в 08:50, John Smith :
>>>
Hi Evgenii, here the logs.
https://www.dropbox.com/s/ke71qsoqg588kc8/ignite-logs.zip?dl=0
On Fri, 8 May 2020 at 09:21, John Smith wrote:
> Ok let me try get them...
>
> On Thu., May 7, 2020, 1:14 p.m. Evgenii Zhuravlev, <
> e.zhuravlev...@gmail.com> wrote:
>
>> Hi,
>>
>> It looks like the third server node was not a part of this cluster
>> before restart. Can you share full logs from all server nodes?
>>
>> Evgenii
>>
>> чт, 7 мая 2020 г. в 09:11, John Smith :
>>
>>> Hi, running 2.7.0 on 3 deployed on VMs running Ubuntu.
>>>
>>> I checked the state of the cluster by going
>>> to: /ignite?cmd=currentState
>>> And the response was:
>>> {"successStatus":0,"error":null,"sessionToken":null,"response":true}
>>> I also checked: /ignite?cmd=size=
>>>
>>> 2 nodes where reporting 3 million records
>>> 1 node was reporting 2 million records.
>>>
>>> When I connected to visor and ran the node command... The details
>>> where wrong as it only showed 2 server nodes and only 1 client, but 3
>>> server nodes actually exist and more clients are connected.
>>>
>>> So I rebooted the node that was claiming 2 million records instead
>>> of 3 and when I re-ran the node command displayed all the proper nodes.
>>> Also after the reboot all the nodes started reporting 2 million
>>> records instead of 3 million so there some sort of rebalancing or
>>> correction (the cache has a 90 day TTL)?
>>>
>>>
>>>
>>> Before reboot
>>>
>>> +=+
>>> | # | Node ID8(@), IP |Consistent ID
>>> | Node Type | Up Time | CPUs | CPU Load | Free Heap |
>>>
>>> +=+
>>> | 0 | xx(@n0), xx.69 | xx | Server| 20:25:30 | 4
>>> | 1.27 % | 84.00 % |
>>> | 1 | xx(@n1), xx.1 | xx | Client| 13:12:01 | 3
Re: Apache Ignite Kubernetes Stateful deployment with parallel pod management
Thanks Alex for the response. Parallel pod management is working. Earlier I added readiness and liveness probes, with initial delay of 180 seconds for the Ignite pods in stateful set because of which no traffic was allowed to pods, and hence the discovery failed. After removing these probes, Parallel pod management is working fine. Regards, Syed Zaheer Basha -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
How to list caches and the number of key-values inside them
I would like to know if there is a command line tool that will help us list
down the caches and the respective number of keys present in each cache.
The closest I have got is using `control.sh` to list down the names of the
caches.
Also since we are currently only using the redis layer to create the dbs, I
wrote this little shell script to do it.
```
dbsize() {
/usr/share/apache-ignite/bin/control.sh --cache list redis-* | grep -o
"redis-ignite-internal-cache-\w*" | sed -r
's/redis-ignite-internal-cache-//g' | xargs -n 1 --replace="{}" sh -c
'printf "Database Number = {} | DBSIZE = "; redis-cli -p 11211 -n {} dbsize'
}
```
But I really wonder, if there is a clean of doing this!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
