Alexey,
What if we create a thin wrapper around PortableBuilder that implements
IgniteObject? Since PortableBuilder already has getField() method, you only
need to delegate to a builder in this wrapper. Then we can invoke the
interface Dmitriy suggested, obtain hashCode and set it back to the
Guys,
A quick update on the release status.
I have spotted several potential slowdowns that I would like to add to 1.5
release. I am currently working in a separate branch and preliminary
testing showed that we can get about 4-5% performance improvement in tx-put
benchmark. There are other
Raul,
You are right, ignite-1282 is the integration branch for the Binary format.
There are a couple outstanding issues need to be fixed, but in general the
branch is stable enough to work with.
2015-11-12 15:51 GMT+03:00 Raul Kripalani :
> I think it's ignite-1282, right? I
performance degradation for atomic caches. I expect the ticket to be merged
> in 2 days.
>
> On Tue, Nov 3, 2015 at 6:47 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > wrote:
>
> > IGNITE-950 is taking more time to be merged than I originally esti
I think it's a critical issue; good thing is that the ticket already
contains a proper description, so at least it is clear what to fix. I'll
try to take a look at this over the weekend.
2015-11-05 15:45 GMT+03:00 Vladimir Ozerov :
> Igniters,
>
> One more thing - I found a
Kuznetsov <akuznet...@gridgain.com>:
> I'm working on IGNITE-1753 Rework CacheJdbcPojoStore to new API (
> https://issues.apache.org/jira/browse/IGNITE-1753).
> Issue is implemented and waiting for IGNITE-950 to be merged into
> ignite-1282.
>
> On Wed, Nov 4, 2015 at 5
I am not sure what behavior we want to achieve here. Do we want to disable
cache peer class loading for binary marshalling altogether? This
restriction is safe to remove only if user chooses to work exclusively with
Binary object representation.
If user still wants to work with class object
I am finalizing changes related to IGNITE-1486 branch. Currently the API
changes are finished and now I am mostly fixing the CI tests. In fact, I
just submitted the latest fix related to the new .NET platform
functionality and hope CI tests will pass, in this case it will be ready to
be merged
g>:
> Alexey,
>
> I am confused. Why do you have 2 branches?
>
> D.
>
> On Mon, Nov 2, 2015 at 2:32 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > wrote:
>
> > I am finalizing changes related to IGNITE-1486 branch. Currently the API
> &
feedback to
the proposed changes.
Looking forward for the comments :)
2015-11-05 16:11 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:
> I think it's a critical issue; good thing is that the ticket already
> contains a proper description, so at least it is clear what to fix
Igniters,
As a part of work on IGNITE-950 [1] ticket we wanted to implement a
functionality which would allow users to plug their own implementation of
IgniteObject, which in turn would allow to introspect objects for fields
without deserialization and use of reflection.
The design draft is
Hi folks,
I merged performance optimizations for tx mode which gave up to 5%
improvement for tx-put benchmark to ignite-1.5.
I also pushed finalizaing changes to Binary configuration in ignite-1945
branch and currently waiting for TC. If all is ok, will cooperate with
Vladimir Ozerov and merge
Ken,
I have also provided some feedback regarding the IGNTIE-1226 ticket (sorry
it took a long time to respond to your previous email).
2015-08-31 18:48 GMT-07:00 Valentin Kulichenko <
valentin.kuliche...@gmail.com>:
> Hi Ken,
>
> I was just in process of reviewing them :) Please give me couple
Folks,
Did we make any changes to Indexing lately? I see that queries suite is
failing in 1.4/master branch with the following assertion. If this is a
known issue, can we at least properly mute the test so that it does not
overtime? Test is
Yakov, Igniters,
I have found at least one issue related to ignite-1171 hang, it is caused
by a race between discovery custom message and collectDiscoveryData() call
(updated the ticket). I remember we wanted to call collectDiscoveryData()
during the NodeAddFinishedMessage processing, however it
2015-09-17 10:55 GMT-07:00 Vladimir Ozerov :
This is not something weird, but rather how things work everywhere except
> of Java. getAndUpdate() is not what we need, because it is a CAS loop, not
> CAS.
>
This is an implementation detail. For a distributed data structure it
Andrey,
The answer is yes, all mutator operations invoked outside of a user
transaction do create an implicit transaction. There is no way to change
the isolation or concurrency of an implicit transaction, and I cannot think
of a case when this should be changed. You can think of any implicit
Vova,
We still have the logic that allows us to use reflection to get values in
indexing, so basically the change is an additional check during the query
processor start.
My concern regarding (2) is that a server node must have model classes in
the classpath in order to check that we should
I like the idea, however it has obvious downsides. First, if a user class
contains a collection, we force user to implement additional interface,
even if the collection is a simple ArrayList. Second, I do not see how this
plain collection can be the value for the cache - user will always need to
I see nothing wrong with generating a server config for a client node. This
may be used in a case when server nodes are started with plain empty config
and clients are deploying caches to the grid.
Denis,
I will take a look at this tomorrow.
>
> I like it.
I will add this method if there will be no objections.
> Also, does it make sense to pass a copy of the BinaryObject into
> the EntryProcessor?
I am not sure I understood the question correctly. We never copy
BinaryObject since it is immutable.
Yup. I got to the bottom of the issue, will push the fix soon.
If we do not want withKeepBinary() flag to affect the way objects are
passed to a cache store, then I think current approach with a cache
configuration flag is better than having a separate interface for store.
With separate interfaces it is impossible to implement a 'universal' store
which will
Server/client mode does not (and should not) change the marshaller
settings, to my knowledge there are no such places in the code that might
change this behavior.
There might be something in the Zep integration test, but I need some time
to install Zep and take a look at the code.
Hi Chandresh,
I cannot point out any specifics related to python because I have no
background in it, however I think I can give you a couple of starting
points. You can take a look at how python RDD API is implemented in Spark
and how Java Ignite RDD interacts with Scala Ignite RDD
(classes
The fix for https://issues.apache.org/jira/browse/IGNITE-2200 has been
merged to ignite-1.5
+1 (binding)
Sure, I will provide my feedback in a couple of hours.
Looks like it is a duplicate of IGNITE-2135.
The issue is caused by not setting deployment info bean in query request
when binary marshaller is used.
I anticipate the fix to be pretty simple, though need some more time to add
a proper test and to run CI. Will send an update once all is done.
Igniters,
Another issue has been reported related to unexpected exception in examples
when optimized marshaller is used:
https://issues.apache.org/jira/browse/IGNITE-2266
I will try to fix this before 2257 is fixed to fit in 1.5, however I do not
think it is a blocker for the release.
Status for the opened tickets from my side:
https://issues.apache.org/jira/browse/IGNITE-2078The original issue
reported has been fixed. Now sometimes the example returns duplicate
records for TXT queries, this is related to changing topology not being
handled in TXT queries. This is an old issue
+1 [Binding]
Verified checksum and signature, checked RAT, built binaries from sources
and started a couple of nodes, ran some examples, briefly checked javadoc.
Andrey,
If I leave behind my knowledge about Ignite internals, my expectation would
be that an EntryProcessor is invoked on all affinity - both primary and
backup - nodes in the grid. The main reason behind this expectation is that
usually a serialized EntryProcessor instance is smaller than
I think this is a great idea, however this exercise only makes sense if
FindBugs profile is plugged to CI and provides continuous checks for the
new code being committed.
Since Alexey K. already has some feedback for the first run, I will
cooperate with him to create a minimal set of checks that
Agree, +1 for the EA. We are preparing to change the default marshaller, so
it will be great to obtain the feedback from the community and see if we
missed something.
Folks,
There is a couple of sporadical hangs in the branch, need more time to
investigate and fix. Will take a look tomorrow SPb time.
2015-11-20 21:11 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:
> Yakov,
>
> I still see some of the tests fail on TC in ignite-12
I will add the check for non-get-named method if there are no objections, I
see no reason for not having it.
2015-11-24 13:25 GMT+03:00 Roman Shtykh :
> +1 for adding support, particularly because in Scala we don't necessarily
> have get... and set... (an underscore
Taras,
According to IgniteRDD non-predefined types (including Object) are mapped
to StructType with no fields, which should be recognizable by Spark. Can
you explain why MatchError happens?
Dmitriy, Yakov, Alexey K.,
What are your thoughts?
Val,
Looks good to me, can't wait to see it in master :)
We can add an explicit warning in 1.x when a cache with the null name is
used and remove it in 2.0.
2016-06-01 15:49 GMT-07:00 Dmitriy Setrakyan :
> -1
>
> I don’t think this will give us any advantage other than many frustrated
> users who will need to change their code.
Saikat,
Please also correct the test to check new methods for PARTITIONED and
REPLICATED cache - I see that you only test them for local cache and
partition 0 (I added a comment to the ticket).
2016-06-20 9:17 GMT-07:00 Saikat Maitra :
> Thank you Ilya, I will review
Dmitriy,
Are you suggesting that we need to pass partitions state to a topology
validator so that user needs to check it manually. I do not think this is
convenient for an end-user and like the approach with the policy that
Vladimir suggested better.
Raul,
I assume you want to add IgniteCacheEx
and this
> new method becomes very subtle. One just looks like a subset of another. Am
> I wrong?
>
> D.
>
> On Thu, Jan 14, 2016 at 2:38 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Dmitriy,
> >
> > Are you suggesting that
Hi,
As Andrey pointed out, now you can grab an expiry policy factory from
Ignite's cache configuration, create an instance and get durations you
need. I agree that this way a bit awkward and it only covers a configured
ExpiryPolicy, currently there is no way to check if an instance of
IgniteCache
Folks,
Recently I have seen a couple of emails suggesting tasks/improvements that
we cannot do in 1.x releases due to API compatibility reasons, so they are
postponed to 2.0. I would like to keep track of these tasks in some way in
our Jira to make sure we do not have anything obsolete when it
Currently lock-only functionality is exposed via j.u.c.Lock interface on
IgniteCache. We have two choices here:
* Release such locks on transaction commit, which would break the contract
of j.u.c.Lock
* Do not release such locks on transaction commit, which, in my opinion,
conflicts with the
In PESSIMISTIC transaction a value is always read after a lock is acquired,
so a locked value cannot be updated. Am I missing something? Do you have a
specific scenario in mind?
So, basically, we want to add lockAll() method that locks entries without
returning their values to a client - this is a good idea. I do not want,
however, to call it SNAPSHOT isolation, because this is not what it is.
Igniters,
I want to bring community attention to the ticket [1] once again since it
raises a lot of confusion among Ignite users [2] and I would like this to
be addressed in 1.6.
The alternatives for the case when entry processor is used in CLOCK mode
are:
* Print a warning, but still allow to
After a second thought, I think the warning should be printed anyways when
CLOCK versioning mode is used because it does not guarantee happens-before
relationship when cache operations are invoked on different nodes (see an
example in pseudo-code below). Essentially, when cache operations are
If all keys are known in advance, how is it different from starting a
pessimistic transaction and invoking getAll() on those keys? Introducing a
new concept with such restrictions does not makes sense to me.
2016-02-04 1:27 GMT+03:00 Dmitriy Setrakyan :
> Igniters,
>
> I
Val,
The code looks good to me. The only place that made me wonder was
out.unsafeEnsure(1 + 4) call which extends the stream by 5 bytes, however
we can write significantly more bytes. I see that we use the same approach
in other places, so I was wondering if this is a required call or a
This is correct, I took the original test that existed for Optimized
marshaller and copied it for Binary marshaller. Was not aware of multi-jvm
specifics. Just ran the provided example with Optimized marshaller - it
does not work either.
2016-01-28 11:08 GMT+03:00 Denis Magda
+1 for printing out a warning and ignoring the affinity function from the
configuration. There is no other way to 'fix' the configuration other than
remove the wrong affinity function, so it can be done at startup time right
away.
2016-02-03 9:43 GMT+03:00 Ken Cheng :
> I
Dmitriy,
I did a code review of the PR with Artem and I think I understood the need
for two mappers. First, name mapper is used to map platform (Java, .NET,
C++) class name to a common platform-independent type name. This name is
used in SQL table name and it is also passed to ID mapper. ID
Agree that this should be fixed. However, from the end-user perspective
this should never be an issue because at some point in future Ignite might
need to append some extra bytes to the Externalizable object layout, thus
reading beyond that limit will break the unmarshalling process.
If, by the
Denis,
Changes in IGNITE-2647 look good to me.
Yakov,
Nothing major from my side is planned for 1.6. There is a ticket
IGNITE-2610 that I am currently working on (it is almost finished), but I
believe it does not make any restrictions for the schedule.
Folks,
The current implementation of IgniteCache.lock(key).lock() has the same
semantics as the transactional locks - cache topology cannot be changed
while there exists an ongoing transaction or an explicit lock is held. The
restriction for transactions is quite fundamental, the lock() issue can
Hi,
I have merged the changes fixing IGNITE-2610 and verified the test in
IGNITE-1661. Please verify and close the ticket if all is ok.
Yes, the fix has been merged to master.
On Feb 18, 2016 22:06, "Dmitriy Setrakyan" <dsetrak...@apache.org> wrote:
> Alexey, can this be verified in master?
>
> On Thu, Feb 18, 2016 at 7:47 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
>
Andrey,
Are you talking about a continuous query or a distributed event listener?
Hi,
The current version of test is not very clean and it works only because
withKeepBinary() is a noop. The correct version would be to use plain cache
for non-binary-object entry processor and use withKeepBinary for
binary-object entry processor. You can see that EntryProcessor creation is
Dmitriy is right, currently REPLICATED cache works the same way as
PARTITIONED does, and in PARTITIONED cache filters should be evaluated on
backups in order to maintain a backup queue in case a primary node fails.
For the case when query is executed on an affinity node of a REPLICATED
cache
Note that withKeepBinary() is just a way to tell a cache not to deserialize
values when doing a get or running an entry processor. The concept of
binary object does not belong solely to caches - you can get an instance of
IgniteBinary interface from Ignite and use binary objects in computations,
Guys,
I fixed code style a bit and pushed my changes to the branch.
Couple of questions:
- I see that some of the Errors caught do not get re-thrown (e.g. if
interruptAll flag is set). I believe we should at least re-throw OOME in
any case.
- readResolve method is missing for CacheLockImpl.
Any reason you want to make it serializable? This is an internal class and
it is never transmitted over the network to a remote node. As Yakov pointed
out, cache store factory is what being transmitted to remote nodes with the
configuration.
Folks,
The PR looks good to me. There is one concern - even though the type
parameters were placed to IgniteContext by mistake, removing them will
break backward compatibility. Are we ok with that?
Val, can you comment?
2016-05-21 8:32 GMT-07:00 MaBiao :
> @agoncharuk
Hi Naden,
But the IgniteContext within Spark doesn't allow you to read configuration
> files from YARN.
>
I am a little bit confused. Ignite can be configured via basic Spring XML,
and you can definitely read those XML files off HDFS or any other source.
Is there any reason why XML does not
Denis,
Updates are always queued on primary nodes when write-behind is enabled,
regardless of atomicity mode. This is required because otherwise updates
can be written to the database in a wrong order.
We did not queue database updates on backups because we did not have a
mechanism that would
Yes, this is correct, if there is no write-behind, then in TRANSACTIONAL
cache the database write happens from the originating node, and in ATOMIC
cache - from primary nodes.
This is a cross-post from a user list.
We faced this issue for a lot of times before and got a lot of users
complaining about the whole cluster freeze. We can protect a cluster from
such a situation simply by dropping non-responsive nodes from the cluster.
Of course, we need to get to the bottom
Great stuff! Guys, please go ahead and file corresponding tickets or
reassign the fix version to 2.0 if a ticket exists so that we keep track of
the scope. It would also be great if you review the ones I created and see
if they make sense.
>
> >
> > I know there is IgniteDataStreamer for writing cache, but how about
> > reading cache as stream for iterate all elements with scan performane
> 1-3M
> > tuple/sec?
> >
>
> We already have Scan queries which allow for paginated iteration with
> filters. Are you suggesting something beyond
>
> Alexey, I like the idea in general, but killing non-responsive nodes seems
> a bit drastic to me. How about this approach:
>
> - print out IDs/IPs of non-responsive nodes at all times
> - introduce a certain kill timeout for non-responsive nodes (-1 means
> disabled)
> - the timeout should be
According to the JLS [1], adding or removing annotations has no effect on
the correct linkage of the binary representations of programs in the Java
programming language. Even if these annotations were RUNTIME, a user could
successfully use Ignite unless he explicitly uses those classes in runtime.
The ticket is created:
https://issues.apache.org/jira/browse/IGNITE-3616
2016-07-15 1:51 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:
> Alexey, I like the idea in general, but killing non-responsive nodes seems
>> a bit drastic to me. How about this approach:
>&g
Dmitriy,
The question is how do you calculate the value of the hashCode? Do you want
it to be specified explicitly in INSERT statement?
2016-08-01 19:47 GMT+03:00 Dmitriy Setrakyan :
> Alex,
>
> In your case, why not just explicitly set hashcode every time you create an
>
So, no excitement about Ignite 2.0? :)
I went ahead and created a 2.0 version in Ignite Jira, and included the
following tickets so far based on the chance that this ticket will require
breaking changes in APIs/Configuration
- IGNITE-3469 - Get rid of deprecated APIs and code
- IGNTIE-3477 -
Big +1 on this in general.
I would also relax our guarantees on operations submitted from the same
thread. Currently we guarantee that sequential invocations of async
operations happen in the same order. I think that if a user wants such
guarantees, he must define these dependencies explicitly by
Suppose I have a PersonKey {int id;} class and Person {@SqlQueryField int
id; @SqlQueryField String name; @SqlQueryField int age;} class.
How would an insert statement look like?
INSERT INTO Person (_key, _val) values (...)
INSERT INTO Person (_key, _val, id, name, age) values (...) -> Obviously
For me the main question here is how we define Key and Value for a cache
when using SQL interface. Unless we are going to re-architect the whole
Ignite, it will still be a key-value storage in the first place, so
whenever we do an INSERT, we must generate Key and Value. Given that values
in INSERT
There were also numerous contributions merged with small API improvements.
+1 for making a new release.
Since the changes are mostly bugfixes, should we make it a point release?
Replied in the ticket.
at Maitra <saikat.mai...@gmail.com>:
> > > Thanks a lot Alexey.
> > >
> > > Regards
> > > Saikat
> > >
> > > On Mon, Jul 4, 2016 at 12:01 PM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> >
Denis
> > > > > > > > >>
> > > > > > > > >> > On Aug 2, 2016, at 7:09 AM, Vladimir Ozerov <
> > > > > voze...@gridgain.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> >
> &
specify a hashCode in insert statement, or force him to use our "automatic"
algorithm of hashCode calculation in his key objects. Both ways look fishy
to me.
2016-08-02 2:20 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> On Mon, Aug 1, 2016 at 10:01 AM, Alexey Goncharu
> >
> > Thank you
> > Saikat
> >
> > On Mon, Jun 20, 2016 at 10:36 PM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> >> Saikat,
> >>
> >> Please also correct the test to check new methods for P
I think we should have duplicate filtering logic in discovery manager. As
far as I remember, we wanted custom events to be consistent with other
discovery events and we rely on the fact that node joined and node left
event won't be received twice.
2017-02-03 14:40 GMT+03:00 Sergey Chugunov
Alexei,
Can you describe the semantics of the task in greater detail?
* What if two jobs on different nodes access and try to put the same key?
(this can be resolved by allowing a job only access local primary keys)
* How do you define the lock acquisition order and prevent deadlocks? I
assume
Andrey,
>From the top of my head I would guess that this is done so because
keyIterator handles on-heap, off-heap and swap, while entrySet() return
only on-heap entries. Please check that your change does not break
iteration with off-heap and swap enabled (if it does, it just means that we
need
dException("Failed to
> commit transaction " +
> "(transaction has been rolled back on backup node): " + req.version(),
> *cause*));
> __
>
> How do you unsure that *"cause"* can't come to
> GridCacheUtils#retryTopologySafe(final
Alexander,
Do you have a use-case in mind which results in IgniteTopologyException
thrown from public API with retryFuture unset?
retryFuture makes sense only for certain cache operations and may be set
only in certain places in the code where we already know what to wait for.
I do not see how
Generally I agree with Yakov here, now order of updates is fully consistent
with the order of cache locks acquisition. You change the transaction type
=> you change the order of cache locks => you change the order of db writes.
I can assume that in some cases such a reordering may be useful,
Nikita,
The fix looks wrong to me. The point of the assertion was to ensure an
invariant,
see
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager#executeQuery
-V2 handler is created only when remote filter factory is not null.
The test observes both fields
m
> >:
>
> > thanx! my next PR review will be in up source
> >
> > пт, 17 февр. 2017 г. в 13:05, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> >
> > Aleksey,
> >
> > I added a comment on GitHub, however, the communi
Aleksey,
I am not sure your change is a correct fix. The whole point of
EntryProcessor is to avoid sending values over the network and send only
entry processor result, which is typically smaller than the value stored in
a cache.
Looks like this case is covered by the following ticket:
Dmitriy,
What do you mean by deploying cache metadata? The node filter itself must
be deserialized on each node because we need to evaluate it. As for the
CacheConfiguration, we do not need to deserialize the configuration on
every node, this will be a good change for Ignite 2.0.
2017-02-16
1 - 100 of 980 matches
Mail list logo