Re: LOCAL cache on client

2018-02-01 Thread Valentin Kulichenko
I meant "they should *explicitly* provide data region configuration", of course. -Val On Thu, Feb 1, 2018 at 10:58 AM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Agree with Mike. I don't think it's a good idea to implicitly create data > regions on cli

Re: Get TTL of the specific (K,V) entry

2018-01-31 Thread Valentin Kulichenko
Cross-posting to dev. Igniters, This actually makes sense to me. Why don't we add IgniteCache#ttl(K key) method that would return current TTL for the key? Looks like this is already provided by GridCacheMapEntry#ttl() method, so we only need to properly expose it to public API. Am I right? If

Eviction policies with persistence

2018-01-31 Thread Valentin Kulichenko
Folks, On "Eviction Policies" documentation page [1] we have the following callout: > Configured eviction policy has no effect if Ignite persistence is enabled > Note that if Ignite Persistence is enabled, then the page-based evictions have no effect because the oldest pages will be purged from

Re: Ignite work directory usage?

2018-02-08 Thread Valentin Kulichenko
Sergey, These mappings are supposed to be the same on all nodes, so if the file already exists, we can safely ignore this, or use a lock to avoid concurrent access. Actually, I think we already fixed this in the past, it's weird that issue came up again. But in any case, switching marshaller

Re: What is the purpose of a binary schema and schema registry?

2018-02-06 Thread Valentin Kulichenko
Hi John, There are multiple ways to get several schemas for the same type. As Pavel mentioned, one of the examples is when Binarylizable generates different sets of fields under different circumstances. However, more common use case is for two client nodes to have different versions of the same

Re: Orphaned, duplicate, and main-class tests!

2018-02-06 Thread Valentin Kulichenko
Anton, I tend to agree with Ilya that identifying and fixing all the possible broken tests in one go is not feasible. What is the proper way in your view? What are you suggesting? -Val On Mon, Feb 5, 2018 at 2:18 AM, Anton Vinogradov wrote: > Ilya, > > 1) Still see

Re: Eviction policies with persistence

2018-02-06 Thread Valentin Kulichenko
it > >> is > >>> page based eviction, but not entry-based. Actually data is not removed, > >> but > >>> only written to disk. We can address this page later by ID. > >>> PDS eviction is primarily the replacement of pages fro

Re: Apache Ignite 2.4 release

2018-02-09 Thread Valentin Kulichenko
Nikolay, To merge it to 2.4, you need to merge the change to ignite-2.4 release. Let's do this if we come to an agreement in the neighbor thread. -Val On Thu, Feb 8, 2018 at 8:21 PM, Nikolay Izhikov wrote: > Hello, Dmitriy. > > IGNITE-7337 are merged to master [1] > > Do

Re: Saving Spark Data Frames merged to master

2018-02-09 Thread Valentin Kulichenko
, will be released soon :) > > > > On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org> > > wrote: > > > > > Hello, Igniters. > > > > > > Good news. > > > > > > IGNITE-7337 [1](Spark Data Frames: support s

Re: Removing "fabric" from Ignite binary package name

2018-02-09 Thread Valentin Kulichenko
t;>>>>>> On 20 Dec 2017, at 23:15, Denis Magda <dma...@apache.org> wrote: > > >>>>>>> > > >>>>>>> Petr, thanks, such a swift turnaround! > > >>>>>>> > > >>>>>>> Have you

Re: Ignite work directory usage?

2018-02-09 Thread Valentin Kulichenko
'metastore' for marshaller cache? > > Sincerely, > Dmitriy Pavlov > > пт, 9 февр. 2018 г. в 1:05, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Sergey, > > > > These mappings are supposed to be the same on all nodes, so if the fi

Re: IgniteSet implementation: changes required

2018-02-09 Thread Valentin Kulichenko
y to implement workaround for recovery from > PDS. > > For the collocated mode we can, for example, enforce REPLICATED cache > mode. > > > > Why don't you like the idea with separate cache? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7565 > > [2] htt

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-12 Thread Valentin Kulichenko
Nikolay, When you're talking about join optimization, what exactly are you referring to? Since other parts of data frames integration are already merged, I think it's a good time to resurrect this thread? Does it make sense to review it right now? Or you want to make some more changes? -Val On

Re: Apache Ignite 2.4 release

2018-02-11 Thread Valentin Kulichenko
IGNITE-7337 (Data frame save functionality) is merged into 2.4. -Val On Fri, Feb 9, 2018 at 11:47 AM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Nikolay, > > To merge it to 2.4, you need to merge the change to ignite-2.4 release. > Let's do this if we co

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Valentin Kulichenko
t; Extends scaladoc, etc. > > I will write you when PR is fully ready. > > [1] https://github.com/apache/ignite/pull/3397 > > > > В Пн, 12/02/2018 в 13:45 -0800, Valentin Kulichenko пишет: > > Nikolay, > > > > When you're talking about join optimization, what

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Valentin Kulichenko
; > I think we will have this information in a matter of 1-2 months. > > [1] https://github.com/apache/ignite/pull/3397 > [2] https://github.com/apache/ignite/pull/3397/files#diff- > 5a861613530bbce650efa50d553a0e92R227 > [3] https://gist.github.com/nizhikov/a4389fd78636869dd38c1392

Re: Should we annotate @CacheLocalStore as @Depricated?

2018-02-13 Thread Valentin Kulichenko
1] https://issues.apache.org/jira/browse/IGNITE-7588 > > > > On Tue, Jan 30, 2018 at 2:27 PM, Anton Vinogradov > > <avinogra...@gridgain.com> wrote: > >> +1 > >> > >> On Tue, Jan 30, 2018 at 9:02 AM, Yakov Zhdanov <yzhda...@apache.org>

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Valentin Kulichenko
heProxyImpl.query(IgniteCacheProxyImpl.java:664) > at org.apache.ignite.internal.processors.cache. > IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:615) > at org.apache.ignite.internal.processors.cache. > GatewayProtectedCacheProxy.query(GatewayProtectedCach

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Valentin Kulichenko
until: > > 1. We create documentation for all join limitations. > 2. Create the way to check is certain join satisfy current limitations. > > [1] http://apache-ignite-developers.2346864.n4.nabble.com/ > SparkDataFrame-Query-Optimization-Prototype-tp26249p26361.html >

Re: Removing "fabric" from Ignite binary package name

2018-02-07 Thread Valentin Kulichenko
; On 19 Dec 2017, at 22:55, Denis Magda <dma...@apache.org> wrote: > > >>>> > > >>>> All the bids were accepted and the verdict is executed: > > >>>> https://issues.apache.org/jira/browse/IGNITE-7251 < > > https://issues.apache.o

Re: Batch size parameter at DataStreamerCacheUpdaters.batched()

2018-02-15 Thread Valentin Kulichenko
Roman, DataStreamerCacheUpdaters class is actually not a part of public API, so I don't see a reason to change it unless there is a need for this internally in Ignite. -Val On Thu, Feb 15, 2018 at 5:52 AM, Roman Guseinov wrote: > Hello Igniters, > > In some cases, batched

Re: Apache Ignite 2.4 release

2018-02-16 Thread Valentin Kulichenko
Guys, While we're on this topic, what is the difference between BACKGROUND and NONE in terms of semantics and provided guarantees? To me it looks like both guarantee to recover the state since last checkpoint and anything else can potentially be lost, so from user perspective they are the same.

Re: Spark data frames

2018-02-16 Thread Valentin Kulichenko
That's a great idea! Nikolay, let me know if you need any help with the presentation, I will be happy to help. -Val On Fri, Feb 16, 2018 at 12:19 AM, Nikolay Izhikov wrote: > Ok, Igniters. > > I will do it in a few weeks. > I need time to prepare to the talk. > > В Пт,

Re: Batch size parameter at DataStreamerCacheUpdaters.batched()

2018-02-16 Thread Valentin Kulichenko
As far as I remember, it used to be public and then was moved to internal. The main issue with these updaters was that batching is dangerous because you can get deadlocks if keys are not sorted (which is the case for BATCHED). There is also BATCHED_SORTED, but it requires keys to be Comparable and

Re: Suppressing "development" warning

2017-12-20 Thread Valentin Kulichenko
is a > category, then it can be turned on and off using standard logger > configuration. > > D. > > On Mon, Dec 11, 2017 at 3:28 PM, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Igniters, > > > > We have bunch of warnings in the product wh

Re: Suppressing "development" warning

2017-12-21 Thread Valentin Kulichenko
Ticket created: https://issues.apache.org/jira/browse/IGNITE-7284 -Val On Wed, Dec 20, 2017 at 6:12 PM, Dmitriy Setrakyan <dsetrak...@apache.org> wrote: > Sounds good, markers should work. > > On Wed, Dec 20, 2017 at 1:00 PM, Valentin Kulichenko < > valentin.kulich

Re: BinaryObjectImpl.deserializeValue with specific ClassLoader

2018-01-03 Thread Valentin Kulichenko
Ticket is still open. Vladimir, looks like it's assigned to you. Do you have any plans to work on it? https://issues.apache.org/jira/browse/IGNITE-5038 -Val On Wed, Jan 3, 2018 at 1:26 PM, Abeneazer Chafamo wrote: > Is there any update on the suggested

Re: Spark data frames integration merged

2018-01-03 Thread Valentin Kulichenko
Revin, I doubt IgniteRDD#getPrefferredLocations has any affect on data frames, but this is an interesting point. Nikolay, as a developer of this functionality, can you please comment on this? -Val On Wed, Jan 3, 2018 at 1:22 PM, Revin Chalil wrote: > Thanks Val for the

Nightly builds from master

2018-01-03 Thread Valentin Kulichenko
Guys, The latest build on the nightly builds page [1] is from May 31. Any idea why? [1] https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/lastSuccessfulBuild/ -Val

Re: BinaryTypeConfiguration behavior in case of using wildcards.

2018-06-21 Thread Valentin Kulichenko
+1. Looks like a bug. -Val On Thu, Jun 21, 2018 at 12:26 PM Denis Magda wrote: > Hello Slava, > > BinaryContext implementation matches only classes that reside in > > the "org.apache.ignite.examples" package > > > This looks like an oversight on our side. Think we need to fix it. > > -- >

Re: Dirty Reads and READ_COMMITTED

2018-07-25 Thread Valentin Kulichenko
D - Data is read without a > lock > > and is never cached in the transaction itself."). Which should be wrong. > > Read locks are acquired but they are released as soon as the read is > > complete (and they are not held until the transaction commits or rolls > > back). > > &

Re: Data regions on client nodes

2018-07-27 Thread Valentin Kulichenko
ways > > > > zero [1]. That's why WARN message shown here [2] would be not not > quite > > > > right > > > > if we have a lot of client nodes in cluster. > > > > > > > > > > > > [1] > > > > > > > >

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-31 Thread Valentin Kulichenko
java#L88 > > [3] > > > https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Apache.Ignite.Examples/Sql/SqlDmlExample.cs#L91 > > [4] > > > https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/Sca

DataFrame integration does not support ARRAY type

2018-07-31 Thread Valentin Kulichenko
Hi Nikolay, Can you please take a look at this thread on SO? https://stackoverflow.com/questions/51621280/saving-a-spark-dataset-to-apache-ignite-with-array-column-and-savemode-overwrite Looks like org.apache.ignite.spark.impl.QueryUtils#dataType method should also support ArrayType as one of

Re: Local Continuous Query

2018-07-31 Thread Valentin Kulichenko
Denis, I think this is correct behavior. If you deploy a local query on a single node with a REPLICATED cache, you expect to be notified with all the updates. This is not the case for PARTITIONED caches. -Val On Tue, Jul 31, 2018 at 3:19 AM Denis Mekhanikov wrote: > Igniters, > > As you may

Re: Spark DataFrames With Cache Key and Value Objects

2018-08-01 Thread Valentin Kulichenko
h the key/val and the > relational fields in a dataframe schema. > > Stuart. > > > On 1 Aug 2018, at 04:23, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > > I don't think there are exact plans to remove _key and _value fields as > > it'

Re: Spark DataFrames With Cache Key and Value Objects

2018-08-01 Thread Valentin Kulichenko
for predicate pushdown to Ignite. > > I’m likewise keen to hear Nikolay’s point of view as he is obviously the > expert. > > Thanks for your help so far. > > Stuart. > > On 1 Aug 2018, at 18:17, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > &

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-27 Thread Valentin Kulichenko
rlying cache objects, which is not possible currently. > > Can you elaborate on the reason _key and _val columns in Ignite SQL > will be removed? > > Stuart. > > > On 27 Jul 2018, at 19:39, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > >

Re: Deprecating LOCAL cache

2018-07-27 Thread Valentin Kulichenko
t; > > > > > > > > On Wed, Jul 25, 2018 at 12:10 PM Dmitrii Ryabov < > > > > somefire...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 to make LO

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-27 Thread Valentin Kulichenko
Stuart, Nikolay, I really don't like the idea of exposing '_key' and '_val' fields. This is legacy stuff that hopefully will be removed altogether one day. Let's not use it in new features. Actually, I don't even think it's even needed. Spark docs [1] suggest two ways of creating a typed

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-27 Thread Valentin Kulichenko
the end): > https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd > > Stuart. > > On 27 Jul 2018, at 20:05, Valentin Kulichenko < > valentin.kuliche...@gmail.com> > wrote: > > Well, the second approach would use the optimizations, no? > > -Val > &g

Re: Spark SQL Table Name Resolution

2018-08-07 Thread Valentin Kulichenko
Stuart, Two tables can have same names only if they are located in different schemas. Said that, sdding schema name support makes sense to me for sure. We can implement this using either separate SCHEMA_NAME parameter, or similar to what you suggested in option 3 but with schema name instead of

Re: Folks, how do I generate tar.gz release for Ignite?

2018-08-07 Thread Valentin Kulichenko
Hi John, Please refer to DEVNOTES.txt, it describes the process step by step. -Val On Tue, Aug 7, 2018 at 12:14 PM John Wilson wrote: > Hi, > > How do I generate tar.gz for Ignite from source? > > Thanks, >

Re: Table Names in Spark Catalog

2018-08-20 Thread Valentin Kulichenko
Stuart, Nikolay, I see that the 'Table' class (returned by listTables method) has a 'database' field. Can we use this one to report schema name? In any case, I think we should look into how this is done in data source implementations for other databases. Any relational database has a notion of

Re: Continuous queries and MVCC

2018-08-17 Thread Valentin Kulichenko
Vladimir, 1. Continuous queries are asynchronous in general case, so I don't think it's even possible to provide transactional ordering, especially for the case of distributed transactions. I would leave current guarantees as-is. 2. This one might be pretty useful. If it's not very hard to do,

Re: affinityBackupFilter for AWS Availability Zones

2018-08-23 Thread Valentin Kulichenko
Hi David, With the Docker image you can actually use additional libraries by providing URLs to JARs via EXTERNAL_LIBS property. Please refer to this page: https://apacheignite.readme.io/docs/docker-deployment But anyway, I believe that such contribution might be very valuable for Ignite. Feel

Re: Table Names in Spark Catalog

2018-08-24 Thread Valentin Kulichenko
e case when we have several Ignite > configuration in one XML file. > Now I see, may be this is too rare use-case to support. > > Stuart, Valentin, What is your proposal? > > В Ср, 22/08/2018 в 08:56 -0700, Valentin Kulichenko пишет: > > Nikolay, > > > > Whate

Re: Table Names in Spark Catalog

2018-08-21 Thread Valentin Kulichenko
wrote: > > > Hello, Stuart. > > > > Can you do some research and find out how schema is handled in Data > Frames > > for a regular RDBMS such as Oracle, MySQL, etc? > > > > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет: > > > Stuart, Nikolay,

Re: Service Grid new design overview

2018-08-24 Thread Valentin Kulichenko
Guys, I believe we should preserve the behavior that we have now. What happens to services if we restart a persistent cluster running 2.6? Are services recreated or not? If YES, we should make sure the same happens after redesign. Would be even better if we preserve compatibility, i.e. allow

Re: Table Names in Spark Catalog

2018-08-27 Thread Valentin Kulichenko
s to represent the schema name as the > > database name for the purposes of the Spark catalog. > > > > If anyone knows of an existing way to list all available schemata within > an > > Ignite instance please let me know, otherwise the first task will be > > creating tha

Re: Table Names in Spark Catalog

2018-08-22 Thread Valentin Kulichenko
appy to make the change to have the > > database reference the schema if Nikolay agrees. (I'll first need to do a > > bit of research into how to obtain the list of all available schemata...) > > > > Thanks, > > Stuart. > > > > On Tue, Aug 21, 2018 at

Re: Retries in write-behind store

2018-08-29 Thread Valentin Kulichenko
t; > ср, 29 авг. 2018 г. в 7:14, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Folks, > > > > Is there a way to limit or disable retries of failed updates in the > > write-behind store? I can't find one, it looks like if an update fails,

Baseline topology without persistence

2018-07-18 Thread Valentin Kulichenko
Folks, By default baseline topology is enforced only if persistence is enabled and is not defined in in-memory only scenario. But does this mean that I still CAN use it without persistence? In some cases users want to avoid rebalancing in case of a node failure because it's assumed that this

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpitimeouts

2018-07-18 Thread Valentin Kulichenko
On Mon, Jul 9, 2018 at 1:33 PM Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > If clientFailureDetectionTimeout is not set on server node, will it use > failureDetectionTimeout > instead? > > Either way, this configuration seems to be a bit confusing, but I don't

Data regions on client nodes

2018-07-20 Thread Valentin Kulichenko
Folks, Currently do not create any regions or allocate any offheap memory on client nodes unless it's explicitly configured. This is good behavior, however there is a usability issue caused by the fact that many users have the same config file for both server and clients. This can lead to

Re: Does Ignite Txs Protect against Phantom Reads?

2018-07-24 Thread Valentin Kulichenko
Hi John, Looks like pictures can't be attached directly. Try uploading it somewhere and providing a link. However, since you're talking about ranges, my guess would be that you're using SQL which is currently NOT transactional. This support currently in development though, probably other members

Re: Data regions on client nodes

2018-07-20 Thread Valentin Kulichenko
together. > > D. > > On Fri, Jul 20, 2018 at 3:59 PM, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Folks, > > > > Currently do not create any regions or allocate any offheap memory on > > client nodes unless it's explicitly configured

Re: Deprecating LOCAL cache

2018-07-25 Thread Valentin Kulichenko
It sounds like the main drawback of LOCAL cache is that it's implemented separately and therefore has to be maintained separately. If that's the only issue, why not keep LOCAL cache mode on public API, but implement it as a PARTITIONED cache with a node filter forcefully set? That's similar to

Re: Dirty Reads and READ_COMMITTED

2018-07-25 Thread Valentin Kulichenko
Hi John, Read committed isolation typically implies that lock on read is not held throughout the transaction lifecycle, i.e. released right after read is completed (in Ignite this just means that no lock is required at all). This semantic allows second read to get an updated value that was

Re: GridClosureProcessor.affinityRun() semantics

2018-08-31 Thread Valentin Kulichenko
Dave, In case it's executed even if primary node is outside of the cluster group, then I think it's a bug - I would throw an exception in this case. However, is there any particular reason you're doing this? Is there a use case? I don't see much sense in combining affinityRun with a cluster

Re: How can i make my custom plugin serializable - Failed to serialize object: plugins.CustomSecurityProcessor

2018-08-31 Thread Valentin Kulichenko
There is a non-serializable anonymous class that you have declared within the plugin. You should check what is actually serialized there and whether it's supposed to be serialized or not. -Val On Fri, Aug 31, 2018 at 10:27 AM wt wrote: > How can i make my custom plugin serializable? > > >

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpitimeouts

2018-07-05 Thread Valentin Kulichenko
al: > > 5. Always use clientFailureDetectionTimeout on clients instead of > failureDetectionTimeout > *What*: change code to use clientFailureDetectionTimeout on clients > *When*: update code and readme.io docs in 2.7 > > Thanks, > Stan > > From: Valentin Kulichenko > Sent:

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpitimeouts

2018-07-09 Thread Valentin Kulichenko
onTimeout=20. > When these two nodes communicate, server will use timeouts of 20 seconds > and client will use timeout of 10 seconds. > > Stan > > From: Valentin Kulichenko > Sent: 6 июля 2018 г. 23:17 > To: dev@ignite.apache.org > Subject: Re: IgniteConfiguration, TcpDiscoveryS

Re: Review for IGNITE-8880: Add setIgnite() in SpringCacheManager and SpringTransactionManager

2018-07-09 Thread Valentin Kulichenko
Hi Amir, I reviewed the change and commented in the ticket. -Val On Fri, Jul 6, 2018 at 4:18 PM Amir Akhmedov wrote: > Igniters, > > Please review my changes. That's a simple change, a setter method was > added. > > Thanks, > Amir >

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpitimeouts

2018-07-06 Thread Valentin Kulichenko
imeout is that it may allow > clients to be slower/on a slower network than servers. > > Do you think it isn’t worth to have a separate setting just for that? > > Thanks, > Stan > > From: Valentin Kulichenko > Sent: 5 июля 2018 г. 18:16 > To: dev@ignite.apache.or

Re: Neighbors exclusion

2018-07-13 Thread Valentin Kulichenko
Dmitry, Good point. I think it makes sense to even remove (deprecate) the excludeNeighbors property and always distribute primary and backups to different physical hosts in this scenario. Because why would anyone ever set this to false if we switch default to true? This also automatically fixes

Re: Partition loss policy to disable cache completely

2018-01-23 Thread Valentin Kulichenko
cess it? Imagine I use data on nodes A and B > > > performing reads and writes and node C crashes in the middle of tx. > > Should > > > my tx be rolled back? I think no. > > > > > > As far as difference it seems that IGNORE resets lost status for > affected >

Re: Trimming exceptions in IgniteUtils::cast

2018-03-12 Thread Valentin Kulichenko
Generally, I think we should not trim any exceptions, because this way we can unexpectedly remove useful information. Do we know what was the original reasoning behind this logic? -Val On Mon, Mar 12, 2018 at 3:57 AM, Stanislav Lukyanov wrote: > Hi, > >

Re: Data eviction/expiration from Ignite persistence

2018-03-12 Thread Valentin Kulichenko
Alex, What is behavior going to be after IGNITE-5874 is fixed? Will expired entry be removed from both memory and persistence? -Val On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > The ticket [1] is in patch available state looks good, the only thing

Re: [TeamCity] DataFrame Examples

2018-02-27 Thread Valentin Kulichenko
Nikolay, Spark integration is not related to Scalar, the only thing they have in common is Scala. I think we should have a separate configuration for ignite-spark module. If anything Spark related is currently in Scalar suite, it should be moved from there. -Val On Tue, Feb 27, 2018 at 3:03 AM,

Re: [TeamCity] DataFrame Examples

2018-02-27 Thread Valentin Kulichenko
FS are also tested there. > > Sincerely, > Dmitriy Pavlov > > вт, 27 февр. 2018 г. в 22:49, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Nikolay, > > > > Spark integration is not related to Scalar, the only thing they have in > > common is Scala.

Re: Data eviction/expiration from Ignite persistence

2018-03-08 Thread Valentin Kulichenko
Guys, What is the result of this discussion? Do we still not support eviction and expiration on persistence level? If so, any plans to change this? -Val On Tue, Nov 21, 2017 at 9:45 AM, Denis Magda wrote: > We might break the compatibility for the next major release or even

Re: Why does DataStreamer swallow exceptions?

2018-03-14 Thread Valentin Kulichenko
s is a colossal usability problem (I'm pretty sure I've seen > numerous messages about it on userlist) and I'll fill an issue if nobody is > objecting. > > Ilya. > > -- > Ilya Kasnacheev > > 2018-03-05 22:50 GMT+03:00 Valentin Kulichenko < > valentin.kuliche...@gmail.c

Re: Partition recovery issue on partition loss.

2018-03-13 Thread Valentin Kulichenko
This indeed looks like a bigger issue. Basically, there is no clear way (or no way at all) to synchronize code that listens to partition loss event, and the code that calls resetLostPartitions() method. Example scenario: 1. Cache is configured with 3rd party persistence. 2. One or more nodes fail

Re: IEP-14: Ignite failures handling (Discussion)

2018-03-13 Thread Valentin Kulichenko
Ivan, If grid hangs, graceful shutdown would most likely hang as well. Almost never you can recover from a bad state using graceful procedures. I agree that we should not create two defaults, especially in this case. It's not even strictly defined what is embedded node in Ignite. For example, if

Re: Service grid redesign

2018-04-04 Thread Valentin Kulichenko
I don't think peer class loading is even possible for services. I believe we should reuse DeploymentSpi [1] for versioning. [1] https://apacheignite.readme.io/docs/deployment-spi -Val On Wed, Apr 4, 2018 at 12:52 PM, Denis Magda wrote: > Sorry, that was me who renamed the

Re: Service grid redesign

2018-04-06 Thread Valentin Kulichenko
Yes, the class deployment itself has to be explicit. I.e., there has to be a manual step where user updates the class, and the exact step required would depend on DeploymentSpi implementation. But then Ignite takes care of everything else - service redeployment and restart is automatic. Dmitriy

Re: Atomic caches

2018-04-10 Thread Valentin Kulichenko
Guys, I also not sure I understand the purpose of methods like [1] that accept instance of AtomicConfiguration to create new atomic structure. Per my knowledge, all atomics are stored in a single cache which is configured by AtomicConfiguration provided on startup as part of IgniteConfiguration.

Re: affinityRun/Call in C++

2018-04-10 Thread Valentin Kulichenko
Is there a ticket? Let's create one if not. -Val On Tue, Apr 10, 2018 at 6:17 AM, Vladimir Ozerov <voze...@gridgain.com> wrote: > Val, > > They are simply not implemented yet. I am not aware of concrete plans to > support them. > > On Mon, Apr 9, 2018 at 11:33

Re: Triggering rebalancing on timeout or manually if the baseline topology is not reassembled

2018-04-11 Thread Valentin Kulichenko
Denis, In my understanding, in this case you should remove node from BLT and that will trigger the rebalancing, no? -Val On Wed, Apr 11, 2018 at 12:23 PM, Denis Magda wrote: > Igniters, > > As we know the rebalancing doesn't happen if one of the nodes goes down, > thus,

affinityRun/Call in C++

2018-04-09 Thread Valentin Kulichenko
Guys, Is there a way to run collocated compute job in C++? I can't find affinityRun and affinityCall method in C++ compute API, am I missing something? If we really don't have them, is there any particular reason for this and/or plans to add them? -Val

Re: Service grid redesign

2018-04-09 Thread Valentin Kulichenko
should be added. > > > > > > I don't think, that a lot of people will implement their own > > > *DeploymentSpi*-s, so we should make work with *UriDeploymentSpi* as > > > comfortable as possible. > > > > > > Denis > > > > > &g

Re: Service grid redesign

2018-04-09 Thread Valentin Kulichenko
has to be brought up after a full > cluster restart w/o user intervention. To achieve this we need to persist > the service's configuration somewhere. > > -- > Denis > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > &g

Re: Service grid redesign

2018-04-05 Thread Valentin Kulichenko
roper DeploymentSpi. > > Please correct me, if I'm wrong. > > It would be good, though, to add some examples on service redeployment, > > when implementation class changes. > > > > Denis > > > > чт, 5 апр. 2018 г. в 2:33, Valentin Kulichenko < > > val

Re: iep-6 metrics ticket review

2018-04-10 Thread Valentin Kulichenko
This is on my plate, will try to take a look this week. -Val On Mon, Apr 9, 2018 at 10:28 AM, Denis Magda wrote: > Val, > > As an initial reviewer and reporter, could you have a look and sign the > contribution off? > > -- > Denis > > On Mon, Apr 9, 2018 at 12:56 AM, Aleksey

Re: Service grid redesign

2018-04-20 Thread Valentin Kulichenko
o listen to needed ports, then a > corresponding exception will be propagated to the user code. > On the other hand, if exception is thrown from the *execute() *method, then > service won't be undeployed. > > Denis > > пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko < > va

Re: Service grid redesign

2018-04-16 Thread Valentin Kulichenko
> > > > > >> > > > > > >> Also, I heard that presently we store a service configuration in > > the > > > > > >> system > > > > > >> cache that doesn't give us a way to deploy a new version of a > > &g

Re: Service grid redesign

2018-04-19 Thread Valentin Kulichenko
gt; when it is rebalanced to another node. > > As Denis said, if we are not going to prevent nodes from starting on > service failures, then we should at least generate corresponding events. > Otherwise there won't be any way to react to service initialization > failures during n

Re: Timeline for support of compute functions by thin clients

2018-04-02 Thread Valentin Kulichenko
Folks, Any other thoughts on this? Should we create tickets for compute support if there are no objections? -Val On Thu, Mar 22, 2018 at 4:27 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > I agree that compute and services functionality is important for thin

Re: Timeline for support of compute functions by thin clients

2018-04-03 Thread Valentin Kulichenko
nce would be not to have compute functionality on thin > clients, as it would introduce extra security risk. > > Any particular reason why you are asking for this feature? > > ⁣D.​ > > On Apr 2, 2018, 8:47 PM, at 8:47 PM, Valentin Kulichenko < > valentin.kuliche...@gmail.co

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

2018-03-26 Thread Valentin Kulichenko
e. Make sure to deactivate the >> cluster before shutdown."); >> > > Best Regards, > Ivan Rakov > > > On 24.03.2018 1:40, Valentin Kulichenko wrote: > >> Dmitry, >> >> Thanks for clarification. So it sounds like if we fix all other modes as >> we

Re: What's about releasing Ignite 2.5 a bit earlier?

2018-03-22 Thread Valentin Kulichenko
Nikolay, Sounds like a good idea. I will do my best to speed up the process and review asap. -Val On Thu, Mar 22, 2018 at 11:37 AM, Nikolay Izhikov wrote: > Hello, guys > > I agree with earlier release. > > I propose to include my task IGNITE-7077 to 2.5 release. > >

Re: Timeline for support of compute functions by thin clients

2018-03-22 Thread Valentin Kulichenko
I agree that compute and services functionality is important for thin client. It doesn't seem to be very hard to implement, but would provide much better flexibility, as users would be able to do remote invocation of arbitrary code, use collocated processing, etc. Having an ability to do this from

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

2018-03-23 Thread Valentin Kulichenko
e ready > to > > >>>>>>>>> > > >>>>>>>> release > > >>>>>>>> AI 2.5 with 7% drop. > > >>>>>>>>> 2) Introduce LOG_ONLY_SAFE, make it default, add release note > > >>&g

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

2018-03-23 Thread Valentin Kulichenko
triy Pavlov > > сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > I agree. In my view, any possibility to get a corrupted storage is a bug > > which needs to be fixed. > > > > BTW, can someone explain semant

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

2018-03-19 Thread Valentin Kulichenko
Guys, What do we understand under "data corruption" here? If a storage is in corrupted state, does it mean that it needs to be completely removed and cluster needs to be restarted without data? If so, I'm not sure any mode that allows corruption makes much sense to me. How am I supposed to use a

Re: Why does DataStreamer swallow exceptions?

2018-03-05 Thread Valentin Kulichenko
Ilya, IgniteDataStreamer#addData method returns future which should be completed with error if one is thrown on server side. Does this happen or not? -Val On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov wrote: > Hello, Ilya. > > > I think it's time to end this, if that

Re: Enabling swap space and Ignite Persistence

2018-03-02 Thread Valentin Kulichenko
I stumbled across couple of use cases where swap space was more suitable than persistence. However, enabling both for a same region definitely doesn't make sense to me, I would throw an exception in this case. -Val On Fri, Mar 2, 2018 at 9:46 AM, Denis Magda wrote: > Hi

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts

2018-03-01 Thread Valentin Kulichenko
+1. Low level timeouts that we still have in discovery and communication are very hard to explain and I doubt there is anyone who fully understands how they currently work. They bring a lot of complexity and almost zero value. Let's deprecate them and leave only failureDetectionTimeout plus other

Re: Access to change jira status

2018-06-29 Thread Valentin Kulichenko
Hi Uday, I added you to contributors list, you should be able to assign the ticket to yourself now. Welcome to the community! -Val On Fri, Jun 29, 2018 at 3:40 AM uday kale wrote: > Hi, > > I am unable to change the jira status of IGNITE-1260 or assign it to > myself. Can someone grant me the

<    1   2   3   4   5   6   7   8   9   10   >