Re: .Net client call tx,close in other thread

2017-07-25 Thread Nikolay Izhikov
Exception --- Stderr: --- Test started: CacheAbstractTransactionalTest.TestTxStateAndExceptions Test finished: CacheAbstractTransactionalTest.TestTxStateAndExceptions -- Nikolay Izhikov nizhikov@gmail.com -- Nikolay Izhikov nizhikov@gmail.com

Re: Make Teamcity Green Again

2017-07-20 Thread Nikolay Izhikov
+1 to Dmitry Pavlov from me. Green team city would be very helpful. > If you are a newcomer, please share your feeling: are you confused by test > failures on Teamcity? I'm working on my very first ignite issue and it's not easy to understand TC build results. 21.07.2017 00:30, Dmitriy

Re: Ignite Benchmarking Rules

2017-09-15 Thread Nikolay Izhikov
Hello, Dmitriy. I think experienced members of community have specific number for benchmarking. Can we start from reference hardware configuration: Num of CPU, RAM and HDD(SDD?) configuration, network configs, etc. Can someone share that kind of knowledge - Which hardware is best for

Re: Binary compatibility of persistent storage

2017-09-19 Thread Nikolay Izhikov
Hello, Vladimir. I prefer 2 option. > 2) No compatibility, but provide migration instruments I think we have to provide some tool for converting WAL files from any older format to current. 19.09.2017 14:16, Vladimir Ozerov пишет: igniters, Ignite doesn't have compatibility for binary

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-09-14 Thread Nikolay Izhikov
he >> > > onEvent() then invoked on the listener anyway, regardless of the >> filter >> > > result? Finally, the listener will have a local callback field, which >> > will >> > > be called on the originating node. This sounds way mor

Re: Spark+Ignite SQL syntax proposal

2017-10-08 Thread Nikolay Izhikov
Hello, Ray. I think it can be done as a second step. After DataFrame for a current Spark release would be merged. Thoughts? 07.10.2017 16:42, Ray пишет: Hi Nikolay, Could you also implement the DataFrame support for spark-2.10 module? There's some legacy spark users still using Spark 1.6,

Re: Monitoring of active transactions

2017-09-08 Thread Nikolay Izhikov
Hello, Ilya. Great! Thanks! Can I extend your idea a bit? I think it be very useful monitor all-user provided listeners and callbacks also. To handle following scenarios: 1а. User starts a job with ExecutorService, IgniteCompute or similar. 1b. User creates ContinuousQuery with

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-08-29 Thread Nikolay Izhikov
Zhdanov пишет: Igniters, Does anyone else see potential issues on user side with current approach? Sam, is this JCache requirement? --Yakov 2017-08-29 15:16 GMT+03:00 Nikolay Izhikov <nizhikov@gmail.com>: Yakov. I think exception equals `true` is intended behavior. Filter eval

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-08-30 Thread Nikolay Izhikov
is required by jcache, but now I can not find anything about this neither in spec nor in jcache tck tests. So I think we can change current behavior. Thanks On Tue, Aug 29, 2017 at 9:48 PM, Nikolay Izhikov <nizhikov@gmail.com> wrote: Yakov. I found following description in jcache java

ContinuousQueryWithTransformer implementation questions - 2

2017-08-29 Thread Nikolay Izhikov
Hello, Igniters. I'm working on IGNITE-425 [1] issue. Text of issue: === Currently if updated entry passes the filter, it is sent to node initiated the query entirely. It would be good to provide user with the ability to transform entry and, for example, select only fields that are

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-08-30 Thread Nikolay Izhikov
Hello, Yakov. The new class is OK - got it. Thanks! > Should we extract a super class? Yes, we should. I already have done it. See my last commit in PR - https://github.com/apache/ignite/pull/2372/commits/af1ed2e4dbef4ba5999f8566198cb75ad922f93b > We can put hard requirement that filter

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-08-29 Thread Nikolay Izhikov
Hi, Yakov. If filter throws exception entry would be passed to listener. > Nikolay, I would also suggest you extract some super class for continuous > query. It will help to avoid code duplicates. Yes, I will do this after reaching consensus on API changes. 29.08.2017 14:04, Yakov Zhdanov

Re: ContinuousQueryWithTransformer implementation questions - 2

2017-08-29 Thread Nikolay Izhikov
Yakov. I think exception equals `true` is intended behavior. Filter evaluation implementation from master - [1] Test from master to check filter exception(without explicit asserts checking listeners call) - [2] Here is my quick test with asserts on listener call after filter exception: ```

Re: Time and scope for Apache Ignite 2.4

2017-12-19 Thread Nikolay Izhikov
Hello, Denis > - Nikolay Izhikov: Data Frames. I'm waiting for Valentin Kulichenko review. If and when further comments for DataFrame pull request appears I will fix them immediately. В Вт, 19/12/2017 в 12:07 -0800, Denis Magda пишет: > Igniters, > > It’s time to sync up on

Re: List of running Continuous queries or CacheEntryListener per cache or node

2017-12-21 Thread Nikolay Izhikov
Hello, Dmitry. I think it a great idea. Do we a feature to list all running ComputeTasks? I, personally, think we have to implement possibility to track all user-provided tasks - CacheListener, ContinuousQuery, ComputeTasks, etc. В Чт, 21/12/2017 в 10:13 +0300, Dmitry Karachentsev пишет: >

Re: Transparent Data Encryption (TDE) in Apache Ignite

2018-05-14 Thread Nikolay Izhikov
ope of page. Continious IV accumulating CBC mode for > whole store seems to be not working. Same question about IV is applicable > to WAL records. > > Could we consider ECB? or limit CBC with zero or well known IV? > > Sincerely, > Dmitriy Pavlov > > сб, 5 мая 2018 г.

Re: Transparent Data Encryption (TDE) in Apache Ignite

2018-05-14 Thread Nikolay Izhikov
ation, but will allow us to move by small > steps, it also allows us to merge smaller changes to master. > > Sincerely, > Dmitriy Pavlov > > пн, 14 мая 2018 г. в 16:53, Nikolay Izhikov <nizhi...@apache.org>: > > > Hello, guys. > > > > We had private di

Re: stopAllGrids() used by default and further steps

2018-05-14 Thread Nikolay Izhikov
Yes. В Пн, 14/05/2018 в 20:48 +0300, Dmitry Pavlov пишет: > Hi Nikolay, > > Would you have a minute to finalize this review? > > Sincerely, > Dmitriy Pavlov > > пт, 4 мая 2018 г. в 16:05, Maxim Muzafarov : > > Dmitry, > > > > Task of migration to JUnit 4/5 sounds very

Re: Transparent Data Encryption (TDE) in Apache Ignite

2018-04-27 Thread Nikolay Izhikov
Hello, Igniters. We've discussed TDE design privately with some respected community members including Vladimir Ozerov and Alexey Goncharyuk. Here the list of questions we have to address before starting TDE implementation: 1. MEK, CEK rotation. Should we provide the way to change(regenerate)

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

2018-04-28 Thread Nikolay Izhikov
g the existing test file > > for the Scala code. > > 4. I updated the TestSuite with bullet #3 above. > > 5. I combined all changes into a single pull request. > > > > Note that one of the tests still fails locally due to the path issue. I > > have no idea how to f

Re: Transparent Data Encryption (TDE) in Apache Ignite

2018-05-05 Thread Nikolay Izhikov
attacks. We should measure the treatment and find the way to deal with it. This question requires additional research. We have to understand: Is there any issue here? But at first glance, We can encrypt blocks in reverse order – from last to first. Last blocks don't have well know so there is no i

Re: Thin clients compatibility policy

2018-06-06 Thread Nikolay Izhikov
+1 From me. As I wrote in previous mail-threads, I think we need to create test framework to be able to test compatibility for all clients we have. AFAIK, currently, there is no possibility to automatically check compatibility. В Ср, 06/06/2018 в 11:39 +0300, Vladimir Ozerov пишет: > Igniters,

IEP-18 TDE: Prototype

2018-06-09 Thread Nikolay Izhikov
Hello, guys. I've implement prototype of TDE implementation [1] Vladimir, can you do some prereview of this prototype? Any feedback on public API or any other part of implementation are welcome. I have several questions I want to discuss. 1. Right place for a cache(data) key:

Re: Ignite 2.6 emergency release suggestion

2018-06-13 Thread Nikolay Izhikov
Hello, guys. I try my best to do review in a next few days. В Вт, 12/06/2018 в 11:28 -0700, Denis Magda пишет: > Agree with Ray. The ticket has been already reviewed and requires us to run > tests for an isolated module - Spark. > > Dmitriy Pavlov, Nickolay Izhikov, could you step in as final

Re: What's new for thin client protocol v1.1.0 vs v1.0.0

2018-06-04 Thread Nikolay Izhikov
Hello, Igniters. AFAIK, we have backward compatibility for a thin client protocol. But, currently, we doesn't have any kind of tests to check it. Do we have a ticket to create backward compatibility tests? В Пн, 04/06/2018 в 13:30 +0300, Igor Sapego пишет: > Hi Aleksandr, > > There are no

Re: Upgrading Ignite to JCache 1.1

2018-06-05 Thread Nikolay Izhikov
; > On Tue, Jun 5, 2018 at 1:31 PM, Nikolay Izhikov wrote: > > Vyacheslav, > > > > Let's create build plan on TC for this profile. > > Why we need to merge it in master now? > > > > В Вт, 05/06/2018 в 13:29 +0300, Vyacheslav Daradur пишет: > > &g

Re: Upgrading Ignite to JCache 1.1

2018-06-05 Thread Nikolay Izhikov
Alex, please try to run this build plan for your branch https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_JCacheTck11 В Вт, 05/06/2018 в 13:56 +0300, Nikolay Izhikov пишет: > Guys, we had a private talk with Vyacheslav and Dmitriy Pavlov. > Here are its resulst: >

Re: Upgrading Ignite to JCache 1.1

2018-06-05 Thread Nikolay Izhikov
Vyacheslav, Let's create build plan on TC for this profile. Why we need to merge it in master now? В Вт, 05/06/2018 в 13:29 +0300, Vyacheslav Daradur пишет: > Nikolay, there isn't anything broken in PR. > > The PR is needed to add new build-plan on TC. > > We need tools to check that fixes for

Re: Upgrading Ignite to JCache 1.1

2018-06-05 Thread Nikolay Izhikov
Hello, Igniters. Actually, I don't understand why we should merge in master something broken. Currently, Ignite is not ready for JCache 1.1. Only change I see in PR is new profile [1]. Is it required to have it to continue jcache 1.1 support implementation? I think Alexandex can proceed with

Re: errors during ignite mvn compilation with skipTest=false

2018-06-06 Thread Nikolay Izhikov
Hello, vbm. You should add "-DskipTests". Please, check out [1]. mvn clean install -Pall-java,all-scala,licenses *-DskipTests* [1] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ProjectBuild В Ср, 06/06/2018 в 09:47 -0700, vbm пишет: > Hi, > > When I

Re: data extractor

2018-07-01 Thread Nikolay Izhikov
Hello, Dmitriy. Should we support extraction of encrypted data? There will be 2 type of keys we should load to successfully extract data: * master key: keystore + password required. * cache keys: masterkey + access to metastore required. TDE task is almost done, please, take a look. ticket -

TeamCity. Spark Data Frame. RDD suite should be reconfigured

2018-06-25 Thread Nikolay Izhikov
Hello, Igniters. I found that RDD suite doesn't run all scala tests and should be reconfigured. 1. Parameter. TEST_SUITE = "IgniteRDDTestSuite,IgniteDataFrameSuite" 2. Buil Steps. Step3. Additional Maven command line parameters: "-Pall-java,scala,scala-test". Petr, can you edit this settings?

Re: TDE: Upgrade Team City JDK

2018-06-25 Thread Nikolay Izhikov
+0300, Petr Ivanov пишет: > I’ve modified filter requirements for JDK for Spring build configuration to > run only on u151/u171 of JDK-1.8.0 (it will be 55 agent). > Hope this will be enough. > > > > > On 25 Jun 2018, at 12:54, Nikolay Izhikov wrote: > > > > H

Re: TDE: Upgrade Team City JDK

2018-06-26 Thread Nikolay Izhikov
Petr, can you make suggested jdk upgrade? В Пн, 25/06/2018 в 14:28 +0300, Nikolay Izhikov пишет: > Petr. > > One more thing: > > We need to install "Java Cryptography Extension (JCE) Unlimited Strength > Jurisdiction Policy Files" [1] to enable usage of JDK pr

Re: Ignite 2.6 emergency release suggestion

2018-06-26 Thread Nikolay Izhikov
Hello, Igniters. We have private discussion with Vladimir Ozerov and Dmitry Pavlov. IGNITE-6055 has to be included in 2.6, also. Final review from Vladimir required to merge this ticket. Do we have a chance to include this ticket to 2.6? В Вт, 26/06/2018 в 19:26 +0300, Dmitry Pavlov пишет: >

Re: Ignite 2.6 emergency release suggestion

2018-06-26 Thread Nikolay Izhikov
Also, Igniters, be aware https://issues.apache.org/jira/browse/IGNITE-8534 merged to master and cherry picked to 2.6 branch. В Вт, 26/06/2018 в 19:29 +0300, Nikolay Izhikov пишет: > Hello, Igniters. > > We have private discussion with Vladimir Ozerov and Dmitry Pavlov. > >

Re: TDE: Upgrade Team City JDK

2018-06-27 Thread Nikolay Izhikov
herefore restriction) to these > suites. > > On Wed, Jun 27, 2018 at 11:39 AM, Petr Ivanov wrote: > > > IgniteSpiTestSuite is called from ’SPI’ build type and I’ve added agent > > filter there. > > IgniteKernalSelfTestSuite is not found in build types. > > > >

Re: Ignite 2.6 emergency release suggestion

2018-06-27 Thread Nikolay Izhikov
Hello, Dmitriy. I fixed this issue, already. Please, check it. В Ср, 27/06/2018 в 16:55 +0300, Dmitry Pavlov пишет: > Hi Denis, > > currently anyway we have TC issue came from spark version migrations. > > I also currently experiencing challenges with my local environment setup. > So I guess

Re: TDE: Upgrade Team City JDK

2018-06-27 Thread Nikolay Izhikov
7/06/2018 в 18:44 +0300, Dmitry Pavlov пишет: > Hi Nikolay, > > can we add option enables AES 128 for test mode? This should work well > without update env. > > Sincerely, > > ср, 27 июн. 2018 г. в 18:43, Petr Ivanov : > > > > > > > > On 27 Jun 2

Re: TDE: Upgrade Team City JDK

2018-06-25 Thread Nikolay Izhikov
> > > > On 25 Jun 2018, at 10:33, Nikolay Izhikov wrote: > > > > Hello, Igniters. > > > > I'm trying to run tests for TDE branch. > > > > It seems I'm facing with known JDK bug [1] > > Here is log from Team City [2]. > > > &g

Re: TeamCity. Spark Data Frame. RDD suite should be reconfigured

2018-06-26 Thread Nikolay Izhikov
For `RDD` suite, we need use `test` goal instead of `surefire:test`, also. Because, scala tests runs with org.scalatest:scalatest-maven-plugin:2.0.0 В Вт, 26/06/2018 в 14:53 +0300, Nikolay Izhikov пишет: > Please, remove `scala-2.10` from profiles, also. > > В Вт, 26/06/2018 в 14

TDE: Upgrade Team City JDK

2018-06-25 Thread Nikolay Izhikov
Hello, Igniters. I'm trying to run tests for TDE branch. It seems I'm facing with known JDK bug [1] Here is log from Team City [2]. As I can see current Team City jdk is 1.8.0_77-b03 Issue JDK-8149411 fixed in 8u102. Can we upgrade JDK on Team City? [1]

Re: TDE: Upgrade Team City JDK

2018-06-26 Thread Nikolay Izhikov
> Looks like it fixed the issue. > > > > > On 26 Jun 2018, at 19:27, Nikolay Izhikov wrote: > > > > Petr, can you make suggested jdk upgrade? > > > > В Пн, 25/06/2018 в 14:28 +0300, Nikolay Izhikov пишет: > > > Petr. > > > > > >

Re: TDE: Upgrade Team City JDK

2018-06-27 Thread Nikolay Izhikov
> > On 27 Jun 2018, at 00:42, Nikolay Izhikov wrote: > > > > Hello, Petr. > > > > Thanks a lot! > > I see success spring encryption tests now [1] > > > > I've added encryption tests to core module also. > > Can we set same rule for a core modul

Re: TeamCity. Spark Data Frame. RDD suite should be reconfigured

2018-06-26 Thread Nikolay Izhikov
IgniteSessionWriteSpec,IgniteSQLDataFrameSpec,IgniteSQLDataFrameWriteSpec’ > — both ‘IgniteRDDTestSuite' and ‘IgniteDataFrameSuite’ are present. > 2. Step3 Maven task had ‘-Pall-java,scala-2.10,tensorflow’ — should I add > ’scala’ and 'scala-test’ profiles as well? > > > >

Re: TeamCity. Spark Data Frame. RDD suite should be reconfigured

2018-06-26 Thread Nikolay Izhikov
Please, remove `scala-2.10` from profiles, also. В Вт, 26/06/2018 в 14:39 +0300, Petr Ivanov пишет: > I’ve modified list of suites and added ‘scala’ and ’scala-test’ to profiles > list. > Please check build. > > > > > On 26 Jun 2018, at 14:28, Nikolay Izhikov wrote: &

Re: Spark data frames integration merged

2017-12-29 Thread Nikolay Izhikov
ata frame into Ignite via > DataFrameWrite API [3]. > > [1] https://issues.apache.org/jira/browse/IGNITE-3084 > [2] https://issues.apache.org/jira/browse/IGNITE-7077 > [3] https://issues.apache.org/jira/browse/IGNITE-7337 > > Nikolay Izhikov, thanks for the contribution and for all the hard > work! > > -Val

Re: Spark data frames integration merged

2018-01-05 Thread Nikolay Izhikov
Hello, guys. Currently `getPreferredLocations` implemented in `IgniteRDD -> IgniteAbstractRDD`. But DataFrame implementation uses `IgniteSQLDataFrameRDD -> IgniteSqlRDD -> IgniteAbstractRDD` Where `->` is extension. So, for now, getPreferredLocation doesn't implemented for a IgniteDataFrame.

Re: Scala 2.10 support

2018-01-10 Thread Nikolay Izhikov
+1 from me. В Ср, 10/01/2018 в 05:24 -0700, vveider пишет: > Hi, Igniters! > > > While working on Java 9 support adaptation, came to a problem with > scala > 2.10 and corresponding modules (scala-2.10, spark-2.10 and > visor-console-2.10) which will not compile under Java 9. As I see > here [1]

Re: Spark data frames integration merged

2018-01-08 Thread Nikolay Izhikov
; [1] https://issues.apache.org/jira/browse/IGNITE-3084 > > [2] https://issues.apache.org/jira/browse/IGNITE-7077 > > [3] https://issues.apache.org/jira/browse/IGNITE-7337 > > > > Nikolay Izhikov, thanks for the contribution and for all the hard > > work! > > > > -Val > >

[SparkDataFrame] Query Optimization. Prototype

2018-01-19 Thread Nikolay Izhikov
Hello, guys. I have done prototype of implementation of optimization query from Spark to Ignite [1]. Please, take a look at PR [2]. But still there are some issues I want to clarify with community: * Can I know scale and precision of DECIMAL column? Example - [3] * Ignite have

Re: [SparkDataFrame] Query Optimization. Prototype

2018-01-21 Thread Nikolay Izhikov
in another thread? [1] https://docs.microsoft.com/ru-ru/sql/t-sql/functions/cast-and-convert-transact-sql В Пт, 19/01/2018 в 14:18 -0800, Valentin Kulichenko пишет: > Hi Nikolay, > > I will review this sometime next week. > > -Val > > On Fri, Jan 19, 2018 at 2:4

Re: [SparkDataFrame] Query Optimization. Prototype

2018-01-23 Thread Nikolay Izhikov
Serge, thank you! Vladimir, guys, can you take a look at another questions: * Can I know scale and precision of DECIMAL column? Example - [3] * Ignite have some limitation for a *distributed* join. For example, we can execute join only for indexed columns. Example - [4].

[SparkDataFrame] Saving DataFrame to Ignite

2018-01-26 Thread Nikolay Izhikov
Hello, guys. I've finish implementation of IGNITE-7337 [1] Support of saving data frame to Ignite. Please, review. Valentin, I think it would be better to review this task prior IGNITE-7077 (query optimization). Because, this issue is relatively smaller and easier to review. And, from my point

How to delete all data from SQL table

2018-01-25 Thread Nikolay Izhikov
Hello, guys. I wonder what is best way to delete all rows from table? Other RDBMS offer command like `TRUNCATE TABLE` or similar. Is it would be faster if user execute `DROP TABLE XXX`, `CREATE TABLE XXX` instead of `DELETE FROM XXX`?

Ignite SQL: How to drop table without explicit cache name

2018-02-01 Thread Nikolay Izhikov
Hello, guys. I working on support of saving Spark DataFrame to Ignite [1]. I want to execute "DROP TABLE XXX" query. To execute some SQL query in regulary way one need some cache. That cache has to differs from table cache to execute "DROP TABLE" successfully. I founded 3 different ways to

Saving Spark Data Frames merged to master

2018-02-08 Thread Nikolay Izhikov
Hello, Igniters. Good news. IGNITE-7337 [1](Spark Data Frames: support saving a data frame in Ignite) are merged to master. For now we can both - read from and write to Ignite SQL table with Data Frame API. Big thanks to Valentin Kulichenko for a quick review. So it seems we can include this

Re: Apache Ignite 2.4 release

2018-02-08 Thread Nikolay Izhikov
done faster though. > > > Alexey Goncharuk, Dmitriy Pavlov, any ideas? > > D. > > On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov <nizhi...@apache.org> wrote: > > > Hello, Igniters. > > > > Please, consider including IGNITE-7337 - Spark Dat

Continuous Query With Transformer are ready for merge

2018-02-06 Thread Nikolay Izhikov
Hello, Igniters. Improvement of Continuous Query public API are ready for merge. AFAIK Anton Vinogradov and Yakov Zhdanov are happy with proposed API. [1] With this improvement we can transform entry on the server node before sending it to the listener to reduce network utilization. Example of

IGNITE-6005 is ready for review

2018-02-12 Thread Nikolay Izhikov
Hello, Igniters. I've fix some relatively small issue(100 loc) IGNITE-6005 [1], PR [2] The fix in DataStructuresProcessor.java As far as I can see from git log Ilya Lantukh maintains this piece of code. Ilya, can you please take a look at the fix? [1]

Re: Apache Ignite 2.4 release

2018-02-11 Thread Nikolay Izhikov
Hello, Dmitriy. Data Frame Read feature is documented in https://issues.apache.org/jira/browse/IGNITE-7345 AFAIK page isn't published yet. I will document Data Frame Save feature in a few days. There is a ticket for it - https://issues.apache.org/jira/browse/IGNITE-7655 В Вс, 11/02/2018 в

API to enlist running user tasks

2018-02-13 Thread Nikolay Izhikov
Hello, Igniters. We have some requests from users [1] to have ability to get list of all running continuous queries. I propose to implement such ability. To implement it we have to extend our JMX beans to provide the following information: * List of continuous queries for cache: *

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Nikolay Izhikov
nent, please feel free to create it on your own. As > far as testing of joins, I think it makes sense to start working on it when > we finish ANSI compliance testing which is already in progress. > > On Wed, Jan 24, 2018 at 12:27 PM, Nikolay Izhikov <nizhikov@gmail.com> > wro

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Nikolay Izhikov
ex should not cause query to > fail. What is the exception? > > -Val > > On Tue, Feb 13, 2018 at 8:07 AM, Nikolay Izhikov <nizhi...@apache.org> wrote: > > Hello, Valentin. > > > > > When you're talking about join optimization, what exactly are you >

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Nikolay Izhikov
t; find one under "sql" component, please feel free to create it on your own. > > As far as testing of joins, I think it makes sense to start working on it > > when we finish ANSI compliance testing which is already in progress. > > > > On Wed, Jan 24, 2018 at

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Nikolay Izhikov
to create it on your own. > > As far as testing of joins, I think it makes sense to start working on it > > when we finish ANSI compliance testing which is already in progress. > > > > On Wed, Jan 24, 2018 at 12:27 PM, Nikolay Izhikov <nizhikov@gmail.com> > >

Re: IGNITE-5714 Context switching for pessimistic transactions

2018-02-13 Thread Nikolay Izhikov
Hello, Alexey. Could you please, write little more about your implementation 1. Design of implementation. 2. Advantages of you implementation in compare with other ways? 3. Transactions performance penalties/improvements? В Вт, 13/02/2018 в 14:17 +, ALEKSEY KUZNETSOV пишет: > Hi,

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-13 Thread Nikolay Izhikov
you enabled non-collocated joins. I was not > aware of this limitation though, do we have this documented somewhere? > > -Val > > On Tue, Feb 13, 2018 at 8:21 AM, Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Val, > > > > Source code check: https://gi

Re: Spark data frames

2018-02-14 Thread Nikolay Izhikov
Hello, Dmitry. If other community member are also iterested in that kind of information I can try to do the talk. В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет: > Hi Nikolay, > > I've notices there are a number of very lively discussions on dev list about > SparkDataFrames. But I, for

Re: API to enlist running user tasks

2018-02-13 Thread Nikolay Izhikov
ation timestamp (?) > >* start timestamp (?) > > > > * List of running jobs for node: > >* node ID task started from. > >* task class name. > >* other job parameters. > >* creation timestamp (?) > >* start timestamp (?) &

Re: API to enlist running user tasks

2018-02-13 Thread Nikolay Izhikov
how about regular SQL queries? > > On Tue, Feb 13, 2018 at 6:31 AM, Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Hello, Igniters. > > > > We have some requests from users [1] to have ability to get list of all > > running continuous que

Re: Saving Spark Data Frames merged to master

2018-02-09 Thread Nikolay Izhikov
ging something to 2.4 except blockers is not possible at this phase of > release process. > > Hope, 2.5, with your changes, will be released soon :) > > On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Hello, Igniters. > &

Re: Continuous Query With Transformer are ready for merge

2018-02-08 Thread Nikolay Izhikov
Anton, The merge is done! Thank you, very much. Your help can't be overestimated. В Чт, 08/02/2018 в 12:38 +0300, Anton Vinogradov пишет: > Looks like no one have objections. > > Nikolay, > Let's merge. > > On Tue, Feb 6, 2018 at 8:45 PM, Nikolay Izhikov <nizhi

Re: Apache Ignite 2.4 release

2018-02-07 Thread Nikolay Izhikov
Hello, Igniters. Please, consider including IGNITE-7337 - Spark Data Frames: support saving a data frame in Ignite [1] in the 2.4 release. It seems we can merge it into the master in a day or few. [1] https://issues.apache.org/jira/browse/IGNITE-7337 В Ср, 07/02/2018 в 08:35 -0800, Denis

TeamCity. Ignite RDD tests

2018-02-15 Thread Nikolay Izhikov
Hello, Igniters. I'm working on issue [1]. Team City doesn't collect info about scalatest execution because of wrong pom.xml I'm fixed it in PR [3] It happens there is a 2 broken tests written in Scala - [4] 1. IgniteRDDSpec.IgniteRDD should successfully store data to ignite using saveValues

Re: Spark data frames

2018-02-16 Thread Nikolay Izhikov
> > > Walkthrough" by Denis Magda, which has been presented some times ago. > > > > > > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan > > > <dsetrak...@apache.org> wrote: > > > > I am definitely interested. Great idea! > > >

Re: TeamCity. Ignite RDD tests

2018-02-16 Thread Nikolay Izhikov
with reference to issue in JIRA. > > Sincerely, > Dmitriy Pavlov > > пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov <nizhi...@apache.org>: > > > Hello, Igniters. > > > > I'm working on issue [1]. > > > > Team City doesn't collect info about scalatest

Re: TeamCity. Ignite RDD tests

2018-02-16 Thread Nikolay Izhikov
Hello, Dmitriy. Seems, that I haven't suffictient access to do test mute. Can you give me access? Or please, mute it. В Пт, 16/02/2018 в 12:03 +, Dmitry Pavlov пишет: > I think you should be able to mute any test. If not, please write to me. > > пт, 16 февр. 2018 г. в 15:02, Nikola

Re: [SparkDataFrame] Query Optimization. Prototype

2018-02-15 Thread Nikolay Izhikov
6361.html > > > > В Вт, 13/02/2018 в 09:55 -0800, Valentin Kulichenko пишет: > > > Nikolay, > > > > > > Looks like this is because you enabled non-collocated joins. I was not > > > aware of this limitation though, do we have this docum

Re: Apache Ignite 2.4 release

2018-02-15 Thread Nikolay Izhikov
+1 В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет: > Igniters, > > AI 2.4 release was shifted a bit and over this time we implemented two > important SQL features: > 1) COPY command for fast file upload to the cluster [1] > 2) Streaming mode for thin driver [2] > > Both commands are very

IGNITE-7409. Ready to be merged

2018-02-15 Thread Nikolay Izhikov
Hello, Igniters. There is an issue [1] with very small patch - 4 LOC. State is Patch Available. This part of code was introduced be me in [2]. I reviewed patch and want to merge it. Any objections? [1] https://issues.apache.org/jira/browse/IGNITE-7409 [2]

Re: IGNITE-6005 is ready for review

2018-02-21 Thread Nikolay Izhikov
of this fix causes 100% timeout in Data Structures suite and wastes > > agents for a long time. > > > > Sincerely, > > Dmitriy Pavlov > > > > пн, 12 февр. 2018 г. в 17:54, Ilya Lantukh <ilant...@gridgain.com>: > > > > > Hi Nikolay, > > &

Re: IgniteDataStreamer silently fails on a server node

2018-02-21 Thread Nikolay Izhikov
ted to that > future. Do you not see it there? > > D. > > On Mon, Feb 19, 2018 at 1:00 PM, Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Hello, Igniters. > > > > While working on IGNITE-7727 I found strange behavior of > > IgniteDataStreamer: > &

Re: IGNITE-6005 is ready for review

2018-02-21 Thread Nikolay Izhikov
> And it can be failed with link to > https://issues.apache.org/jira/browse/IGNITE-5975 > > > Sincerely, > Dmitriy Pavlov > > чт, 22 февр. 2018 г. в 10:07, Nikolay Izhikov <nizhi...@apache.org>: > > > Andrey, > > > > I made TC run. > >

Re: IGNITE-6005 is ready for review

2018-02-21 Thread Nikolay Izhikov
apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll=buildTypeStatusDiv_IgniteTests24Java8=pull%2F2773%2Fhead > > > > So, Nikolay, I suggest to start RunAll in this suite group. > > > > I can check results later in compare with master. > > > > ср, 21 февр.

Re: TeamCity. Ignite RDD tests

2018-02-22 Thread Nikolay Izhikov
> > Please see investigations assigned to you. What can we change to avoid > these tests to be included into run other suites? > > Sincerely, > Dmitriy Pavlov > > пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov <nizhi...@apache.org>: > > > Hello, Igniters. > &g

Re: TeamCity. Ignite RDD tests

2018-02-22 Thread Nikolay Izhikov
possible IgniteRDDSpec.IgniteRDD became unit tests, which is started > with code compile? > > This can be reason these tests appeared in suites which is not related to > IgniteRDD, such as Scala Examples. > > чт, 22 февр. 2018 г. в 13:20, Nikolay Izhikov <nizhi...@apac

Re: TeamCity. Ignite RDD tests

2018-02-20 Thread Nikolay Izhikov
Hello, Dmitriy. Sorry, I thought I did it by myself. Thank you, anyway. В Вт, 20/02/2018 в 15:24 +, Dmitry Pavlov пишет: > Hi Nikolay, > > A number of TC tests began to fail with link to > https://issues.apache.org/jira/browse/IGNITE-7727 > > I've muted all tests, which I found. Please

Re: IgniteDataStreamer silently fails on a server node

2018-02-22 Thread Nikolay Izhikov
Looks like second attempt is successful for such case. > > On Wed, Feb 21, 2018 at 1:36 PM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > > > On Wed, Feb 21, 2018 at 12:10 AM, Nikolay Izhikov <nizhi...@apache.org> > > wrote: > > > >

IgniteDataStreamer silently fails on a server node

2018-02-19 Thread Nikolay Izhikov
Hello, Igniters. While working on IGNITE-7727 I found strange behavior of IgniteDataStreamer: If we have IgniteUuid as an indexed type update silently brokes on a server node. Client doesn't have any notification about fails. All calls of `addData`, `close`, etc. succeed on a client side but

Re: List of running Continuous queries or CacheEntryListener per cache or node

2017-12-27 Thread Nikolay Izhikov
> > Thanks! > -Dmitry > > 21.12.2017 13:18, Nikolay Izhikov пишет: > > Hello, Dmitry. > > > > I think it a great idea. > > > > Do we a feature to list all running ComputeTasks? > > > > I, personally, think we have to implement possibility

Re: IEP-22: Direct Data Load proposal

2018-06-20 Thread Nikolay Izhikov
Hello, Vladimir. Does this IEP fit with IEP-18: TDE? Do we allow to user to load data into encrypted cache? В Ср, 20/06/2018 в 18:08 +0300, Vladimir Ozerov пишет: > Igniters, > > Initial data load is one of the most important use cases for our product. > This is one the first things user try

Re: JCache 1.1: IGNITE-8715 Problems with Closeable objects from Factory

2018-06-21 Thread Nikolay Izhikov
Hello, Alex. Issue looks trivial from your description. Can you clarify - What is the issues with obvious decision you describe? В Чт, 21/06/2018 в 17:48 +0300, Александр Меньшиков пишет: > Hi, Igniters! > Can someone give me advice about IGNITE-8715 [1]? > It's a part of IEP-21 [2] about

Re: Entries are not loaded into cache during cache store loading

2018-06-19 Thread Nikolay Izhikov
Hello, Aleksey. By "cache store" you mean 3-d party cache store [1] ? https://apacheignite.readme.io/docs/3rd-party-store В Вт, 19/06/2018 в 13:10 +0300, Aleksey Kuznetsov пишет: > Hi, Igniters! > > During test bug fixing [1], I stumbled on the following issue: > When we perform cache store

Re: Exchange stucks while node restoring state from WAL

2018-08-03 Thread Nikolay Izhikov
Hello, Maxim. > 1) Is it correct that readMetastore() happens after node starts> but before > including node into the ring?> I think yes. You can have some kind of metainformation required on node join. > 5) Does in our final solution for new joined node readMetastore> and > restoreMemory

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-27 Thread Nikolay Izhikov
Hello, Stuart. I like your idea. > 1. Ignite BinaryObjects, in which case we’d need to supply a Spark Encoder > implementation for BinaryObjects > 2. Kryo-serialised versions of the objects. Seems like first option is simple adapter. Am I right? If yes, I think it's a more efficient way

Re: Apache Ignite 2.7: scope, time and release manager

2018-07-27 Thread Nikolay Izhikov
> > > > > > p. > > > > > > > > [1] https://jira.apache.org/jira/browse/IGNITE-7782 > > > > [2] https://jira.apache.org/jira/browse/IGNITE-7783 > > > > [3] https://jira.apache.org/jira/browse/IGNITE- > > > > > > > > > >

TDE. Phase 1 is ready for review

2018-07-27 Thread Nikolay Izhikov
Hello, Igniters. TDE. Phase 1 [1] is ready for review [2]. I meet some corner cases during development I want to describe: 1. Creation of Cache Group Encryption Keys: 1.1. To build cache create request we has to create Cache Group Encryption Key. 1.2. To create Cache Group

Re: Spark DataFrames With Cache Key and Value Objects

2018-07-27 Thread Nikolay Izhikov
UDF so that the > user interface would be standard Spark: > > Dataset dataFrame = ... > DataSet dataSet = > dataFrame.select(“_val_).as(Encoders.kryo(MyValClass.class)) > > I’ll create a ticket and maybe put together a test case for further > discussion? > > Stuart. > &

Re: Apache Ignite 2.7: scope, time and release manager

2018-07-31 Thread Nikolay Izhikov
Hello, Igniters. Release page created [1]. [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7 В Пт, 27/07/2018 в 00:52 -0700, Denis Magda пишет: > Sure, it can wait. Thanks! Take a rest ) > > On Thu, Jul 26, 2018 at 11:39 PM Nikolay Izhikov > wrote: > &

  1   2   3   4   >