Re: Ignite nightly build job on Apache Jenkins server

[Dmitriy Setrakyan]

On Mon, Mar 5, 2018 at 7:58 AM, Pavel Tupitsyn  wrote:

> Hi Igor,
>
> This build did not make much sense anyway:
> 1) (At least) platform artifacts (.NET) can't be built there, so the
> package is incomplete
> 2) We have TeamCity, why duplicate maintenance efforts?
>

What is the link to the nightly Java builds on TeamCity?


>
> Nightly release builds are very useful, but they should be done on
> TeamCity.
> (We have them for .NET NuGet:
> https://cwiki.apache.org/confluence/display/IGNITE/Nightly+Builds)
>
> Thanks,
> Pavel

Re: [VOTE] Apache Ignite 2.4.0 RC1

[Pavel Tupitsyn]

Regarding release notes:

1. No changes in Ignite.C++?

2. > Fixed output of big numbers in SQL query results
Are we including trivial bugfixes now?



On Tue, Mar 6, 2018 at 10:26 AM, Pavel Tupitsyn 
wrote:

> +1
>
> Checked Ignite.NET:
> * Build from sources
> * Download binaries and run examples
> * Build NuGet packages on TeamCity [1] and test them
>
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=1126956
>
> On Mon, Mar 5, 2018 at 10:42 PM, Vladimir Ozerov 
> wrote:
>
>>  Igniters,
>>
>> We have uploaded a 2.4.0 release candidate to
>> https://dist.apache.org/repos/dist/dev/ignite/2.4.0-rc1/
>>
>> Git tag name is
>> 2.4.0-rc1
>>
>> This release includes the following changes:
>>
>> Ignite:
>> * Introduced Baseline Affinity Topology
>> * Ability to disable WAL for cache in runtime through IgniteCluster API or
>> ALTER TABLE command
>> * Added ability to convert WAL to human-readable form
>> * Ability to enable/disable JDBC, ODBC and thin client endpoints
>> * Project moved to Java 8
>> * Java 7 is no longer supported
>> * Apache Ignite can be run on Java 9
>> * Introduced Apache Ignite packaging with RPM
>> * Spark Data Frames support
>> * Added integration with Spring 3.x
>> * Added handler to stop node when IO error is encountered
>> * EvictionPolicyFactory support added to CacheConfiguration
>> * Added JMX metrics for memory regions
>> * Added detailed memory consumption on start and OOM reporting
>> * Added custom thread pools monitoring
>> * Support Ignite MR in Kerberized environment without IGFS
>> * S3 IP finder: support server side encryption and bucket endpoint
>> configuration
>> * Multiple fixes and performance optimizations
>>
>> Ignite.NET:
>> * Thin Client initial release
>> * .NET Core / Linux / macOS support
>> * Mono platform support
>> * Removed C++ JNI layer
>>
>> SQL:
>> * Added DEFAULT support to CREATE TABLE command
>> * Added ALTER TABLE DROP COLUMN support
>> * Added INLINE_SIZE option to CREATE INDEX command
>> * Added PARALLEL option to CREATE INDEX command
>> * Added optional on-heap row cache
>> * INSERT command now can be executed without explicit column names (INSERT
>> INTO table VALUES ...)
>> * Allowed multiple caches to share the same schema
>> * Added support for Java 8 Date and Time API
>> * Added "notNull" property to QuerySqlField annotation
>> * Improved COUNT(*) performance
>> * Fixed a problem causing an exception in certain nodes do not have
>> primary
>> partitions for REPLICATED cache
>> * Fixed per-thread H2 connection leak.
>> * Fixed partition exchange hang due to SQL query running inside a
>> transaction.
>>
>> JDBC Driver:
>> * Optimized batched operations processing
>>
>> ODBC Driver:
>> * Added support of multiple statements execution with one call
>> * Added support of SQL_ATTR_CONNECTION_TIMEOUT
>> * Added support of SQL_ATTR_QUERY_TIMEOUT
>> * Optimized batched operations processing
>> * Proper handling of ungraceful TCP disconnects (keep-alive connection)
>> * Fixed error preventing close of executed DML statement
>> * Fixed table names returned by SQLTables for DDL-created tables
>>
>> Machine Learning:
>> * Implemented Fuzzy c-means algorithm
>> * Implemented gradient descent for OLS Linear Regression
>> * Implemented K nearest neighbor algorithm
>> * Introduced API for datasets
>> * Introduced API for ML model and trainers
>> * Introduced common mechanism for group training of models
>> * Added distributed version of multilayer perceptron
>> * Added local version of multilayer perceptron
>> * Added basic import/export functionality for ml models
>> * Added decision tree algorithm
>> * Performance optimizations for distributed operations
>>
>> Web Console:
>> * Implemented component for cluster selection and activation
>> * Implemented support for multiple statements on Queries screen
>> * Implemented progress indication for 'execute', 'scan' and 'export'
>> actions on Queries screen
>> * Added support for ClientConnectorConfiguration
>> * Added several SQL examples on Queries screen
>> * Added "Connected clusters" component in header
>> * Added support for custom SMTP server configuration
>> * Added detection of CSV separator based on browser locale
>> * Added "Copy to clipboard" action on Queries screen
>> * Added duration and node ID in results header and 'Show query' modal
>> * Improved support for binary JDBC types on import from RDBMS
>> * Fixed Web Console under IE11
>> * Fixed configuration generation for imported model in case of Oracle
>> NUMBER(5) data type
>> * Fixed output of big numbers in SQL query results
>>
>> Visor:
>> * Added "-quiet" flag for batch mode
>> * Added ability to start cache rebalance
>> * Added output of data region metrics to "node" command
>> * Added dialog to put/get/remove values to/from cache for simple key types
>> * Show valid message for caches when cluster is inactive
>> * Fixed 'cache -a' in case of node filter
>> * Fixed reading last command line in 

Re: [VOTE] Apache Ignite 2.4.0 RC1

[Pavel Tupitsyn]

+1

Checked Ignite.NET:
* Build from sources
* Download binaries and run examples
* Build NuGet packages on TeamCity [1] and test them

[1] https://ci.ignite.apache.org/viewLog.html?buildId=1126956

On Mon, Mar 5, 2018 at 10:42 PM, Vladimir Ozerov 
wrote:

>  Igniters,
>
> We have uploaded a 2.4.0 release candidate to
> https://dist.apache.org/repos/dist/dev/ignite/2.4.0-rc1/
>
> Git tag name is
> 2.4.0-rc1
>
> This release includes the following changes:
>
> Ignite:
> * Introduced Baseline Affinity Topology
> * Ability to disable WAL for cache in runtime through IgniteCluster API or
> ALTER TABLE command
> * Added ability to convert WAL to human-readable form
> * Ability to enable/disable JDBC, ODBC and thin client endpoints
> * Project moved to Java 8
> * Java 7 is no longer supported
> * Apache Ignite can be run on Java 9
> * Introduced Apache Ignite packaging with RPM
> * Spark Data Frames support
> * Added integration with Spring 3.x
> * Added handler to stop node when IO error is encountered
> * EvictionPolicyFactory support added to CacheConfiguration
> * Added JMX metrics for memory regions
> * Added detailed memory consumption on start and OOM reporting
> * Added custom thread pools monitoring
> * Support Ignite MR in Kerberized environment without IGFS
> * S3 IP finder: support server side encryption and bucket endpoint
> configuration
> * Multiple fixes and performance optimizations
>
> Ignite.NET:
> * Thin Client initial release
> * .NET Core / Linux / macOS support
> * Mono platform support
> * Removed C++ JNI layer
>
> SQL:
> * Added DEFAULT support to CREATE TABLE command
> * Added ALTER TABLE DROP COLUMN support
> * Added INLINE_SIZE option to CREATE INDEX command
> * Added PARALLEL option to CREATE INDEX command
> * Added optional on-heap row cache
> * INSERT command now can be executed without explicit column names (INSERT
> INTO table VALUES ...)
> * Allowed multiple caches to share the same schema
> * Added support for Java 8 Date and Time API
> * Added "notNull" property to QuerySqlField annotation
> * Improved COUNT(*) performance
> * Fixed a problem causing an exception in certain nodes do not have primary
> partitions for REPLICATED cache
> * Fixed per-thread H2 connection leak.
> * Fixed partition exchange hang due to SQL query running inside a
> transaction.
>
> JDBC Driver:
> * Optimized batched operations processing
>
> ODBC Driver:
> * Added support of multiple statements execution with one call
> * Added support of SQL_ATTR_CONNECTION_TIMEOUT
> * Added support of SQL_ATTR_QUERY_TIMEOUT
> * Optimized batched operations processing
> * Proper handling of ungraceful TCP disconnects (keep-alive connection)
> * Fixed error preventing close of executed DML statement
> * Fixed table names returned by SQLTables for DDL-created tables
>
> Machine Learning:
> * Implemented Fuzzy c-means algorithm
> * Implemented gradient descent for OLS Linear Regression
> * Implemented K nearest neighbor algorithm
> * Introduced API for datasets
> * Introduced API for ML model and trainers
> * Introduced common mechanism for group training of models
> * Added distributed version of multilayer perceptron
> * Added local version of multilayer perceptron
> * Added basic import/export functionality for ml models
> * Added decision tree algorithm
> * Performance optimizations for distributed operations
>
> Web Console:
> * Implemented component for cluster selection and activation
> * Implemented support for multiple statements on Queries screen
> * Implemented progress indication for 'execute', 'scan' and 'export'
> actions on Queries screen
> * Added support for ClientConnectorConfiguration
> * Added several SQL examples on Queries screen
> * Added "Connected clusters" component in header
> * Added support for custom SMTP server configuration
> * Added detection of CSV separator based on browser locale
> * Added "Copy to clipboard" action on Queries screen
> * Added duration and node ID in results header and 'Show query' modal
> * Improved support for binary JDBC types on import from RDBMS
> * Fixed Web Console under IE11
> * Fixed configuration generation for imported model in case of Oracle
> NUMBER(5) data type
> * Fixed output of big numbers in SQL query results
>
> Visor:
> * Added "-quiet" flag for batch mode
> * Added ability to start cache rebalance
> * Added output of data region metrics to "node" command
> * Added dialog to put/get/remove values to/from cache for simple key types
> * Show valid message for caches when cluster is inactive
> * Fixed 'cache -a' in case of node filter
> * Fixed reading last command line in batch mode
> * Updated eviction policy factory in configs
>
> Complete list of closed issues:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%
> 20fixVersion%20%3D%202.4%20AND%20(status%20%3D%
> 20closed%20or%20status%20%3D%20resolved)
>
> DEVNOTES
> https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=blob_
> 

Re: checksum file Release Distribution Policy

[Dmitriy Setrakyan]

On Mon, Mar 5, 2018 at 12:57 PM, Denis Magda  wrote:

> Igniters,
>
> Do we comply with the next release requirements? Vladimir as a 2.4 release
> manager, could you double check that we are in a good state?
>

I think we do. We only provide SHA files for our releases.

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

Thank you, it's - nizhikov

В Пн, 05/03/2018 в 15:09 -0800, Denis Magda пишет:
> Nikolay, what's your Wiki ID? I'll grant you required permissions.
> 
> --
> Denis
> 
> On Sun, Mar 4, 2018 at 11:00 PM, Nikolay Izhikov  wrote:
> > Hello, Denis.
> > 
> > > I would encourage you creating an IEP
> > 
> > That is exactly what we want to do :)
> > 
> > But seems I have not sufficient privileges to do it on Ignite wiki.
> > 
> > https://cwiki.apache.org/confluence/display/IGNITE/Active+Proposals
> > 
> > Can you or someone give me such rights?
> > 
> > В Чт, 01/03/2018 в 22:23 -0800, Denis Magda пишет:
> > > Dmitriy R., Nilokay,
> > >
> > > Thanks for the analysis and handout of the architectural design. No 
> > > doubts,
> > > it would be a valuable addition to Ignite.
> > >
> > > I would encourage you creating an IEP on the wiki and break the work into
> > > pieces discussing specific part with the community.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Thu, Mar 1, 2018 at 9:29 PM, Nikolay Izhikov  
> > > wrote:
> > >
> > > > Hello, Dmitriy.
> > > >
> > > > Thank you for feedback!
> > > >
> > > > > Will it be supported?
> > > >
> > > > Yes.
> > > >
> > > > TDE shouldn't broke any of existing Ignite features.
> > > > It adds some encrypt/decrypt level when we writing and reading pages
> > > > in/from PDS.
> > > >
> > > > В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет:
> > > > > I have looked at the design, but could not find anything about running
> > > >
> > > > SQL
> > > > > queries against the encrypted data. Will it be supported?
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov 
> > > >
> > > > wrote:
> > > > >
> > > > > > Hell, Dima!
> > > > > >
> > > > > > Thank you for document!
> > > > > >
> > > > > > I'm ready to implement this feature with you.
> > > > > >
> > > > > > Igniters, please, share you thoughts about proposed design
> > > > > >
> > > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > > > > >
> > > > > > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > > > > > > Hello, Igniters!
> > > > > > >
> > > > > > > I investigated the issue and wrote some details in a draft 
> > > > > > > document
> > > > > > > [1]. I think we should made IEP for TDE because it is a big change
> > > >
> > > > and
> > > > > > > should be described in a single place, but not in a message
> > > > > > > conversation.
> > > > > > > Please, look it and write your thoughts. What is not 
> > > > > > > understandable,
> > > > > > > what should be detailed or described?
> > > > > > >
> > > > > > > > Where are we going to store keys (MEK) physically? Would it be
> > > >
> > > > PKCS#11
> > > > > > > > storage? Where we will store passwords to unlock storage or it
> > > >
> > > > will be
> > > > > > > > responibilty of user?
> > > > > > >
> > > > > > > I think we should provide interface for MEK storage to let users 
> > > > > > > use
> > > > > > > storages they want. I suppose at the first step we should provide
> > > >
> > > > very
> > > > > > > simple implementation, which will store MEK on every node and MEK
> > > >
> > > > will
> > > > > > > be extracted by administrator during cluster activation process. 
> > > > > > > Once
> > > > > > > MEK is extracted from key store, we decrypt CEKs and destroy open
> > > >
> > > > MEK,
> > > > > > > leaving open only cache keys.
> > > > > > >
> > > > > > > I think external storage is user's worry and we shouldn't give 
> > > > > > > users
> > > > > > > built-in external storage like Oracle Wallet or Microsoft Azure 
> > > > > > > Key
> > > > > > > Vault because it will increase Ignite's complexity too much.
> > > > > > >
> > > > > > > And yes, we should to comply with the standards like PKCS#11.
> > > > > > >
> > > > > > > > One more thing is how "node gets MEK from coordinator", if we 
> > > > > > > > send
> > > > > > > > cleartext MEK, such security becomes useless also.
> > > > > > >
> > > > > > > Yeah, that's why we should use secured connection. As I know, we 
> > > > > > > have
> > > > > > > SSL implementation over JDK implementation, am I right? But we 
> > > > > > > must
> > > > > > > ensure to use latest SSL/TLS version.
> > > > > > >
> > > > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> 
> 

signature.asc
Description: This is a digitally signed message part

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Denis Magda]

Nikolay, what's your Wiki ID? I'll grant you required permissions.

--
Denis

On Sun, Mar 4, 2018 at 11:00 PM, Nikolay Izhikov 
wrote:

> Hello, Denis.
>
> > I would encourage you creating an IEP
>
> That is exactly what we want to do :)
>
> But seems I have not sufficient privileges to do it on Ignite wiki.
>
> https://cwiki.apache.org/confluence/display/IGNITE/Active+Proposals
>
> Can you or someone give me such rights?
>
> В Чт, 01/03/2018 в 22:23 -0800, Denis Magda пишет:
> > Dmitriy R., Nilokay,
> >
> > Thanks for the analysis and handout of the architectural design. No
> doubts,
> > it would be a valuable addition to Ignite.
> >
> > I would encourage you creating an IEP on the wiki and break the work into
> > pieces discussing specific part with the community.
> >
> > --
> > Denis
> >
> >
> > On Thu, Mar 1, 2018 at 9:29 PM, Nikolay Izhikov 
> wrote:
> >
> > > Hello, Dmitriy.
> > >
> > > Thank you for feedback!
> > >
> > > > Will it be supported?
> > >
> > > Yes.
> > >
> > > TDE shouldn't broke any of existing Ignite features.
> > > It adds some encrypt/decrypt level when we writing and reading pages
> > > in/from PDS.
> > >
> > > В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет:
> > > > I have looked at the design, but could not find anything about
> running
> > >
> > > SQL
> > > > queries against the encrypted data. Will it be supported?
> > > >
> > > > D.
> > > >
> > > > On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov  >
> > >
> > > wrote:
> > > >
> > > > > Hell, Dima!
> > > > >
> > > > > Thank you for document!
> > > > >
> > > > > I'm ready to implement this feature with you.
> > > > >
> > > > > Igniters, please, share you thoughts about proposed design
> > > > >
> > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > > > >
> > > > > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > > > > > Hello, Igniters!
> > > > > >
> > > > > > I investigated the issue and wrote some details in a draft
> document
> > > > > > [1]. I think we should made IEP for TDE because it is a big
> change
> > >
> > > and
> > > > > > should be described in a single place, but not in a message
> > > > > > conversation.
> > > > > > Please, look it and write your thoughts. What is not
> understandable,
> > > > > > what should be detailed or described?
> > > > > >
> > > > > > > Where are we going to store keys (MEK) physically? Would it be
> > >
> > > PKCS#11
> > > > > > > storage? Where we will store passwords to unlock storage or it
> > >
> > > will be
> > > > > > > responibilty of user?
> > > > > >
> > > > > > I think we should provide interface for MEK storage to let users
> use
> > > > > > storages they want. I suppose at the first step we should provide
> > >
> > > very
> > > > > > simple implementation, which will store MEK on every node and MEK
> > >
> > > will
> > > > > > be extracted by administrator during cluster activation process.
> Once
> > > > > > MEK is extracted from key store, we decrypt CEKs and destroy open
> > >
> > > MEK,
> > > > > > leaving open only cache keys.
> > > > > >
> > > > > > I think external storage is user's worry and we shouldn't give
> users
> > > > > > built-in external storage like Oracle Wallet or Microsoft Azure
> Key
> > > > > > Vault because it will increase Ignite's complexity too much.
> > > > > >
> > > > > > And yes, we should to comply with the standards like PKCS#11.
> > > > > >
> > > > > > > One more thing is how "node gets MEK from coordinator", if we
> send
> > > > > > > cleartext MEK, such security becomes useless also.
> > > > > >
> > > > > > Yeah, that's why we should use secured connection. As I know, we
> have
> > > > > > SSL implementation over JDK implementation, am I right? But we
> must
> > > > > > ensure to use latest SSL/TLS version.
> > > > > >
> > > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
>

[jira] [Created] (IGNITE-7886) Provide a list of node Ids on the Baseline page on Web Console

[Prachi Garg (JIRA)]

Prachi Garg created IGNITE-7886:
---

 Summary: Provide a list of node Ids on the Baseline page on Web 
Console
 Key: IGNITE-7886
 URL: https://issues.apache.org/jira/browse/IGNITE-7886
 Project: Ignite
  Issue Type: Bug
Reporter: Prachi Garg
Assignee: Alexey Kuznetsov
 Fix For: 2.4


It seems that the only way to get node IDs (needed for baseline topology) is 
from Web Console ->Monitoring ->Dashboard ->Node -> select node -> click Info 
-> General. However, Monitoring is not part of Ignite. Additionally, on the 
baseline page, on the web console, there is no list of nodes (or node Ids) to 
select from; it's blank. There should be a list of nodes that can be selected 
for baseline topology. Could this issue be related to - 
https://issues.apache.org/jira/browse/IGNITE-7885 ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: MTCGA: Tests of the week

[Denis Magda]

Astonishing progress! Thanks for keep driving this activity and making the
project more and more reliable.

--
Denis

On Mon, Mar 5, 2018 at 1:47 PM, Dmitry Pavlov  wrote:

> Hi, Denis, Igniters,
>
> I've updated the report for the last weekend mass run-all, please find it
> in wiki
> https://cwiki.apache.org/confluence/pages/viewpageattachments.action?
> pageId=73631266
>
> In 30 clean runs, there was 371 failed tests, 21 suites. During previous
> mass run there were 804 tests including 355 C ++, suites 29. This indicate
> progress, but still there is a lot of work to be done.
>
> Thanks to Alexey Goncharuk, Igor Sapego, and Maxim Muzafarov and all folks
> were involved into this previous weeks.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 27 февр. 2018 г. в 13:00, Dmitry Pavlov :
>
> > Hi Denis,
> >
> > It is not easy to say exact number of tests, because test failures are
> > quite randomized. Correct metric can be obtained from mass 'RunAll' runs,
> > for example, during weekend.
> >
> > In the same time brief estimation gives number of failures around 30-60
> > per 'RunAll' instead of 60-80 (we don't count 355 CPP test here because
> it
> > is one problem).
> >
> > If we check issues create/resolve chart, it is available
> > https://cwiki.apache.org/confluence/display/IGNITE/
> Make+Teamcity+Green+Again,
> > progress is obivious, ~ 27 issues reported and ~ 13 were resolved.
> >
> > Hope someone in the community can pick up remaining unsassigned issues:
> >
> > https://issues.apache.org/jira/issues/?jql=labels%
> 3DMakeTeamcityGreenAgain%20AND%20createdDate%3E-3w%
> 20AND%20resolution%20is%20EMPTY%20and%20assignee%20is%20EMPTY
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> >
> > вт, 27 февр. 2018 г. в 0:45, Denis Magda :
> >
> >> Hi Dmitriy,
> >>
> >> Could you share some statistics? How many tests were fixed or merged in
> >> one? Just curiuos.
> >>
> >> --
> >> Denis
> >>
> >> On Mon, Feb 26, 2018 at 9:36 AM, Dmitry Pavlov 
> >> wrote:
> >>
> >> > Hi Folks,
> >> >
> >> > Fixing the tests already brings some results. The number of falling
> >> tests
> >> > in a single run was reduced. Thanks to all, especially to Peter
> Ivanov,
> >> > Alexey Goncharuk, Pavel Kovalenko
> >> >
> >> > There are 4 tickets that were not picked up by community (list at the
> >> end
> >> > of letter). Please look through and assign issue if you are able to
> help
> >> > with this.
> >> >
> >> > Sincerely,
> >> > Dmitriy Pavlov
> >> >
> >> >  - IGNITE-7791 -Ignite Client Nodes: failed test
> >> > IgniteClientReconnectCacheTest.testReconnectCacheDestroyedAnd
> Created()
> >> > - IGNITE-7789 - Ignite Client Nodes: failed test
> >> > IgniteClientReconnectApiExceptionTest.testErrorOnDisconnect()
> >> > - IGNITE-7708 - Ignite Compute has flaky failures in
> >> ConfigVariationTests
> >> > - IGNITE-7692 - affinityCall and affinityRun may execute code on
> backup
> >> > partitions
> >> >
> >>
> >
>

[jira] [Created] (IGNITE-7885) Error on Baseline page on Web Console

[Prachi Garg (JIRA)]

Prachi Garg created IGNITE-7885:
---

 Summary: Error on Baseline page on Web Console
 Key: IGNITE-7885
 URL: https://issues.apache.org/jira/browse/IGNITE-7885
 Project: Ignite
  Issue Type: Bug
Reporter: Prachi Garg
Assignee: Alexey Kuznetsov
 Fix For: 2.4


Every time I go to the baseline page on the web console, I get the following 
error -
{code:java}
Failed to handle request: [req=EXE, 
taskName=org.apache.ignite.internal.visor.compute.VisorGatewayTask, 
params=[null, org.apache.ignite.internal.visor.baseline.VisorBaselineTask, 
org.apache.ignite.internal.visor.baseline.VisorBaselineTaskArg, COLLECT, -1, 
java.util.List, java.lang.String, null], 
err=org.apache.ignite.internal.visor.baseline.VisorBaselineTaskArg]{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Shipping SQL script + CSV files in Ignite examples

[Denis Magda]

The vote passed by lazy consensus :) Here is a ticket:
https://issues.apache.org/jira/browse/IGNITE-7884

Vladimir, look forward to co-operating with you on that task.

--
Denis

On Mon, Feb 26, 2018 at 4:49 PM, Denis Magda  wrote:

> Igniters,
>
> To demonstrate Ignite SQL capabilities at different events, I usually:
>
>1. take an SQL script
>2. preload the data from the script with SQLLine tool that is already
>shipped in Ignite
>3. run DML operations from Web Console, code or other SQL tools
>
> While SQLLine tool is available once Ignite is downloaded, the user has
> to look for a sample SQL script and adapt it to Ignite SQL specification
> (mostly DDL).
>
> Let's ship a script or two in Ignite "examples" folder to facilitate the
> user. In general, I would add :
>
>- DDL + DML script (.sql) that creates a schema and preloads data with
>INSERTS. The script will include a command that turns on/off the streaming
>mode that is in the development.
>- DDL script + CSV file with data. This set of scripts are useful for
>COPY command that should be supported soon.
>
> Presently, I would definitely recommend to include the script below that
> represents a world database of Cities and Countries. It shows how to define
> affinity collocation, how to use Ignite WITH statement, etc:
> https://github.com/dmagda/ignite_world_demo/blob/master/ignite_world.sql
>
> Parts of this database are already in use in our SQL getting started.
>
> As for CSV,  we can take that Cities+Countries files and convert INSERT
> commands to CSV records.
>
> Thoughts, ideas?
>
> --
> Denis
>

[jira] [Created] (IGNITE-7884) Ship SQL scripts with same database in Ignite

[Denis Magda (JIRA)]

Denis Magda created IGNITE-7884:
---

 Summary: Ship SQL scripts with same database in Ignite
 Key: IGNITE-7884
 URL: https://issues.apache.org/jira/browse/IGNITE-7884
 Project: Ignite
  Issue Type: New Feature
  Components: examples, sql
Affects Versions: 2.5
Reporter: Denis Magda
 Fix For: 2.5


As per discussion on @dev list, let's ship two kind of SQL scripts that can be 
used to create and populate a database quickly:

[http://apache-ignite-developers.2346864.n4.nabble.com/Shipping-SQL-script-CSV-files-in-Ignite-examples-td27440.html]

 

The database below is suggested to be used because it represents real data, 
allows setting up affinity collocation and demonstrate meaningful SQL queries 
with joins:

[https://github.com/dmagda/ignite_world_demo/blob/master/ignite_world.sql]

 

First, this script can be used as is and executed in any SQL tool such as 
SQLline.

Second, to demonstrate the use of {{COPY}} command we should take the same 
script, create a file that will store DDL statements and turn INSERTS into a 
CSV file.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: MTCGA: Tests of the week

[Dmitry Pavlov]

Hi, Denis, Igniters,

I've updated the report for the last weekend mass run-all, please find it
in wiki
https://cwiki.apache.org/confluence/pages/viewpageattachments.action?pageId=73631266

In 30 clean runs, there was 371 failed tests, 21 suites. During previous
mass run there were 804 tests including 355 C ++, suites 29. This indicate
progress, but still there is a lot of work to be done.

Thanks to Alexey Goncharuk, Igor Sapego, and Maxim Muzafarov and all folks
were involved into this previous weeks.

Sincerely,
Dmitriy Pavlov

вт, 27 февр. 2018 г. в 13:00, Dmitry Pavlov :

> Hi Denis,
>
> It is not easy to say exact number of tests, because test failures are
> quite randomized. Correct metric can be obtained from mass 'RunAll' runs,
> for example, during weekend.
>
> In the same time brief estimation gives number of failures around 30-60
> per 'RunAll' instead of 60-80 (we don't count 355 CPP test here because it
> is one problem).
>
> If we check issues create/resolve chart, it is available
> https://cwiki.apache.org/confluence/display/IGNITE/Make+Teamcity+Green+Again,
> progress is obivious, ~ 27 issues reported and ~ 13 were resolved.
>
> Hope someone in the community can pick up remaining unsassigned issues:
>
> https://issues.apache.org/jira/issues/?jql=labels%3DMakeTeamcityGreenAgain%20AND%20createdDate%3E-3w%20AND%20resolution%20is%20EMPTY%20and%20assignee%20is%20EMPTY
>
> Sincerely,
> Dmitriy Pavlov
>
>
>
> вт, 27 февр. 2018 г. в 0:45, Denis Magda :
>
>> Hi Dmitriy,
>>
>> Could you share some statistics? How many tests were fixed or merged in
>> one? Just curiuos.
>>
>> --
>> Denis
>>
>> On Mon, Feb 26, 2018 at 9:36 AM, Dmitry Pavlov 
>> wrote:
>>
>> > Hi Folks,
>> >
>> > Fixing the tests already brings some results. The number of falling
>> tests
>> > in a single run was reduced. Thanks to all, especially to Peter Ivanov,
>> > Alexey Goncharuk, Pavel Kovalenko
>> >
>> > There are 4 tickets that were not picked up by community (list at the
>> end
>> > of letter). Please look through and assign issue if you are able to help
>> > with this.
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>> >  - IGNITE-7791 -Ignite Client Nodes: failed test
>> > IgniteClientReconnectCacheTest.testReconnectCacheDestroyedAndCreated()
>> > - IGNITE-7789 - Ignite Client Nodes: failed test
>> > IgniteClientReconnectApiExceptionTest.testErrorOnDisconnect()
>> > - IGNITE-7708 - Ignite Compute has flaky failures in
>> ConfigVariationTests
>> > - IGNITE-7692 - affinityCall and affinityRun may execute code on backup
>> > partitions
>> >
>>
>

Fwd: checksum file Release Distribution Policy

[Denis Magda]

Igniters,

Do we comply with the next release requirements? Vladimir as a 2.4 release
manager, could you double check that we are in a good state?

--
Denis
-- Forwarded message --
From: Henk P. Penning 
Date: Mon, Mar 5, 2018 at 3:18 AM
Subject: checksum file Release Distribution Policy
To: he...@apache.org


Hi Pmcs,

   The Release Distribution Policy[1] changed regarding checksum files.
   See under "Cryptographic Signatures and Checksums Requirements" [2].

 MD5-file == a .md5 file
 SHA-file == a .sha1, sha256 or .sha512 file

  Old policy :

 -- MUST provide a MD5-file
 -- SHOULD provide a SHA-file [SHA-512 recommended]

  New policy :

 -- MUST provide a SHA- or MD5-file
 -- SHOULD provide a SHA-file
 -- SHOULD NOT provide a MD5-file

 Providing MD5 checksum files is now discouraged for new releases,
 but still allowed for past releases.

  Why this change :

 -- MD5 is broken for many purposes ; we should move away from it.
https://en.wikipedia.org/wiki/MD5#Overview_of_security_issues

  Impact for PMCs :

 -- for new releases :
-- please do provide a SHA-file (one or more, if you like)
-- do NOT provide a MD5-file

 -- for past releases :
-- you are not required to change anything
-- for artifacts accompanied by a SHA-file /and/ a MD5-file,
   it would be nice if you removed the MD5-file

 -- if, at the moment, you provide MD5-files,
please adjust your release tooling.

  Please mail me (he...@apache.org) if you have any questions etc.

  FYI :

   Many projects are not (entirely, strictly) checksum file compliant.
   For an overview/inventory (by project) see :

https://checker.apache.org/dist/unsummed.html

  At the moment :

 -- no checksum : 176 packages in 28 projects ; non-compliant
 -- only MD5: 495 packages in 44 projects ; update tooling
 -- only SHA: 135 packages in 13 projects ; now comliant

   In many cases, only a few (among many) checksum file are missing ;
   you may want to fix that.

   [1] http://www.apache.org/dev/release-distribution
   [2] http://www.apache.org/dev/release-distribution#sigs-and-sums

  Thanks, groeten,

  Henk Penning -- apache.org infrastructure ; dist & mirrors.

   _
Henk P. Penning, ICT-beta R Uithof MG-403_/ \_
Faculty of Science, Utrecht UniversityT +31 30 253 4106 / \_/ \
Leuvenlaan 4, 3584CE Utrecht, NL

F +31 30 253 4553 \_/ \_/
http://www.staff.science.uu.nl/~penni101/ M penn...@uu.nl \_/

Re: Ignite contibutors page

[Denis Magda]

Hi Dmitriy,

Thanks for kicking off the discussion. I'll wait for a couple of days for
other response and update the page in bulk.

--
Denis

On Mon, Mar 5, 2018 at 12:32 PM, Dmitry Pavlov 
wrote:

> Hi Igniters,
>
> I noticed that I'm not mentioned on the community contributors page:
> https://ignite.apache.org/community/resources.html
>
> Perhaps there is someone else, who is not yet added. Or information is not
> up-to-date.
>
> Please check the data for yourself and respond if you need to be added / or
> information needs to be updated.
>
> Sincerely,
> Dmitry Pavlov
>

Ignite contibutors page

[Dmitry Pavlov]

Hi Igniters,

I noticed that I'm not mentioned on the community contributors page:
https://ignite.apache.org/community/resources.html

Perhaps there is someone else, who is not yet added. Or information is not
up-to-date.

Please check the data for yourself and respond if you need to be added / or
information needs to be updated.

Sincerely,
Dmitry Pavlov

[GitHub] ignite pull request #3531: IGNITE-7685: Fixed allocation rate.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3531


---

[GitHub] ignite pull request #3525: Ignite 7517

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3525


---

Re: Why does DataStreamer swallow exceptions?

[Valentin Kulichenko]

Ilya,

IgniteDataStreamer#addData method returns future which should be completed
with error if one is thrown on server side. Does this happen or not?

-Val

On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov  wrote:

> Hello, Ilya.
>
> > I think it's time to end this, if that was the case. DataStreamer should
> > not be a special case and it should guarantee data safety. WDYT?
>
> +1 from me.
>
> I'm also facing this issue.
>
> Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
> Discussion - http://apache-ignite-developers.2346864.n4.nabble.
> com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
>
>
>
> В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> > Dear Igniters, why do I have a hunch that DataStreamer would readily
> > swallow exceptions?
> >
> > DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
> > deployment errors.
> >
> > Some people are worried they can fill a leaking vessel without noticing
> > anything off.
> > Also in line 2156 fsync() on WAL can throw exceptions, which will be
> > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> >
> > This question is not purely theoretical, we have also replaced one of
> these
> > eaters with throw with IGNITE-7519.
> >
> > There was a fault in PDS implementation, which did not propagate to
> client.
> > A serious issue IMHO.
> >
> > As a data streamer user I will expect that flush()/close() will throw any
> > pending exceptions and will only be silent if all data landed safely in
> the
> > cluster.
> >
> > I also have this feeling that DataStreamer was written using very
> internal
> > APIs so that it can compromise guarantees that cache and SQL APIs are
> bound
> > by. I think I've heard something about not recovering properly in case of
> > node failures.
> > I think it's time to end this, if that was the case. DataStreamer should
> > not be a special case and it should guarantee data safety. WDYT?
> >
>

[VOTE] Apache Ignite 2.4.0 RC1

[Vladimir Ozerov]

 Igniters,

We have uploaded a 2.4.0 release candidate to
https://dist.apache.org/repos/dist/dev/ignite/2.4.0-rc1/

Git tag name is
2.4.0-rc1

This release includes the following changes:

Ignite:
* Introduced Baseline Affinity Topology
* Ability to disable WAL for cache in runtime through IgniteCluster API or
ALTER TABLE command
* Added ability to convert WAL to human-readable form
* Ability to enable/disable JDBC, ODBC and thin client endpoints
* Project moved to Java 8
* Java 7 is no longer supported
* Apache Ignite can be run on Java 9
* Introduced Apache Ignite packaging with RPM
* Spark Data Frames support
* Added integration with Spring 3.x
* Added handler to stop node when IO error is encountered
* EvictionPolicyFactory support added to CacheConfiguration
* Added JMX metrics for memory regions
* Added detailed memory consumption on start and OOM reporting
* Added custom thread pools monitoring
* Support Ignite MR in Kerberized environment without IGFS
* S3 IP finder: support server side encryption and bucket endpoint
configuration
* Multiple fixes and performance optimizations

Ignite.NET:
* Thin Client initial release
* .NET Core / Linux / macOS support
* Mono platform support
* Removed C++ JNI layer

SQL:
* Added DEFAULT support to CREATE TABLE command
* Added ALTER TABLE DROP COLUMN support
* Added INLINE_SIZE option to CREATE INDEX command
* Added PARALLEL option to CREATE INDEX command
* Added optional on-heap row cache
* INSERT command now can be executed without explicit column names (INSERT
INTO table VALUES ...)
* Allowed multiple caches to share the same schema
* Added support for Java 8 Date and Time API
* Added "notNull" property to QuerySqlField annotation
* Improved COUNT(*) performance
* Fixed a problem causing an exception in certain nodes do not have primary
partitions for REPLICATED cache
* Fixed per-thread H2 connection leak.
* Fixed partition exchange hang due to SQL query running inside a
transaction.

JDBC Driver:
* Optimized batched operations processing

ODBC Driver:
* Added support of multiple statements execution with one call
* Added support of SQL_ATTR_CONNECTION_TIMEOUT
* Added support of SQL_ATTR_QUERY_TIMEOUT
* Optimized batched operations processing
* Proper handling of ungraceful TCP disconnects (keep-alive connection)
* Fixed error preventing close of executed DML statement
* Fixed table names returned by SQLTables for DDL-created tables

Machine Learning:
* Implemented Fuzzy c-means algorithm
* Implemented gradient descent for OLS Linear Regression
* Implemented K nearest neighbor algorithm
* Introduced API for datasets
* Introduced API for ML model and trainers
* Introduced common mechanism for group training of models
* Added distributed version of multilayer perceptron
* Added local version of multilayer perceptron
* Added basic import/export functionality for ml models
* Added decision tree algorithm
* Performance optimizations for distributed operations

Web Console:
* Implemented component for cluster selection and activation
* Implemented support for multiple statements on Queries screen
* Implemented progress indication for 'execute', 'scan' and 'export'
actions on Queries screen
* Added support for ClientConnectorConfiguration
* Added several SQL examples on Queries screen
* Added "Connected clusters" component in header
* Added support for custom SMTP server configuration
* Added detection of CSV separator based on browser locale
* Added "Copy to clipboard" action on Queries screen
* Added duration and node ID in results header and 'Show query' modal
* Improved support for binary JDBC types on import from RDBMS
* Fixed Web Console under IE11
* Fixed configuration generation for imported model in case of Oracle
NUMBER(5) data type
* Fixed output of big numbers in SQL query results

Visor:
* Added "-quiet" flag for batch mode
* Added ability to start cache rebalance
* Added output of data region metrics to "node" command
* Added dialog to put/get/remove values to/from cache for simple key types
* Show valid message for caches when cluster is inactive
* Fixed 'cache -a' in case of node filter
* Fixed reading last command line in batch mode
* Updated eviction policy factory in configs

Complete list of closed issues:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.4%20AND%20(status%20%3D%20closed%20or%20status%20%3D%20resolved)

DEVNOTES
https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=refs/tags/2.4.0-rc1

RELEASE NOTES
https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=refs/tags/2.4.0-rc1

Please start voting.

+1 - to accept Apache Ignite 2.4.0-RC1
0 - don't care either way
-1 - DO NOT accept Apache Ignite 2.4.0-RC1 (explain why)

This vote will go for 72 hours.

Vladimir.

Re: Use of lambdas and other Java 8 features

[Vyacheslav Daradur]

Dmitry, Vladimir, thank you for the clarification!

On Mon, Mar 5, 2018 at 6:23 PM, Dmitry Pavlov  wrote:
> By the way, usage of lambdas instead of anonymous class may have positive
> performance effect.
>
> And Java 8 provides us a number of new features, which allows Ignite
> operate faster (e.g. Arrays.parrallelSort, list.sort(), etc).
>
> пн, 5 мар. 2018 г. в 18:15, Vladimir Ozerov :
>
>> Hi Vyacheslav,
>>
>> I would not recommend using them without strong reasons. While convenient
>> for developers in general, some of these features has hidden pitfails,
>> which may affect performance and serialization. But as our product is very
>> performance sensitive, it is better to have slightly more code which is
>> simple and expressive, rather than save several lines of code and loose
>> understanding on what is going on.
>>
>> On Mon, Mar 5, 2018 at 6:07 PM, Vyacheslav Daradur 
>> wrote:
>>
>> > Hi, Igniters!
>> >
>> > Since we official support Java 8, I want to find out the following
>> > question:
>> >
>> > Are there any restrictions on using lambdas and other Java 8 features?
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>> >
>>



-- 
Best Regards, Vyacheslav D.

Data compression design proposal

[Vyacheslav Daradur]

Hi Igniters!

I’d like to do next step in our data compression discussion [1].

Most Igniters vote for per-data-page compression.

I’d like to accumulate  main theses to start implementation:
- page will be compressed with the dictionary-based approach (e.g.LZV)
- page will be compressed in batch mode (not on every change)
- page compression should been initiated by an event, for example, a
page’s free space drops below 20%
- compression process will be under page write lock

Vladimir Ozerov has written:
>> What we do not understand yet:
>> 1) Granularity of compression algorithm.
>> 1.1) It could be per-entry - i.e. we compress the whole entry content, but
>> respect boundaries between entries. E.g.: before - [ENTRY_1][ENTRY_2],
>> after - [COMPRESSED_ENTRY_1][COMPRESSED_ENTRY_2] (as opposed to [COMPRESSED 
>> ENTRY_1 and ENTRY_2]).
>> v1.2) Or it could be per-field - i.e. we compress fields, but respect binary
>> object layout. First approach is simple, straightforward, and will give
>> acceptable compression rate, but we will have to compress the whole binary
>> object on every field access, what may ruin our SQL performance. Second
>> approach is more complex, we are not sure about it's compression rate, but
>> as BinaryObject structure is preserved, we will still have fast
>> constant-time per-field access.

I think there are advantages in both approaches and we will be able to
compare different approaches and algorithms after prototype
implementation.

Main approach in brief:
1) When page’s free space drops below 20% will be triggered compression event
2) Page will be locked by write lock
3) Page will be passed to page’s compressor implementation
4) Page will be replaced by compressed page

Whole object or a field reading:
1) If page marked as compressed then the page will be handled by
page’s compressor implementation, otherwise, it will be handled as
usual.

Thoughts?

Should we create new IEP and register tickets to start implementation?
This will allow us to watch for the feature progress and related
tasks.


[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-tc20679.html


-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #3605: ignite-6643-2.3 Marshalling improvements

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/3605

ignite-6643-2.3 Marshalling improvements



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-6643-2.3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3605.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3605


commit b67feb0f175bfbd6ffbefe82a8d693c8ab7d4213
Author: Vasiliy Sisko 
Date:   2017-10-09T10:55:23Z

IGNITE-5767 Web console: Use byte array type instead of java.lang.Object 
for binary JDBC types.
(cherry picked from commit 3184437)

commit 8e1560322b87d79b3d3250832a3969ac4032d6fc
Author: Alexey Kuznetsov 
Date:   2017-10-06T18:10:08Z

IGNITE-6574 Remove pending requests in case STATUS_AUTH_FAILURE && 
credentials == null.
(cherry picked from commit 85261a3)

commit 22100d4f2d11677fd583246cbec7169947c0873a
Author: Oleg Ostanin 
Date:   2017-10-18T11:14:41Z

IGNITE-5608: Added sqlline utility to the build. This closes #2861.

commit 7a0300ae35894c389b126e95615f720a99a3d360
Author: devozerov 
Date:   2017-10-18T11:18:08Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit 5df25fc8adf01a8e4999563f3f31b79c195801d4
Author: devozerov 
Date:   2017-10-18T12:03:28Z

IGNITE-6662: SQL: fixed affinity key field name resolution during both 
parsig and table creation. This closes #2875.

commit ad01f9b099d0bf92537378859ad6d5a52de57748
Author: Alexey Kuznetsov 
Date:   2017-10-19T02:43:20Z

IGNITE-6647 Web Console: Implemented support of schema migration scripts.
(cherry picked from commit c65399c)

commit 0c66344bc752dac98b256dd140fcab95d1662862
Author: Pavel Tupitsyn 
Date:   2017-10-19T09:36:39Z

IGNITE-6627 .NET: Fix repeated known metadata updates

This closes #2876

commit 9c4411af5f6b6bf7686e52d91daa6b82e089d57a
Author: Pavel Tupitsyn 
Date:   2017-10-19T15:42:12Z

IGNITE-6675 .NET: Fix ignored IgniteConfiguration.IgniteHome

This closes #2886

commit 5dfeb916036984d3d8e12ad6d2d43e17a19f25ba
Author: tledkov-gridgain 
Date:   2017-10-19T19:20:18Z

IGNITE-6529: JDBC: fixed not-null column metadata. This closes #2884.

commit 008d87057734953b4e30059841a14eb2fbc3ddb7
Author: devozerov 
Date:   2017-10-19T19:30:31Z

IGNITE-6684: Renamed "ignitesql.sh|bat" to "sqlline.sh|bat".

commit 1b8abd214ed2afcd3fd1f6a4c71a19d6fe1a4b01
Author: Alexey Kuznetsov 
Date:   2017-10-20T04:23:23Z

IGNITE-6647 Added missing Mongo injector.
(cherry picked from commit 173ecef)

commit 745677631d260cb51cb601ae38af8528aa5d5c66
Author: Ivan Rakov 
Date:   2017-10-20T07:29:57Z

IGNITE-6030 Allow enabling persistence per data region

commit 6c58b4ac7c4527d583de49c4d8b250436273294c
Author: Alexey Goncharuk 
Date:   2017-10-20T10:43:44Z

IGNITE-6030 Fixed misspelled metric

commit 8ee033fdc50b11c6913e1b6ddc100c28f6bf4341
Author: Pavel Tupitsyn 
Date:   2017-10-20T11:38:11Z

IGNITE-6515 .NET: Enable persistence on per-cache basis

This closes #2891

commit 347696d2426ef5be8294253141e299097a6564cc
Author: Anton Vinogradov 
Date:   2017-10-20T14:15:39Z

Removed redundant libs from libs/optional

commit a221066b3d029afc392be704a810c0e830fc0c49
Author: Alexey Kuznetsov 
Date:   2017-10-20T14:15:02Z

IGNITE-6647 Web Console: Added folder for modules migrations.
(cherry picked from commit 3700717)

commit e6cb5300d51c5184e876b988c4683bc605685874
Author: devozerov 
Date:   2017-10-20T15:11:52Z

AI release notes.

commit d196045bf8b719f65b4025409112140196aa206c
Author: devozerov 
Date:   2017-10-21T15:47:04Z

IGNITE-6689: SQL: Added DATA_REGION option for CREATE TABLE.

commit da8a9d5a968ba071697a28adb01bc59f80d1893c
Author: Pavel Tupitsyn 
Date:   2017-10-23T08:55:33Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 69fdac3acf768ecb9df80d4412c4de5ffd5bc4f5
Author: Dmitriy Shabalin 
Date:   2017-10-23T09:09:47Z

IGNITE-5909 Added list editable component.
(cherry picked from commit 01daee6)

commit ec1a8e7f698e584e94284220aa13ff15449f366e
Author: oleg-ostanin 
Date:   2017-10-24T06:48:04Z

IGNITE-6706: Removed ignite-sqlline module from "optional" build directory. 
This closes #2901.

commit 3e52aca47b0a6a0a47f7d063bd0d2bb51489e523
Author: oleg-ostanin 
Date:   2017-10-24T06:49:55Z

IGNITE-6708: Removed ignite-compatibility module 

Re: IGNITE-5357 is ready for review (Replicated cache reads load balancing)

[Vyacheslav Daradur]

Dmitry, I saw them, but it looks like just randomness.

I've checked it locally several times.
They failed only in one TeamCity's build of four.

Started build once again to be sure.

On Mon, Mar 5, 2018 at 6:59 PM, Dmitry Pavlov  wrote:
> I can see Nikolay Izhikov as reviewer in Upsource.
>
> Nikolay, would you run review first?
>
> I've found several suspicious tests : Test fail rate is less than 1%, it is
> probably new failure
> IgniteCacheTestSuite2:
> GridCachePartitionedTxSingleThreadedSelfTest.testOptimisticReadCommittedRollback
> (fail rate 0,0%)
> IgniteCacheTestSuite2:
> GridCachePartitionedTxSingleThreadedSelfTest.testOptimisticRepeatableReadRollback
> (fail rate 0,0%)
> IgniteCacheTestSuite2:
> GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticReadCommittedCommit
> (fail rate 0,0%)
> IgniteCacheTestSuite2:
> GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticReadCommittedRollback
> (fail rate 0,0%)
> IgniteCacheTestSuite2:
> GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticSerializableCommit
> (fail rate 0,0%)
>
> Vyacheslav, could you please check if these failures are related to the new
> changes?
>
>
> пн, 5 мар. 2018 г. в 18:50, Vyacheslav Daradur :
>
>> I've done some test-builds iteration on the weekends.
>>
>> Tests [1] look well.
>>
>> Does anyone have time to do the final review [2][3] and merge it?
>>
>>
>> [1] https://ci.ignite.apache.org/viewLog.html?buildId=1125676
>> [2] https://github.com/apache/ignite/pull/3578
>> [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509
>>
>>
>> On Fri, Mar 2, 2018 at 10:17 PM, Vyacheslav Daradur 
>> wrote:
>> > Hi, Igniters!
>> >
>> > This task [1] is about 'get' requests distribution between primary and
>> > backup nodes in the replicated cache if 'readFromBackup' flag is
>> > enabled.
>> >
>> > I've prepared a solution [2] suggested by Alexei Scherbakov in Jira
>> > comments. It passed prereviews by Alexei and Nikolay Izhikov.
>> >
>> > TeamCity tests look similar with the master branch.
>> >
>> > Could someone of core module maintainers do the final review [2][3]?
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-5357
>> > [2] https://github.com/apache/ignite/pull/3578
>> > [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>



-- 
Best Regards, Vyacheslav D.

Re: .NET Long Running Test execution timeout

[Dmitry Pavlov]

Hi Pavel,

Thank you!

Let's wait for fix at TeamCity.

Sincerely,
Dmitriy Pavlov

пн, 5 мар. 2018 г. в 19:09, Pavel Tupitsyn :

> Hi Dmitry,
>
> This test has been there forever. It checks that Ignite can start with
> default config.
> I think this is quite useful. TeamCity should be fixed to isolate
> environments properly.
>
> For now I've disabled the test with comment in master branch.
>
> Thanks,
> Pavel
>
>
>
> On Mon, Mar 5, 2018 at 5:53 PM, Igor Sapego  wrote:
>
> > In C++ there are no tests that use TcpDiscoveryMulticastIpFinder. There
> > are examples, though, that use it. As far as I know, currently all
> examples
> > use TcpDiscoveryMulticastIpFinder, and there were some objections
> > against using VmIpFinder instead.
> >
> > Am I right, folks?
> >
> > Best Regards,
> > Igor
> >
> > On Mon, Mar 5, 2018 at 4:05 PM, Dmitry Pavlov 
> > wrote:
> >
> >> Hi Pavel, Igniters,
> >>
> >> There is new TC failure from .NET tests. Issue link:
> >> https://issues.apache.org/jira/browse/IGNITE-7878
> >>
> >> Thanks to V.Ozerov we have located it is because usage
> >> of TcpDiscoveryMulticastIpFinder. Probably there is some other Ignite
> node
> >> running on other agents. Folks from infrastructure probably will fix
> >> isolation of agents in future.
> >>
> >> But we need fast fix, it could be usage of VmIpFinder. Could you please
> >> change configuration in tests? I personally prefer this change instead
> of
> >> removal suite from RunAll. Currently these timeouts creates bottleneck
> in
> >> win. agents and all Run-Alls waits .NET & CPP tests.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >
> >
>

Re: IGNITE-5357 is ready for review (Replicated cache reads load balancing)

[Dmitry Pavlov]

I can see Nikolay Izhikov as reviewer in Upsource.

Nikolay, would you run review first?

I've found several suspicious tests : Test fail rate is less than 1%, it is
probably new failure
IgniteCacheTestSuite2:
GridCachePartitionedTxSingleThreadedSelfTest.testOptimisticReadCommittedRollback
(fail rate 0,0%)
IgniteCacheTestSuite2:
GridCachePartitionedTxSingleThreadedSelfTest.testOptimisticRepeatableReadRollback
(fail rate 0,0%)
IgniteCacheTestSuite2:
GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticReadCommittedCommit
(fail rate 0,0%)
IgniteCacheTestSuite2:
GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticReadCommittedRollback
(fail rate 0,0%)
IgniteCacheTestSuite2:
GridCachePartitionedTxSingleThreadedSelfTest.testPessimisticSerializableCommit
(fail rate 0,0%)

Vyacheslav, could you please check if these failures are related to the new
changes?


пн, 5 мар. 2018 г. в 18:50, Vyacheslav Daradur :

> I've done some test-builds iteration on the weekends.
>
> Tests [1] look well.
>
> Does anyone have time to do the final review [2][3] and merge it?
>
>
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=1125676
> [2] https://github.com/apache/ignite/pull/3578
> [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509
>
>
> On Fri, Mar 2, 2018 at 10:17 PM, Vyacheslav Daradur 
> wrote:
> > Hi, Igniters!
> >
> > This task [1] is about 'get' requests distribution between primary and
> > backup nodes in the replicated cache if 'readFromBackup' flag is
> > enabled.
> >
> > I've prepared a solution [2] suggested by Alexei Scherbakov in Jira
> > comments. It passed prereviews by Alexei and Nikolay Izhikov.
> >
> > TeamCity tests look similar with the master branch.
> >
> > Could someone of core module maintainers do the final review [2][3]?
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5357
> > [2] https://github.com/apache/ignite/pull/3578
> > [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509
> >
> > --
> > Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Ignite nightly build job on Apache Jenkins server

[Pavel Tupitsyn]

Hi Igor,

This build did not make much sense anyway:
1) (At least) platform artifacts (.NET) can't be built there, so the
package is incomplete
2) We have TeamCity, why duplicate maintenance efforts?

Nightly release builds are very useful, but they should be done on TeamCity.
(We have them for .NET NuGet:
https://cwiki.apache.org/confluence/display/IGNITE/Nightly+Builds)

Thanks,
Pavel


On Mon, Mar 5, 2018 at 6:27 PM, Igor Sapego  wrote:

> Hi Igniters,
>
> There is a job on Apache Jenkins for nightly builds of Ignite [1],
> which was useful for providing fast fixes to our users. However,
> it seems that noone supports it any more as the last successfull
> build date is 31.05.2017.
>
> Does anyone know what is the issue with the job and why it
> can not be fixed? Or if it can be fixed, maybe someone would
> like to do that? I can do it myself, if I can get administrative
> access to the Jenkins server. According to [2], any PMC chair
> may grant such access to any committer.
>
> So, Denis, would you grant an access to me, if there are no
> objection and no other committers, that would like to resolve the
> issue?
>
> [1] - https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/
> [2] - https://wiki.apache.org/general/Jenkins
>
> Best Regards,
> Igor
>

Re: IGNITE-5357 is ready for review (Replicated cache reads load balancing)

[Vyacheslav Daradur]

I've done some test-builds iteration on the weekends.

Tests [1] look well.

Does anyone have time to do the final review [2][3] and merge it?


[1] https://ci.ignite.apache.org/viewLog.html?buildId=1125676
[2] https://github.com/apache/ignite/pull/3578
[3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509


On Fri, Mar 2, 2018 at 10:17 PM, Vyacheslav Daradur  wrote:
> Hi, Igniters!
>
> This task [1] is about 'get' requests distribution between primary and
> backup nodes in the replicated cache if 'readFromBackup' flag is
> enabled.
>
> I've prepared a solution [2] suggested by Alexei Scherbakov in Jira
> comments. It passed prereviews by Alexei and Nikolay Izhikov.
>
> TeamCity tests look similar with the master branch.
>
> Could someone of core module maintainers do the final review [2][3]?
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5357
> [2] https://github.com/apache/ignite/pull/3578
> [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509
>
> --
> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.

[jira] [Created] (IGNITE-7883) Cluster can have inconsistent affinity configuration

[Mikhail Cherkasov (JIRA)]

Mikhail Cherkasov created IGNITE-7883:
-

 Summary: Cluster can have inconsistent affinity configuration 
 Key: IGNITE-7883
 URL: https://issues.apache.org/jira/browse/IGNITE-7883
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.3
Reporter: Mikhail Cherkasov
 Fix For: 2.5


A cluster can have inconsistent affinity configuration if you created two 
nodes, one with affinity key configuration and other without it(in IgniteCfg or 
CacheCfg),  both nodes will work fine with no exceptions, but in the same time 
they will apply different affinity rules to keys:

 
{code:java}
public class Test {
private static int id = 0;

public static void main(String[] args) {
Ignite ignite = Ignition.start(getConfiguration(true, false));
Ignite ignite2 = Ignition.start(getConfiguration(false, false));

Affinity affinity = ignite.affinity("TEST");
Affinity affinity2 = ignite2.affinity("TEST");
for (int i = 0; i < 1_000_000; i++) {
AKey key = new AKey(i);

if(affinity.partition(key) != affinity2.partition(key))
System.out.println("FAILED for: " + key);
}

}

@NotNull private static IgniteConfiguration getConfiguration(boolean 
withAffinityCfg, boolean client) {
IgniteConfiguration cfg = new IgniteConfiguration();

TcpDiscoveryVmIpFinder finder = new TcpDiscoveryVmIpFinder(true);
finder.setAddresses(Arrays.asList("localhost:47500..47600"));

cfg.setClientMode(client);

cfg.setIgniteInstanceName("test" + id++);

if(withAffinityCfg) {
CacheConfiguration cacheCfg = new CacheConfiguration("TEST");

cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);

cacheCfg.setKeyConfiguration(new 
CacheKeyConfiguration("multiplan.AKey", "a"));

cfg.setCacheConfiguration(cacheCfg);
}

cfg.setDiscoverySpi(new TcpDiscoverySpi().setIpFinder(finder));

return cfg;
}
}

class AKey {

int a;

public AKey(int a) {
this.a = a;
}

@Override public String toString() {
return "AKey{" +
"b=" + a +
'}';
}
}


{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Ignite nightly build job on Apache Jenkins server

[Igor Sapego]

Hi Igniters,

There is a job on Apache Jenkins for nightly builds of Ignite [1],
which was useful for providing fast fixes to our users. However,
it seems that noone supports it any more as the last successfull
build date is 31.05.2017.

Does anyone know what is the issue with the job and why it
can not be fixed? Or if it can be fixed, maybe someone would
like to do that? I can do it myself, if I can get administrative
access to the Jenkins server. According to [2], any PMC chair
may grant such access to any committer.

So, Denis, would you grant an access to me, if there are no
objection and no other committers, that would like to resolve the
issue?

[1] - https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/
[2] - https://wiki.apache.org/general/Jenkins

Best Regards,
Igor

Re: Use of lambdas and other Java 8 features

[Dmitry Pavlov]

By the way, usage of lambdas instead of anonymous class may have positive
performance effect.

And Java 8 provides us a number of new features, which allows Ignite
operate faster (e.g. Arrays.parrallelSort, list.sort(), etc).

пн, 5 мар. 2018 г. в 18:15, Vladimir Ozerov :

> Hi Vyacheslav,
>
> I would not recommend using them without strong reasons. While convenient
> for developers in general, some of these features has hidden pitfails,
> which may affect performance and serialization. But as our product is very
> performance sensitive, it is better to have slightly more code which is
> simple and expressive, rather than save several lines of code and loose
> understanding on what is going on.
>
> On Mon, Mar 5, 2018 at 6:07 PM, Vyacheslav Daradur 
> wrote:
>
> > Hi, Igniters!
> >
> > Since we official support Java 8, I want to find out the following
> > question:
> >
> > Are there any restrictions on using lambdas and other Java 8 features?
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

Re: Use of lambdas and other Java 8 features

[Vladimir Ozerov]

Hi Vyacheslav,

I would not recommend using them without strong reasons. While convenient
for developers in general, some of these features has hidden pitfails,
which may affect performance and serialization. But as our product is very
performance sensitive, it is better to have slightly more code which is
simple and expressive, rather than save several lines of code and loose
understanding on what is going on.

On Mon, Mar 5, 2018 at 6:07 PM, Vyacheslav Daradur 
wrote:

> Hi, Igniters!
>
> Since we official support Java 8, I want to find out the following
> question:
>
> Are there any restrictions on using lambdas and other Java 8 features?
>
> --
> Best Regards, Vyacheslav D.
>

Re: Use of lambdas and other Java 8 features

[Dmitry Pavlov]

Hi Vyacheslav,

It is possible to use lambdas in test code and node local-only code (e.g.
checkpointing, page memory, etc).

In other words any other classes, which are transferred/may be trasnferred
into network, can't use lambdas.

There is also recommendation to avoid mass change of anonymous classes to
lambdas because of difficulties of backporting.

Sincerely,
Dmitriy Pavlov

пн, 5 мар. 2018 г. в 18:07, Vyacheslav Daradur :

> Hi, Igniters!
>
> Since we official support Java 8, I want to find out the following
> question:
>
> Are there any restrictions on using lambdas and other Java 8 features?
>
> --
> Best Regards, Vyacheslav D.
>

[GitHub] ignite pull request #3604: ignite-6643-2.1Marshalling improvements (backport...

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/3604

ignite-6643-2.1Marshalling improvements (backport)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-6643-2.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3604.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3604


commit 8f625eb44c0a6f6282d651e572ebc69c39229cc7
Author: Alexey Kuznetsov 
Date:   2017-10-01T11:47:08Z

IGNITE-6535 Web Console: Set IGNITE_HOME for Web Agent.
(cherry picked from commit 4cb0fc7)

commit 99eb950da7b1056c99e6904be26b5b55fa74ddd9
Author: Dmitriy Shabalin 
Date:   2017-10-01T11:57:18Z

IGNITE-5863 Fixed select ALL logic.
(cherry picked from commit 2c39351)

commit 44f228ed22172f826f61563cf1fefa7e98288f9c
Author: vsisko 
Date:   2017-10-01T12:00:22Z

IGNITE-6120 Fixed missing "lazy query" flag.
(cherry picked from commit 55886b1)

commit c0de2dc16b5314581ac13ac531fb364cd97af0c1
Author: dpavlov 
Date:   2017-10-04T12:05:48Z

IGNITE-6554 Atomic cache remove operations are not logged to WAL - Fixes 
#2800.

Signed-off-by: Alexey Goncharuk 

commit f3c95c71a81856d5a98cca47df75dba1c54fac54
Author: Pavel Tupitsyn 
Date:   2017-09-27T11:57:47Z

IGNITE-5730 .NET: Fix ignite.jni.dll temp dir race

This closes #2755

commit 371782d2262c6716c44d123014677945d9d47845
Author: Alexey Goncharuk 
Date:   2017-10-05T14:37:04Z

IGNITE-5739 Fixed Ignite node crash on deactivation

commit 5ddf97ff17a94cd075d45152829bae92b9dbb665
Author: dpavlov 
Date:   2017-10-05T15:06:27Z

IGNITE-6539 WAL parser fails if empty log files exist in directory - Fixes 
#2794.

Signed-off-by: Alexey Goncharuk 

commit 58d5fb045fc0c7f544ae325306119a0cb45419c3
Author: dpavlov 
Date:   2017-10-05T15:10:25Z

IGNITE-6553 Standalone WAL iterator fails to handle WAL delete data records 
- Fixes #2797.

commit 2419d1760368ad0493220056099e5a65bf502c07
Author: Vasiliy Sisko 
Date:   2017-10-06T07:25:42Z

IGNITE-6287 Web Console: Improved DDL support.
(cherry picked from commit 2410f07)

commit ee02d6da2ed3e079c7480dc577ddef0748e58fbf
Author: Alexey Kuznetsov 
Date:   2017-10-06T10:00:39Z

IGNITE-6570 Web Console: Move parsing of JSON to pool of workers.
(cherry picked from commit 74f0400)

commit e5b2e3a9fa4aca6f74b58bad5b1e58afda39f257
Author: Alexey Kuznetsov 
Date:   2017-10-06T17:11:37Z

IGNITE-6463 Web Console: Fixed output of big numbers in SQL query results.
(cherry picked from commit 35589a7)

commit c32f0af1c2dfb14dc4d52a282c1d2e50bddcd066
Author: Alexey Kuznetsov 
Date:   2017-10-06T18:10:08Z

IGNITE-6574 Remove pending requests in case STATUS_AUTH_FAILURE && 
credentials == null.
(cherry picked from commit 85261a3)

commit f62884b63663bebd9630ef01df59550033b85f1c
Author: Vasiliy Sisko 
Date:   2017-10-09T10:55:23Z

IGNITE-5767 Web console: Use byte array type instead of java.lang.Object 
for binary JDBC types.
(cherry picked from commit 3184437)

commit aa9093a26ddaf91e7f068663c52e090480cdfe6d
Author: Vasiliy Sisko 
Date:   2017-10-09T12:23:23Z

IGNITE-6287 Web Console: Improved DDL support: added checkbox "Use selected 
cache as default schema name".
(cherry picked from commit a45677c)

commit 912ae4b0fa3971499c1e8f9c4272c9b56b0355d2
Author: Sergey Chugunov 
Date:   2017-10-09T15:35:11Z

IGNITE-6583 Proper getters for rebalance metrics were added; ignite-style 
getters (without get) were deprecated

Signed-off-by: Andrey Gura 

commit aceed9498550833f5a0dcf7fcc003ea2f83378fa
Author: AMRepo 
Date:   2017-10-10T08:57:20Z

IGNITE-6545: Failure during Ignite Service.cancel() can break normal 
shutdown process.

commit f006500391c9712d68d5b90f3da72a421fbda48a
Author: vsisko 
Date:   2017-10-02T16:08:40Z

IGNITE-6422 Visor CMD: Fixed cache statistics output.
(cherry picked from commit 16d2370)

commit 252cb5d2a1962731b39505d6c0d711701a525724
Author: Krzysztof Chmielewski 
Date:   2017-10-10T14:50:59Z

Fixed "IGNITE-6234 Initialize schemaIds to empty set if schemas field is 
null during the deserialization".

Signed-off-by: nikolay_tikhonov 

commit 8eaacd10953f31e75433847747ea7fcf4f129d3b
Author: Alexey Kuznetsov 
Date:   2017-10-12T15:48:35Z

IGNITE-6127 Fixed bytes encoding.
(cherry picked from commit 0f3f7d2)

commit 

Use of lambdas and other Java 8 features

[Vyacheslav Daradur]

Hi, Igniters!

Since we official support Java 8, I want to find out the following question:

Are there any restrictions on using lambdas and other Java 8 features?

-- 
Best Regards, Vyacheslav D.

Re: .NET Long Running Test execution timeout

[Igor Sapego]

In C++ there are no tests that use TcpDiscoveryMulticastIpFinder. There
are examples, though, that use it. As far as I know, currently all examples
use TcpDiscoveryMulticastIpFinder, and there were some objections
against using VmIpFinder instead.

Am I right, folks?

Best Regards,
Igor

On Mon, Mar 5, 2018 at 4:05 PM, Dmitry Pavlov  wrote:

> Hi Pavel, Igniters,
>
> There is new TC failure from .NET tests. Issue link:
> https://issues.apache.org/jira/browse/IGNITE-7878
>
> Thanks to V.Ozerov we have located it is because usage
> of TcpDiscoveryMulticastIpFinder. Probably there is some other Ignite node
> running on other agents. Folks from infrastructure probably will fix
> isolation of agents in future.
>
> But we need fast fix, it could be usage of VmIpFinder. Could you please
> change configuration in tests? I personally prefer this change instead of
> removal suite from RunAll. Currently these timeouts creates bottleneck in
> win. agents and all Run-Alls waits .NET & CPP tests.
>
> Sincerely,
> Dmitriy Pavlov
>

[GitHub] ignite pull request #3603: IGNITE-7882 Always use topology mapping for atomi...

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/3603

IGNITE-7882 Always use topology mapping for atomic updates.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7882

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3603


commit 1418d83df435f5b5418b97e990365be2504d16d1
Author: Pavel Kovalenko 
Date:   2018-03-05T13:50:00Z

IGNITE-7882 Always use topology mapping for atomic updates.




---

[jira] [Created] (IGNITE-7882) Atomic update requests should always use topology mappings instead of affinity

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-7882:
---

 Summary: Atomic update requests should always use topology 
mappings instead of affinity
 Key: IGNITE-7882
 URL: https://issues.apache.org/jira/browse/IGNITE-7882
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.4
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko


Currently for mapping cache atomic updates we can use two ways:
1) Use nodes reporting status OWNING for partition where we send the update.
2) Use only affinity nodes mapping if rebalance is finished.

Using the second way we may route update request only to affinity node, while 
there is also node which is still owner and can process read requests.

It can lead to reading null values for some key, while update for such key was 
successful a moment ago.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

.NET Long Running Test execution timeout

[Dmitry Pavlov]

Hi Pavel, Igniters,

There is new TC failure from .NET tests. Issue link:
https://issues.apache.org/jira/browse/IGNITE-7878

Thanks to V.Ozerov we have located it is because usage
of TcpDiscoveryMulticastIpFinder. Probably there is some other Ignite node
running on other agents. Folks from infrastructure probably will fix
isolation of agents in future.

But we need fast fix, it could be usage of VmIpFinder. Could you please
change configuration in tests? I personally prefer this change instead of
removal suite from RunAll. Currently these timeouts creates bottleneck in
win. agents and all Run-Alls waits .NET & CPP tests.

Sincerely,
Dmitriy Pavlov

Re: Why does DataStreamer swallow exceptions?

[Nikolay Izhikov]

Hello, Ilya.

> I think it's time to end this, if that was the case. DataStreamer should
> not be a special case and it should guarantee data safety. WDYT?

+1 from me.

I'm also facing this issue.

Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
Discussion - 
http://apache-ignite-developers.2346864.n4.nabble.com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html



В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> Dear Igniters, why do I have a hunch that DataStreamer would readily
> swallow exceptions?
> 
> DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
> deployment errors.
> 
> Some people are worried they can fill a leaking vessel without noticing
> anything off.
> Also in line 2156 fsync() on WAL can throw exceptions, which will be
> swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> 
> This question is not purely theoretical, we have also replaced one of these
> eaters with throw with IGNITE-7519.
> 
> There was a fault in PDS implementation, which did not propagate to client.
> A serious issue IMHO.
> 
> As a data streamer user I will expect that flush()/close() will throw any
> pending exceptions and will only be silent if all data landed safely in the
> cluster.
> 
> I also have this feeling that DataStreamer was written using very internal
> APIs so that it can compromise guarantees that cache and SQL APIs are bound
> by. I think I've heard something about not recovering properly in case of
> node failures.
> I think it's time to end this, if that was the case. DataStreamer should
> not be a special case and it should guarantee data safety. WDYT?
> 

signature.asc
Description: This is a digitally signed message part

[GitHub] ignite pull request #3602: IGNITE-7881 Tests for using TreeMap or TreeSet as...

[alamar]

GitHub user alamar opened a pull request:

https://github.com/apache/ignite/pull/3602

IGNITE-7881 Tests for using TreeMap or TreeSet as cache key.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7881

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3602.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3602


commit 567d2007b69884f0448115fd88733e8f9aba0534
Author: Ilya Kasnacheev 
Date:   2018-03-05T12:06:51Z

IGNITE-7881 Tests for using TreeMap or TreeSet as cache key.




---

[jira] [Created] (IGNITE-7881) Add tests for using TreeMap/TreeSet in cache key

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-7881:
---

 Summary: Add tests for using TreeMap/TreeSet in cache key
 Key: IGNITE-7881
 URL: https://issues.apache.org/jira/browse/IGNITE-7881
 Project: Ignite
  Issue Type: Task
  Components: binary, cache
Affects Versions: 2.3
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev
 Fix For: 2.5


It came to our attention that in Ignite 1.x, [Binary]TreeMap/TreeSet could not 
be used in cache keys without configuration or code changes, since equality 
won't work correctly for them.

It was since fixed by IGNITE-4558, but test cases won't hurt to nail this 
behaviour fast.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Why does DataStreamer swallow exceptions?

[Ilya Kasnacheev]

Dear Igniters, why do I have a hunch that DataStreamer would readily
swallow exceptions?

DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
deployment errors.

Some people are worried they can fill a leaking vessel without noticing
anything off.
Also in line 2156 fsync() on WAL can throw exceptions, which will be
swallowed, and IMP this fsync doesn't belong to DataStreamer code.

This question is not purely theoretical, we have also replaced one of these
eaters with throw with IGNITE-7519.

There was a fault in PDS implementation, which did not propagate to client.
A serious issue IMHO.

As a data streamer user I will expect that flush()/close() will throw any
pending exceptions and will only be silent if all data landed safely in the
cluster.

I also have this feeling that DataStreamer was written using very internal
APIs so that it can compromise guarantees that cache and SQL APIs are bound
by. I think I've heard something about not recovering properly in case of
node failures.
I think it's time to end this, if that was the case. DataStreamer should
not be a special case and it should guarantee data safety. WDYT?

-- 
Ilya Kasnacheev

[jira] [Created] (IGNITE-7880) Enum values not shown correctly in Webconsole

[Mikhail Cherkasov (JIRA)]

Mikhail Cherkasov created IGNITE-7880:
-

 Summary: Enum values not shown correctly in Webconsole
 Key: IGNITE-7880
 URL: https://issues.apache.org/jira/browse/IGNITE-7880
 Project: Ignite
  Issue Type: Bug
  Components: visor
Affects Versions: 2.3
Reporter: Mikhail Cherkasov
Assignee: Mikhail Cherkasov
 Fix For: 2.5
 Attachments: image-2018-03-05-13-34-48-992.png

Enum values not shown correctly in Webconsole:

!image-2018-03-05-13-34-48-992.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3601: gg-13377 : Fixed error handling for corrupted pag...

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/3601

gg-13377 : Fixed error handling for corrupted page store.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-13377

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3601.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3601


commit dbc77734f94a49cbca91c1981896aaa47739cc32
Author: Ilya Lantukh 
Date:   2018-03-05T10:27:25Z

gg-13377 : Fixed error handling for corrupted page store.




---

[GitHub] ignite pull request #3600: IGNITE-7860 JDBC thin driver: set default socket ...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/3600

IGNITE-7860 JDBC thin driver: set default socket buffer sizes to 64Kb



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7860

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3600.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3600


commit bd9d6dccc6c176376b2ea629f952f2ff7937f2cd
Author: tledkov-gridgain 
Date:   2018-03-05T09:24:17Z

IGNITE-7860 JDBC thin driver: set default socket buffer sizes to 64Kb




---