[jira] [Commented] (IGNITE-3999) Support case insensitive search in SQL

[Amir Akhmedov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348079#comment-16348079
 ] 

Amir Akhmedov commented on IGNITE-3999:
---

[~al.psc], javadoc and style has been fixed and committed.

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-6842) Stop all nodes after test by default.

[Maxim Muzafarov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maxim Muzafarov reassigned IGNITE-6842:
---

Assignee: Maxim Muzafarov

> Stop all nodes after test by default.
> -
>
> Key: IGNITE-6842
> URL: https://issues.apache.org/jira/browse/IGNITE-6842
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexei Scherbakov
>Assignee: Maxim Muzafarov
>Priority: Major
>  Labels: newbie
> Fix For: 2.5
>
>
> Currently it's required to manually call stopAllGrids() after test completion.
> This leads to errors in subsequent tests if someone forgets to call it and to 
> additional boilerplate code in tests.
> Right choice is to make this default behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7597) visorcmd: remove cache variable (@c) after cache was stopped

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-7597:
--

 Summary: visorcmd: remove cache variable (@c) after cache was 
stopped
 Key: IGNITE-7597
 URL: https://issues.apache.org/jira/browse/IGNITE-7597
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3, 2.4
Reporter: Pavel Konstantinov
 Fix For: 2.5


# start node with several caches
# start visorcmd and connect to grid
# execute cache command to set cache variables (@c0..@cX)
# stop cache @c0
# execute mlist - cache @c0 is still in the list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7337) Spark Data Frames: support saving a data frame in Ignite

[Nikolay Izhikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347989#comment-16347989
 ] 

Nikolay Izhikov commented on IGNITE-7337:
-

{{OPTION_WRITE_PARTITIONS_NUM}} - Is number of partition we want to use for 
DataFrame writing.

Each partition will be written concurrently by separate Spark worker.
If {{OPTION_WRITE_PARTITIONS_NUM}} is greater then DataFrame already has - 
DataFrame will remaing the same.
Actulally, I take this option from Spark internal JDBC implementation.

* Usage in my PR - 
https://github.com/apache/ignite/pull/3438/files#diff-69b49192927a69d21b4afec604f60b77R105
* coalesce doc - 
https://spark.apache.org/docs/2.2.0/api/scala/index.html#org.apache.spark.sql.Dataset@coalesce(numPartitions:Int):org.apache.spark.sql.Dataset[T]
* usage inside Spark JDBC implementation - 
https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L775

Do you think such option will be usefull for Ignite?

> Spark Data Frames: support saving a data frame in Ignite
> 
>
> Key: IGNITE-7337
> URL: https://issues.apache.org/jira/browse/IGNITE-7337
> Project: Ignite
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 2.3
>Reporter: Valentin Kulichenko
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5
>
>
> Once Ignite data source for Spark is implemented, we need to add an ability 
> to store a data frame in Ignite. Most likely if should be enough to provide 
> implementation for the following traits:
> * {{InsertableRelation}}
> * {{CreatableRelationProvider}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7596) Add Functionality for data structures to be able to remove any failed node on data structure creation

[Tim Onyschak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Onyschak reassigned IGNITE-7596:


Assignee: Tim Onyschak

> Add Functionality for data structures to be able to remove any failed node on 
> data structure creation
> -
>
> Key: IGNITE-7596
> URL: https://issues.apache.org/jira/browse/IGNITE-7596
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache, data structures
>Affects Versions: 2.4
>Reporter: Tim Onyschak
>Assignee: Tim Onyschak
>Priority: Major
>
> Improvement of IGNITE-7090 to allow any data structure the ability to fix 
> state when new instances are created. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7596) Add Functionality for data structures to be able to remove any failed node on data structure creation

[Tim Onyschak (JIRA)]

Tim Onyschak created IGNITE-7596:


 Summary: Add Functionality for data structures to be able to 
remove any failed node on data structure creation
 Key: IGNITE-7596
 URL: https://issues.apache.org/jira/browse/IGNITE-7596
 Project: Ignite
  Issue Type: Improvement
  Components: cache, data structures
Affects Versions: 2.4
Reporter: Tim Onyschak


Improvement of IGNITE-7090 to allow any data structure the ability to fix state 
when new instances are created. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7090) Semaphore Stuck when no acquirers to assign permit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347920#comment-16347920
 ] 

ASF GitHub Bot commented on IGNITE-7090:


Github user tonyschak closed the pull request at:

https://github.com/apache/ignite/pull/3138


> Semaphore Stuck when no acquirers to assign permit
> --
>
> Key: IGNITE-7090
> URL: https://issues.apache.org/jira/browse/IGNITE-7090
> Project: Ignite
>  Issue Type: Bug
>  Components: cache, data structures
>Affects Versions: 2.1, 2.4
>Reporter: Tim Onyschak
>Assignee: Tim Onyschak
>Priority: Major
> Fix For: 2.5
>
> Attachments: SemaphoreFailoverNoWaitingAcquirerTest.java
>
>
> If no acquirers are available to take permit of semaphore, the permit never 
> gets release and any further acquirerers will wait forever. 
> On node shut down DataStructuresProcessor.dsMap gets cleared out prior to 
> event listener being able to execute onNodeRemoved, hence owner is never 
> cleared out if it was unable to pass to a different acquirer. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7090) Semaphore Stuck when no acquirers to assign permit

[Tim Onyschak (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347919#comment-16347919
 ] 

Tim Onyschak edited comment on IGNITE-7090 at 2/1/18 2:22 AM:
--

Latest pull request [https://github.com/apache/ignite/pull/3443]


was (Author: timay):
Latest https://github.com/apache/ignite/pull/3443

> Semaphore Stuck when no acquirers to assign permit
> --
>
> Key: IGNITE-7090
> URL: https://issues.apache.org/jira/browse/IGNITE-7090
> Project: Ignite
>  Issue Type: Bug
>  Components: cache, data structures
>Affects Versions: 2.1, 2.4
>Reporter: Tim Onyschak
>Assignee: Tim Onyschak
>Priority: Major
> Fix For: 2.5
>
> Attachments: SemaphoreFailoverNoWaitingAcquirerTest.java
>
>
> If no acquirers are available to take permit of semaphore, the permit never 
> gets release and any further acquirerers will wait forever. 
> On node shut down DataStructuresProcessor.dsMap gets cleared out prior to 
> event listener being able to execute onNodeRemoved, hence owner is never 
> cleared out if it was unable to pass to a different acquirer. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-1535) Evicted entries still are presented

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-1535:
--
Fix Version/s: 2.5

> Evicted entries still are presented 
> 
>
> Key: IGNITE-1535
> URL: https://issues.apache.org/jira/browse/IGNITE-1535
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: ignite-1.4
>Reporter: Sergey Kozlov
>Priority: Critical
> Fix For: 2.5
>
> Attachments: CacheFifoEvictExample.java, grid_evict.xml
>
>
> Test creates 4 caches: 2 PARTITIONED for testing and 2 REPLICATED for 
> comparison. Each cache has FIFO eviction policy with maxSize=750.
> Test puts 1000 entries (order: 1 ... 1000) and check that 1..250 entries are 
> removed and 251..1000 presented in cache.
> 1. Copy grid_evict.xml in examples/config directory
> 2. Copy CacheFifoEvictExample in 
> examples/src/main/java/org/apache/ignite/examples/datagrid
> 3. Build examples
> 4. Run 3 nodes bin/ignite.sh examples/config/grid_evict.xml
> 5. Run CacheFifoEvictExample
> Expected output:
> {noformat}
> >>>cache_1
> put keys 1..1000
> get keys 1..1000
> >>>cache_2
> put keys 1..1000
> get keys 1..1000
> {noformat}
> Bur for PARTITIONED caches I got
> {noformat}
> >>>cache_1
> put keys 1..1000
> get keys 1..1000
> >>> not null value found for 18
> >>> not null value found for 19
> >>> not null value found for 20
> >>> not null value found for 21
> >>> not null value found for 22
> >>> not null value found for 23
> >>> not null value found for 24
> >>> not null value found for 25
> >>> not null value found for 28
> ...
> >>>cache_2
> put keys 1..1000
> get keys 1..1000
> >>> not null value found for 1
> >>> not null value found for 3
> >>> not null value found for 6
> >>> not null value found for 10
> >>> not null value found for 18
> >>> not null value found for 19
> >>> not null value found for 20
> >>> not null value found for 21
> >>> not null value found for 22
> >>> not null value found for 23
> >>> not null value found for 24
> >>> not null value found for 25
> ...
> {noformat}
> Delay between puts and gets doesn't fix the issue but add null values for 
> keys > 250



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-1535) Evicted entries still are presented

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan reassigned IGNITE-1535:
-

Assignee: Alexey Goncharuk

> Evicted entries still are presented 
> 
>
> Key: IGNITE-1535
> URL: https://issues.apache.org/jira/browse/IGNITE-1535
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: ignite-1.4
>Reporter: Sergey Kozlov
>Assignee: Alexey Goncharuk
>Priority: Critical
> Fix For: 2.5
>
> Attachments: CacheFifoEvictExample.java, grid_evict.xml
>
>
> Test creates 4 caches: 2 PARTITIONED for testing and 2 REPLICATED for 
> comparison. Each cache has FIFO eviction policy with maxSize=750.
> Test puts 1000 entries (order: 1 ... 1000) and check that 1..250 entries are 
> removed and 251..1000 presented in cache.
> 1. Copy grid_evict.xml in examples/config directory
> 2. Copy CacheFifoEvictExample in 
> examples/src/main/java/org/apache/ignite/examples/datagrid
> 3. Build examples
> 4. Run 3 nodes bin/ignite.sh examples/config/grid_evict.xml
> 5. Run CacheFifoEvictExample
> Expected output:
> {noformat}
> >>>cache_1
> put keys 1..1000
> get keys 1..1000
> >>>cache_2
> put keys 1..1000
> get keys 1..1000
> {noformat}
> Bur for PARTITIONED caches I got
> {noformat}
> >>>cache_1
> put keys 1..1000
> get keys 1..1000
> >>> not null value found for 18
> >>> not null value found for 19
> >>> not null value found for 20
> >>> not null value found for 21
> >>> not null value found for 22
> >>> not null value found for 23
> >>> not null value found for 24
> >>> not null value found for 25
> >>> not null value found for 28
> ...
> >>>cache_2
> put keys 1..1000
> get keys 1..1000
> >>> not null value found for 1
> >>> not null value found for 3
> >>> not null value found for 6
> >>> not null value found for 10
> >>> not null value found for 18
> >>> not null value found for 19
> >>> not null value found for 20
> >>> not null value found for 21
> >>> not null value found for 22
> >>> not null value found for 23
> >>> not null value found for 24
> >>> not null value found for 25
> ...
> {noformat}
> Delay between puts and gets doesn't fix the issue but add null values for 
> keys > 250



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7504) Decision tree documentation

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda reassigned IGNITE-7504:
---

Assignee: Denis Magda  (was: Yury Babak)

> Decision tree documentation
> ---
>
> Key: IGNITE-7504
> URL: https://issues.apache.org/jira/browse/IGNITE-7504
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Yury Babak
>Assignee: Denis Magda
>Priority: Major
>  Labels: documentation
> Fix For: 2.4
>
>
> We want to add Decision tree documentation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7587) SQL COPY: document the command

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347841#comment-16347841
 ] 

Denis Magda commented on IGNITE-7587:
-

[~kirill.shirokov] , please don't forget to set "fix version". Set it to 2.4 
for now. 

> SQL COPY: document the command
> --
>
> Key: IGNITE-7587
> URL: https://issues.apache.org/jira/browse/IGNITE-7587
> Project: Ignite
>  Issue Type: Improvement
>  Components: documentation, sql
>Affects Versions: 2.4
>Reporter: Kirill Shirokov
>Priority: Major
> Fix For: 2.4
>
>
> SQL COPY command needs to be documented at readme.io.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7587) SQL COPY: document the command

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347841#comment-16347841
 ] 

Denis Magda edited comment on IGNITE-7587 at 2/1/18 1:00 AM:
-

[~kirill.shirokov] , please don't forget to set "fix version". Set it to 2.4 
for now. If it will not be released in 2.4 then change to 2.5


was (Author: dmagda):
[~kirill.shirokov] , please don't forget to set "fix version". Set it to 2.4 
for now. 

> SQL COPY: document the command
> --
>
> Key: IGNITE-7587
> URL: https://issues.apache.org/jira/browse/IGNITE-7587
> Project: Ignite
>  Issue Type: Improvement
>  Components: documentation, sql
>Affects Versions: 2.4
>Reporter: Kirill Shirokov
>Priority: Major
> Fix For: 2.4
>
>
> SQL COPY command needs to be documented at readme.io.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7587) SQL COPY: document the command

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-7587:

Fix Version/s: 2.4

> SQL COPY: document the command
> --
>
> Key: IGNITE-7587
> URL: https://issues.apache.org/jira/browse/IGNITE-7587
> Project: Ignite
>  Issue Type: Improvement
>  Components: documentation, sql
>Affects Versions: 2.4
>Reporter: Kirill Shirokov
>Priority: Major
> Fix For: 2.4
>
>
> SQL COPY command needs to be documented at readme.io.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7582) Document 'No Archiver' WAL mode

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-7582:

Fix Version/s: 2.5

> Document 'No Archiver' WAL mode
> ---
>
> Key: IGNITE-7582
> URL: https://issues.apache.org/jira/browse/IGNITE-7582
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, persistence
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
> Fix For: 2.5
>
>
> Document new 'no archiver' WAL mode activated if WAL Path & WAL Archive Path 
> set to same value.
> Supported under [IGNITE-7017]
> In this mode all WAL segment files are placed in that folder with absolute 
> index in file name. In that case archiver is not activated.
> Configuration example:
> {noformat}
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> DataRegionConfiguration regCfg = new DataRegionConfiguration();
> regCfg.setPersistenceEnabled(true);
> dsCfg.setDefaultDataRegionConfiguration(regCfg);
> dsCfg.setWalPath(walAbsPath);
> dsCfg.setWalArchivePath(walAbsPath);
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7582) Document 'No Archiver' WAL mode

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347840#comment-16347840
 ] 

Denis Magda commented on IGNITE-7582:
-

[~dpavlov] , does this go into 2.5 release? Please don't forget to set the fix 
version. I set it to 2.5 for this ticket.

> Document 'No Archiver' WAL mode
> ---
>
> Key: IGNITE-7582
> URL: https://issues.apache.org/jira/browse/IGNITE-7582
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, persistence
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
> Fix For: 2.5
>
>
> Document new 'no archiver' WAL mode activated if WAL Path & WAL Archive Path 
> set to same value.
> Supported under [IGNITE-7017]
> In this mode all WAL segment files are placed in that folder with absolute 
> index in file name. In that case archiver is not activated.
> Configuration example:
> {noformat}
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> DataRegionConfiguration regCfg = new DataRegionConfiguration();
> regCfg.setPersistenceEnabled(true);
> dsCfg.setDefaultDataRegionConfiguration(regCfg);
> dsCfg.setWalPath(walAbsPath);
> dsCfg.setWalArchivePath(walAbsPath);
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7337) Spark Data Frames: support saving a data frame in Ignite

[Valentin Kulichenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347812#comment-16347812
 ] 

Valentin Kulichenko commented on IGNITE-7337:
-

[~NIzhikov], let's do this renamings:
 * {{OPTION_CREATE_TABLE_OPTIONS}} -> {{OPTION_CREATE_TABLE_PARAMETERS}}
 * {{OPTION_PRIMARY_KEY_FIELDS}} -> {{OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS}}

{{OPTION_CACHE_FOR_DDL}} should go away. Our JDBC drivers and Web Console do 
not require to have a dummy cache, I believe they go though internal API to 
achieve this. Please investigate and propose a solution.

What is {{OPTION_WRITE_PARTITIONS_NUM}}?

> Spark Data Frames: support saving a data frame in Ignite
> 
>
> Key: IGNITE-7337
> URL: https://issues.apache.org/jira/browse/IGNITE-7337
> Project: Ignite
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 2.3
>Reporter: Valentin Kulichenko
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5
>
>
> Once Ignite data source for Spark is implemented, we need to add an ability 
> to store a data frame in Ignite. Most likely if should be enough to provide 
> implementation for the following traits:
> * {{InsertableRelation}}
> * {{CreatableRelationProvider}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7337) Spark Data Frames: support saving a data frame in Ignite

[Valentin Kulichenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347812#comment-16347812
 ] 

Valentin Kulichenko edited comment on IGNITE-7337 at 2/1/18 12:16 AM:
--

[~NIzhikov], let's do these renamings:
 * {{OPTION_CREATE_TABLE_OPTIONS}} -> {{OPTION_CREATE_TABLE_PARAMETERS}}
 * {{OPTION_PRIMARY_KEY_FIELDS}} -> {{OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS}}

{{OPTION_CACHE_FOR_DDL}} should go away. Our JDBC drivers and Web Console do 
not require to have a dummy cache, I believe they go though internal API to 
achieve this. Please investigate and propose a solution.

What is {{OPTION_WRITE_PARTITIONS_NUM}}?


was (Author: vkulichenko):
[~NIzhikov], let's do this renamings:
 * {{OPTION_CREATE_TABLE_OPTIONS}} -> {{OPTION_CREATE_TABLE_PARAMETERS}}
 * {{OPTION_PRIMARY_KEY_FIELDS}} -> {{OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS}}

{{OPTION_CACHE_FOR_DDL}} should go away. Our JDBC drivers and Web Console do 
not require to have a dummy cache, I believe they go though internal API to 
achieve this. Please investigate and propose a solution.

What is {{OPTION_WRITE_PARTITIONS_NUM}}?

> Spark Data Frames: support saving a data frame in Ignite
> 
>
> Key: IGNITE-7337
> URL: https://issues.apache.org/jira/browse/IGNITE-7337
> Project: Ignite
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 2.3
>Reporter: Valentin Kulichenko
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5
>
>
> Once Ignite data source for Spark is implemented, we need to add an ability 
> to store a data frame in Ignite. Most likely if should be enough to provide 
> implementation for the following traits:
> * {{InsertableRelation}}
> * {{CreatableRelationProvider}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7595) Find and switch to alternate documentation engine

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-7595:

Component/s: documentation

> Find and switch to alternate documentation engine
> -
>
> Key: IGNITE-7595
> URL: https://issues.apache.org/jira/browse/IGNITE-7595
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Denis Magda
>Assignee: Prachi Garg
>Priority: Critical
> Fix For: 2.5
>
>
> Current readme.io documentation has many drawbacks that make the life of 
> Ignite technical writers hard. Some of the problems are:
>  * Each "version" is just a copy of the previous one. When fixing something, 
> you have to update
> all the versions.
>  * No good way to review changes.
>  * "Propose edit" functionality is a not suitable for review. You can only 
> accept or reject an
> edit, no way to communicate with a contributor, etc
>  * There is no way to prevent Google from indexing old documentation 
> versions. Thus, it's common to come across old doc version in a google 
> search. 
> We might consider GitHub based documentation or another approach. The 
> discussion is here:
> http://apache-ignite-developers.2346864.n4.nabble.com/Move-documentation-from-readme-io-to-GitHub-pages-td16409.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7595) Find and switch to alternate documentation engine

[Denis Magda (JIRA)]

Denis Magda created IGNITE-7595:
---

 Summary: Find and switch to alternate documentation engine
 Key: IGNITE-7595
 URL: https://issues.apache.org/jira/browse/IGNITE-7595
 Project: Ignite
  Issue Type: Task
Reporter: Denis Magda
Assignee: Prachi Garg
 Fix For: 2.5


Current readme.io documentation has many drawbacks that make the life of Ignite 
technical writers hard. Some of the problems are:
 * Each "version" is just a copy of the previous one. When fixing something, 
you have to update
all the versions.
 * No good way to review changes.
 * "Propose edit" functionality is a not suitable for review. You can only 
accept or reject an
edit, no way to communicate with a contributor, etc
 * There is no way to prevent Google from indexing old documentation versions. 
Thus, it's common to come across old doc version in a google search. 

We might consider GitHub based documentation or another approach. The 
discussion is here:

http://apache-ignite-developers.2346864.n4.nabble.com/Move-documentation-from-readme-io-to-GitHub-pages-td16409.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7594) Tx performance drop after WAL optimization

[Aleksey Chetaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Chetaev updated IGNITE-7594:

Description: 
Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.

WAL Mode: Default

First bad commit: 
[https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d]

  was:
Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.

First bad commit: 
https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d


> Tx performance drop after WAL optimization 
> ---
>
> Key: IGNITE-7594
> URL: https://issues.apache.org/jira/browse/IGNITE-7594
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Aleksey Chetaev
>Priority: Critical
>
> Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.
> WAL Mode: Default
> First bad commit: 
> [https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7594) Tx performance drop after WAL optimization

[Aleksey Chetaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Chetaev updated IGNITE-7594:

Summary: Tx performance drop after WAL optimization   (was: Tx performance 
drop after WAL optimization, in DEFAULT WAL mode)

> Tx performance drop after WAL optimization 
> ---
>
> Key: IGNITE-7594
> URL: https://issues.apache.org/jira/browse/IGNITE-7594
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Aleksey Chetaev
>Priority: Critical
>
> Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.
> First bad commit: 
> https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7594) Tx performance drop after WAL optimization, in DEFAULT WAL mode

[Aleksey Chetaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Chetaev updated IGNITE-7594:

Summary: Tx performance drop after WAL optimization, in DEFAULT WAL mode  
(was: Tx performance drop after WAL optimization in DEFAULT WAL mode)

> Tx performance drop after WAL optimization, in DEFAULT WAL mode
> ---
>
> Key: IGNITE-7594
> URL: https://issues.apache.org/jira/browse/IGNITE-7594
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Aleksey Chetaev
>Priority: Critical
>
> Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.
> First bad commit: 
> https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7594) Tx performance drop after WAL optimization in DEFAULT WAL mode

[Aleksey Chetaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Chetaev updated IGNITE-7594:

Summary: Tx performance drop after WAL optimization in DEFAULT WAL mode  
(was: Tx performance drop after WAL optimization )

> Tx performance drop after WAL optimization in DEFAULT WAL mode
> --
>
> Key: IGNITE-7594
> URL: https://issues.apache.org/jira/browse/IGNITE-7594
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Aleksey Chetaev
>Priority: Critical
>
> Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.
> First bad commit: 
> https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7594) Tx performance drop after WAL optimization

[Aleksey Chetaev (JIRA)]

Aleksey Chetaev created IGNITE-7594:
---

 Summary: Tx performance drop after WAL optimization 
 Key: IGNITE-7594
 URL: https://issues.apache.org/jira/browse/IGNITE-7594
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4
Reporter: Aleksey Chetaev


Perfomance dropes in tx-putAll benchmarks after commit with WAL optimization.

First bad commit: 
https://github.com/apache/ignite/commit/a5ffd4eb18e6e9eab30c176a7bb4008a51b3d59d



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-4361) Documentation: working with Ignite through JCache API

[Prachi Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prachi Garg reassigned IGNITE-4361:
---

Assignee: Denis Magda  (was: Prachi Garg)

> Documentation: working with Ignite through JCache API
> -
>
> Key: IGNITE-4361
> URL: https://issues.apache.org/jira/browse/IGNITE-4361
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Denis Magda
>Assignee: Denis Magda
>Priority: Minor
> Fix For: 2.4
>
>
> Regardless of the fact that Ignite supports JCache API there is no any 
> documentation with code snippets that will elaborate on how to work with 
> Ignite purely with JCache API.
> This documentation [1] must contain a section showing all the ways how a 
> JCache Ignite provider can be created and used after that.
> [1] https://apacheignite.readme.io/docs/jcache



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (IGNITE-6946) Change Ignite Logger configuration on the fly

[Valentin Kulichenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentin Kulichenko closed IGNITE-6946.
---

> Change Ignite Logger configuration on the fly
> -
>
> Key: IGNITE-6946
> URL: https://issues.apache.org/jira/browse/IGNITE-6946
> Project: Ignite
>  Issue Type: Improvement
>  Components: general
>Affects Versions: 2.3
>Reporter: Alexey Popov
>Assignee: Stanislav Lukyanov
>Priority: Major
> Fix For: 2.5
>
>
> It would be great if we can change Ignite Logger configuration on the fly 
> without restarting the cluster node. It will really help to debug an issue 
> while it is reproducible at the current cluster state.
>  It should be done within the configured type of a logger, i.e. it is enough 
> to change logging levels without changing the Logger type itself.
> Such configuration change should be done for all supported logger types (JUL, 
> log4j, log4i2 and others).
> Also it should be done easily, for instance, via Visor, WebConsole or any 
> other user-friendly tool ).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7504) Decision tree documentation

[Oleg Ignatenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347438#comment-16347438
 ] 

Oleg Ignatenko commented on IGNITE-7504:


Approach with adding dataset to examples directory doesn't look viable because 
of licensing considerations (discussed in a bit more details in IGNITE-7593).

Because of that we picked a bit different way that still requires upload from 
external URL but decreases user involvement to absolute minimum, that is they 
need to just pick yes or no when prompted whether to download data. This is 
implemented in IGNITE-7590.

In order to try further improve data used in this example I opened a dedicated 
ticket IGNITE-7593. I targeted this ticket for next release (2.5) because it 
looks rather time consuming and is unlikely to fit into timeframe of release 
2.4.

[~dmagda] I would appreciate if you take a look at considerations laid out 
above. If you have further questions or suggestions please let us know.

 

> Decision tree documentation
> ---
>
> Key: IGNITE-7504
> URL: https://issues.apache.org/jira/browse/IGNITE-7504
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Yury Babak
>Assignee: Yury Babak
>Priority: Major
>  Labels: documentation
> Fix For: 2.4
>
>
> We want to add Decision tree documentation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (IGNITE-7469) Update Apache Ignite documentation about RPM-repository usage

[Prachi Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prachi Garg closed IGNITE-7469.
---

> Update Apache Ignite documentation about RPM-repository usage
> -
>
> Key: IGNITE-7469
> URL: https://issues.apache.org/jira/browse/IGNITE-7469
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Peter Ivanov
>Assignee: Prachi Garg
>Priority: Major
> Fix For: 2.4
>
>
> After IGNITE-7107 release in 2.4 version Apache Ignite will be available 
> through custom RPM repository packed in RPM package. Corresponding 
> documentation (how to add RPM-repository and install from it) should be 
> created.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7593) Improve data used in DecisionTreesExample

[Oleg Ignatenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Ignatenko updated IGNITE-7593:
---
Description: 
Data currently used in {{DecisionTreesExample}} looks not quite optimal:

# It is large, as evidenced in the warning in javadocs: "It is recommended to 
start at least one node prior to launching this example if you intend to run it 
with default memory settings."
# It makes example run for quite a long time.
# It doesn't have license (likely meaning "all rights reserved" by default) 
which makes it troublesome to include in project sources so that current 
approach is to prompt user to download it, additionally complicated by making 
example skip when run unattended from {{IgniteExamplesMLTestSuite}}.

Suggest to find or construct a smaller data for this example which would still 
make sense to demonstrate how algorithm works and in the same time would be 1) 
easier on memory usage, 2) quicker to run and 3) would allow carrying it within 
project instead of prompting user to download it.

  was:
Data currently used in {{DecisionTreesExample}} looks not quite optimal:

# It is large, as evidenced in the warning in javadocs: "It is recommended to 
start at least one node prior to launching this example if you intend to run it 
with default memory settings."
# It makes example run for quite a long time.


> Improve data used in DecisionTreesExample
> -
>
> Key: IGNITE-7593
> URL: https://issues.apache.org/jira/browse/IGNITE-7593
> Project: Ignite
>  Issue Type: Task
>  Components: ml
>Affects Versions: 2.4
>Reporter: Oleg Ignatenko
>Priority: Major
> Fix For: 2.5
>
>
> Data currently used in {{DecisionTreesExample}} looks not quite optimal:
> # It is large, as evidenced in the warning in javadocs: "It is recommended to 
> start at least one node prior to launching this example if you intend to run 
> it with default memory settings."
> # It makes example run for quite a long time.
> # It doesn't have license (likely meaning "all rights reserved" by default) 
> which makes it troublesome to include in project sources so that current 
> approach is to prompt user to download it, additionally complicated by making 
> example skip when run unattended from {{IgniteExamplesMLTestSuite}}.
> Suggest to find or construct a smaller data for this example which would 
> still make sense to demonstrate how algorithm works and in the same time 
> would be 1) easier on memory usage, 2) quicker to run and 3) would allow 
> carrying it within project instead of prompting user to download it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7593) Improve data used in DecisionTreesExample

[Oleg Ignatenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Ignatenko reassigned IGNITE-7593:
--

Assignee: Artem Malykh

> Improve data used in DecisionTreesExample
> -
>
> Key: IGNITE-7593
> URL: https://issues.apache.org/jira/browse/IGNITE-7593
> Project: Ignite
>  Issue Type: Task
>  Components: ml
>Affects Versions: 2.4
>Reporter: Oleg Ignatenko
>Assignee: Artem Malykh
>Priority: Major
> Fix For: 2.5
>
>
> Data currently used in {{DecisionTreesExample}} looks not quite optimal:
> # It is large, as evidenced in the warning in javadocs: "It is recommended to 
> start at least one node prior to launching this example if you intend to run 
> it with default memory settings."
> # It makes example run for quite a long time.
> # It doesn't have license (likely meaning "all rights reserved" by default) 
> which makes it troublesome to include in project sources so that current 
> approach is to prompt user to download it, additionally complicated by making 
> example skip when run unattended from {{IgniteExamplesMLTestSuite}}.
> Suggest to find or construct a smaller data for this example which would 
> still make sense to demonstrate how algorithm works and in the same time 
> would be 1) easier on memory usage, 2) quicker to run and 3) would allow 
> carrying it within project instead of prompting user to download it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7532) kNN Documentation

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda reassigned IGNITE-7532:
---

Assignee: Aleksey Zinoviev  (was: Denis Magda)

> kNN Documentation
> -
>
> Key: IGNITE-7532
> URL: https://issues.apache.org/jira/browse/IGNITE-7532
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Aleksey Zinoviev
>Assignee: Aleksey Zinoviev
>Priority: Major
>  Labels: documentaion
> Fix For: 2.4
>
>
> We want to add kNN Regression and Classification docs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7532) kNN Documentation

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347392#comment-16347392
 ] 

Denis Magda commented on IGNITE-7532:
-

[~zaleslaw] , I did a general review. There are some of the points that need 
your attention/assistance:
 * k-NN classification doc says - _trainingSet dataset loaded that might be 
taken from Iris datasets._ Please provide a link to the Iris datasets you're 
talking about.
 * This paragraph of k-NN regression doesn't make sense to me:

_Also, k-NN regression's output is a property value for the object. This value 
is an average of the values of its k nearest neighbors. If `k` is `1`, then the 
object is simply assigned to the property value of that single nearest 
neighbor._

What is an object? What's a property value? Please refer to other ML 
documentations (Spark and others) so that I can grasp why you're trying to 
explain. Alternatively, you can write down a Russian version of the paragraph 
and I'll translate it to English.
 * k-NN regression says - _`_trainingSet_` dataset that might be loaded from 
the machine.names dataset._ What's the machine.names dataset? We need to paste 
the reference. Make sure all these properly documented in the examples.

> kNN Documentation
> -
>
> Key: IGNITE-7532
> URL: https://issues.apache.org/jira/browse/IGNITE-7532
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Aleksey Zinoviev
>Assignee: Denis Magda
>Priority: Major
>  Labels: documentaion
> Fix For: 2.4
>
>
> We want to add kNN Regression and Classification docs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7593) Improve data used in DecisionTreesExample

[Oleg Ignatenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Ignatenko updated IGNITE-7593:
---
Fix Version/s: 2.5

> Improve data used in DecisionTreesExample
> -
>
> Key: IGNITE-7593
> URL: https://issues.apache.org/jira/browse/IGNITE-7593
> Project: Ignite
>  Issue Type: Task
>  Components: ml
>Affects Versions: 2.4
>Reporter: Oleg Ignatenko
>Priority: Major
> Fix For: 2.5
>
>
> Data currently used in {{DecisionTreesExample}} looks not quite optimal:
> # It is large, as evidenced in the warning in javadocs: "It is recommended to 
> start at least one node prior to launching this example if you intend to run 
> it with default memory settings."
> # It makes example run for quite a long time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7593) Improve data used in DecisionTreesExample

[Oleg Ignatenko (JIRA)]

Oleg Ignatenko created IGNITE-7593:
--

 Summary: Improve data used in DecisionTreesExample
 Key: IGNITE-7593
 URL: https://issues.apache.org/jira/browse/IGNITE-7593
 Project: Ignite
  Issue Type: Task
  Components: ml
Affects Versions: 2.4
Reporter: Oleg Ignatenko


Data currently used in {{DecisionTreesExample}} looks not quite optimal:

# It is large, as evidenced in the warning in javadocs: "It is recommended to 
start at least one node prior to launching this example if you intend to run it 
with default memory settings."
# It makes example run for quite a long time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7590) Change example for decision trees

[Oleg Ignatenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347368#comment-16347368
 ] 

Oleg Ignatenko commented on IGNITE-7590:


I checked PR #3459 on my machine.

Code style and design look OK, and example behaved correctly when I tried it in 
different modes: 1) from {{IgniteExamplesMLTestSuite}} (unattended) 2) 
standalone from IDE when data is not yet downloaded (checked both answering 
_no_ and _yes_ to download prompt) 3) standalone from IDE when data is already 
downloaded and unzipped and available to use so that there is no need in 
download prompt.

Looks good to merge.

> Change example for decision trees 
> --
>
> Key: IGNITE-7590
> URL: https://issues.apache.org/jira/browse/IGNITE-7590
> Project: Ignite
>  Issue Type: Improvement
>  Components: ml
>Reporter: Artem Malykh
>Assignee: Artem Malykh
>Priority: Major
> Fix For: 2.4
>
>
> Change example for decision trees according to comments from [~dmagda] at 
> https://issues.apache.org/jira/browse/IGNITE-7504 .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7590) Change example for decision trees

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347355#comment-16347355
 ] 

ASF GitHub Bot commented on IGNITE-7590:


GitHub user artemmalykh opened a pull request:

https://github.com/apache/ignite/pull/3459

IGNITE-7590: Change example for decision trees



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7590

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3459


commit cfebbea977b040c839bd65161bae09af64f13cff
Author: Artem Malykh 
Date:   2018-01-31T15:04:29Z

IGNITE-7590: Refactored example.

commit 6ca33be44e766433dec91e6fe29a42cd9e0ae52d
Author: Artem Malykh 
Date:   2018-01-31T16:08:03Z

IGNITE-7590: Downloading code.

commit 2c1c97f9807186a4f8089266cf012b44bd5104d9
Author: Artem Malykh 
Date:   2018-01-31T16:26:47Z

Merge branch 'master-apache' into ignite-7590

commit 3da939740d371a3f6d861783b2f6b2317eb5070e
Author: Artem Malykh 
Date:   2018-01-31T17:32:27Z

IGNITE-7590: Fixes.

commit 01fa94a64d3a0e3858a96190cb0154efcf338d3b
Author: Artem Malykh 
Date:   2018-01-31T17:49:36Z

IGNITE-7590: Added 'unattended' mode.

commit df3975f6dd88e574e85da6a2964bccca9f89549f
Author: Artem Malykh 
Date:   2018-01-31T18:15:42Z

IGNITE-7590: Fixes.

commit 203720944687b2b2b42f41fba4dce416bcb828e2
Author: Artem Malykh 
Date:   2018-01-31T18:30:25Z

IGNITE-7590: Fixes.

commit 06d5e93f4b25c0b7bd9d284098eb655c5fbfee2c
Author: Artem Malykh 
Date:   2018-01-31T18:32:54Z

IGNITE-7590: Fixes.




> Change example for decision trees 
> --
>
> Key: IGNITE-7590
> URL: https://issues.apache.org/jira/browse/IGNITE-7590
> Project: Ignite
>  Issue Type: Improvement
>  Components: ml
>Reporter: Artem Malykh
>Assignee: Artem Malykh
>Priority: Major
> Fix For: 2.4
>
>
> Change example for decision trees according to comments from [~dmagda] at 
> https://issues.apache.org/jira/browse/IGNITE-7504 .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7511) FCM documentation

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda reassigned IGNITE-7511:
---

Assignee: Prachi Garg  (was: Denis Magda)

> FCM documentation
> -
>
> Key: IGNITE-7511
> URL: https://issues.apache.org/jira/browse/IGNITE-7511
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Yury Babak
>Assignee: Prachi Garg
>Priority: Major
> Fix For: 2.4
>
>
> We need to update documentation on readme.io and add section about FCM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7511) FCM documentation

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347338#comment-16347338
 ] 

Denis Magda commented on IGNITE-7511:
-

[~oignatenko] , thanks! I've improved the docs a little bit and added "Example" 
section that refers to existing examples (so that people can learn more about 
the API and how to use it).

[~pgarg] , please review the following pages and close the ticket:

[https://apacheignite.readme.io/v2.3/docs/fuzzy-c-means-clustering-24]

https://apacheignite.readme.io/v2.3/docs/k-means-clustering-24

 

> FCM documentation
> -
>
> Key: IGNITE-7511
> URL: https://issues.apache.org/jira/browse/IGNITE-7511
> Project: Ignite
>  Issue Type: Task
>  Components: documentation, ml
>Reporter: Yury Babak
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
>
> We need to update documentation on readme.io and add section about FCM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5059) Implement logistic regression

[Aleksey Zinoviev (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347335#comment-16347335
 ] 

Aleksey Zinoviev commented on IGNITE-5059:
--

[~vladisav] Could I assign this ticket on myself.

I'd really want to add this algorithm to our ML scope as soon as possible.

The reason: after that implementation we will have comparable set of algorithms 
with Spark MLLib after 2.5 release.

Currently L1, L2 norms and SGD are implemented in LinearSVM and LinReg with SGD 
trainers

> Implement logistic regression 
> --
>
> Key: IGNITE-5059
> URL: https://issues.apache.org/jira/browse/IGNITE-5059
> Project: Ignite
>  Issue Type: New Feature
>  Components: ml
>Reporter: Vladisav Jelisavcic
>Assignee: Vladisav Jelisavcic
>Priority: Major
>  Labels: important
>
> Implement logistic regression using ignite ml.math. Model should be able to 
> incorporate L1 and L2 regularization. 
> Model should also work with stochastic gradient descent (SGD) as well as 
> batch and mini-batch gradient descent optimization algorithms.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (IGNITE-7408) Document WAL changes

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda closed IGNITE-7408.
---

> Document WAL changes
> 
>
> Key: IGNITE-7408
> URL: https://issues.apache.org/jira/browse/IGNITE-7408
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Andrey Gura
>Assignee: Denis Magda
>Priority: Major
>  Labels: documentation
> Fix For: 2.4
>
>
> WAL documentation should be updated accordingly to IGNITE-6339 issue.
> The following changes can affect WAL configuration and behavior:
>  # {{DataStorageConfiguration.setWalBufferSize}} added instead of 
> {{PersistentStoreConfiguration.setTlbSize}}. By default WAL buffer size 
> equals WAL segment size if memory mapped file is enabled, and (WAL segment 
> size) / 4 if memory mapped file is disabled.
>  # Memory mapped file is enabled by default and provides better performance. 
> Memory mapped file usage can be turned off. {{IGNITE_WAL_MMAP}} property with 
> {{false}} value should be added to the JVM arguments: 
> {{-DIGNITE_WAL_MMAP=false}}.
>  # If memory mapped file is enabled then {{BACKGROUND}} and {{LOG_ONLY}} WAL 
> modes behave identically and don't have any differences in performance or 
> data integrity guarantees.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (IGNITE-7358) Document ELB based AWS IP Finder

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda closed IGNITE-7358.
---

> Document ELB based AWS IP Finder
> 
>
> Key: IGNITE-7358
> URL: https://issues.apache.org/jira/browse/IGNITE-7358
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Denis Magda
>Assignee: Denis Magda
>Priority: Critical
> Fix For: 2.4
>
>
> Document the new IP finder that can be used for AWS deployments - IGNITE-7304.
> The main documentation should be hosted here:
> https://apacheignite-mix.readme.io/docs/amazon-aws
> and there we need to provide a reference to ELB IP finder:
> https://apacheignite.readme.io/docs/cluster-config



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347233#comment-16347233
 ] 

Alexander Paschenko commented on IGNITE-6917:
-

And I'll rename all points to use the same single through numeration, sorry for 
the inconvenience.

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then transfer data to 
> the cluster, and then execute update inside the cluster, e.g. through 
> streamer.
> First of all we need to create quick and dirty prototype to assess potential 
> performance improvement. It speedup is confirmed, we should build base 
> implementation which will accept only files. But at the same time we should 
> understand how it will evolve in future: multiple file formats (probably 
> including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
> etc..
> [1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
> h1. Proposed syntax
> Curent implementation:
> {noformat}
> COPY 
> FROM "file.name"
> INTO .
> [(col-name, ...)]
> FORMAT  -- Only CSV format is supported in the current 
> release
> [BATCH_SIZE ]
> {noformat}
> We may want to gradually add features to this command in future to have 
> something like this:
> {noformat}
> COPY
> FROM "file.name"[CHARSET ""]
> INTO . [CREATE [IF NOT EXISTS]]
> [(col-name [] [NULLABLE] [ESCAPES], ...) [MATCH HEADER]]
> FORMAT (csv|tsv|...)
> -- CSV format options:
> [FIELDSEP='column-separators-regexp']
> [LINESEP='row-separators-regexp']
> [QUOTE='quote-chars']
> [ESCAPE='escape-char']
> [NULL='null-sequence']
> [COMMENT='single-line-comment-start-char']
> [TRIM_LINES]
> [IMPORT_EMPTY_LINES]
> [CHARSET ""]
> [ROWS -]
> --or--
> [SKIP ROWS ] [MAX ROWS ]
> [COLS -]
> --or--
> [SKIP COLS ] [MAX COLS ]
> [(MATCH | SKIP) HEADER]
> [(REPLACE|IGNORE|ABORT ON [])) DUPLICATE KEYS]
> [BATCH SIZE ( ROWS | [K|M|G|T|P])]
> [COMPRESS "codec-name" [codec options]]
> [LOCK (TABLE|ROWS)]
> [NOLOGGING]
> [BACKEND (DIRECT | STREAMER)]
> {noformat}
> h1. Implementation decisions and notes
> h2. Parsing
> * We support CSV format described in RFC 4180.
> * Custom row and column separators, quoting characters are currently hardcoded
> * Escape sequences, line comment characters are currently not supported
> * We may want to support fixed-length formats (via format descriptors) in 
> future
> * We may want to strip comments from lines (for example, starting with '#')
> * We may want to allow user to either ignore empty lines or treat them as a 
> special case of record having all default values
> * We may allow user to enable whitespace trimming from beginning and end of a 
> line
> * We may want to allow user to specify error handling strategy: e.g., only 
> one quote character is present or escape sequence is invalid.
> h2. File handling
> * File character set to be supported in future
> * Skipped/imported row number (or first/last line or skip header option), 
> skipped/imported column number (or first/last column): to be supported in 
> future
> * Line start pattern (as in MySQL): no support planned
> * We currently support only client-side import. No server-side file import.
> * We may want to support client-side stdin import in future.
> * We do not handle importing multiple files from single command
> * We don't benefit from any kind of pre-sorting pre-partitioning data on 
> client side.
> * We don't include any any metadata, such as line number from client side.
> h3. Transferring data
> * We send file data via batches. In future we will support batch size 
> (specified with rows per batch or data block size 
> per batch).
> * We may want to implement data compression in future.
> * We connect to single node in JDBC driver (no multi-node connections).
> h3. Cache/tables/column handling
> * We don't create table in the bulk load command
> * We may want to have and option for reading header row, which contains 
> column names to match columns
> * In future we may wish to support COLUMNS (col1, _, col2, _, col3) syntax, 
> where 

[jira] [Comment Edited] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347230#comment-16347230
 ] 

Alexander Paschenko edited comment on IGNITE-6917 at 1/31/18 5:28 PM:
--

[~kirill.shirokov],

Regarding 7.2 - yes, please disregard empty columns list. Suppose there's _key 
and _val.

Regarding empty lines - they are always added for readability but we don't add 
them right after line ending with {, be it method or class declaration or loop 
or switch. Yes, with long method declarations this may not look good, but for 
consistency let's stick with what I suggest, thanks.

Regarding 5 - ok, Vlad's always right. Still IMHO main thing is not simpler 
code but better usability here. (Just saying.)


was (Author: al.psc):
[~kirill.shirokov],

Regarding 7.2 - yes, please disregard empty columns list. Suppose there's _key 
and _val.

Regarding empty fields - they are always added for readability but we don't add 
them right after line ending with {, be it method or class declaration or loop 
or switch. Yes, with long method declarations this may not look good, but for 
consistency let's stick with what I suggest, thanks.

Regarding 5 - ok, Vlad's always right. Still IMHO main thing is not simpler 
code but better usability here. (Just saying.)

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then transfer data to 
> the cluster, and then execute update inside the cluster, e.g. through 
> streamer.
> First of all we need to create quick and dirty prototype to assess potential 
> performance improvement. It speedup is confirmed, we should build base 
> implementation which will accept only files. But at the same time we should 
> understand how it will evolve in future: multiple file formats (probably 
> including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
> etc..
> [1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
> h1. Proposed syntax
> Curent implementation:
> {noformat}
> COPY 
> FROM "file.name"
> INTO .
> [(col-name, ...)]
> FORMAT  -- Only CSV format is supported in the current 
> release
> [BATCH_SIZE ]
> {noformat}
> We may want to gradually add features to this command in future to have 
> something like this:
> {noformat}
> COPY
> FROM "file.name"[CHARSET ""]
> INTO . [CREATE [IF NOT EXISTS]]
> [(col-name [] [NULLABLE] [ESCAPES], ...) [MATCH HEADER]]
> FORMAT (csv|tsv|...)
> -- CSV format options:
> [FIELDSEP='column-separators-regexp']
> [LINESEP='row-separators-regexp']
> [QUOTE='quote-chars']
> [ESCAPE='escape-char']
> [NULL='null-sequence']
> [COMMENT='single-line-comment-start-char']
> [TRIM_LINES]
> [IMPORT_EMPTY_LINES]
> [CHARSET ""]
> [ROWS -]
> --or--
> [SKIP ROWS ] [MAX ROWS ]
> [COLS -]
> --or--
> [SKIP COLS ] [MAX COLS ]
> [(MATCH | SKIP) HEADER]
> [(REPLACE|IGNORE|ABORT ON [])) DUPLICATE KEYS]
> [BATCH SIZE ( ROWS | [K|M|G|T|P])]
> [COMPRESS "codec-name" [codec options]]
> [LOCK (TABLE|ROWS)]
> [NOLOGGING]
> [BACKEND (DIRECT | STREAMER)]
> {noformat}
> h1. Implementation decisions and notes
> h2. Parsing
> * We support CSV format described in RFC 4180.
> * Custom row and column separators, quoting characters are currently hardcoded
> * Escape sequences, line comment characters are currently not supported
> * We may want to support fixed-length formats (via format descriptors) in 
> future
> * We may want to strip comments from lines (for example, starting with '#')
> * We may want to allow user to either ignore empty lines or treat them as a 
> special case of record having all default values
> * We may allow user to enable whitespace trimming from beginning and end of a 
> line
> * We may want to allow user to specify error handling strategy: e.g., only 
> one quote character is present or escape sequence is invalid.
> h2. File handling
> * File character set to be supported in future
> * Skipped/imported row number (or first/last line or skip header option), 
> skipped/imported column number (or first/last column): to be 

[jira] [Commented] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347230#comment-16347230
 ] 

Alexander Paschenko commented on IGNITE-6917:
-

[~kirill.shirokov],

Regarding 7.2 - yes, please disregard empty columns list. Suppose there's _key 
and _val.

Regarding empty fields - they are always added for readability but we don't add 
them right after line ending with {, be it method or class declaration or loop 
or switch. Yes, with long method declarations this may not look good, but for 
consistency let's stick with what I suggest, thanks.

Regarding 5 - ok, Vlad's always right. Still IMHO main thing is not simpler 
code but better usability here. (Just saying.)

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then transfer data to 
> the cluster, and then execute update inside the cluster, e.g. through 
> streamer.
> First of all we need to create quick and dirty prototype to assess potential 
> performance improvement. It speedup is confirmed, we should build base 
> implementation which will accept only files. But at the same time we should 
> understand how it will evolve in future: multiple file formats (probably 
> including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
> etc..
> [1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
> h1. Proposed syntax
> Curent implementation:
> {noformat}
> COPY 
> FROM "file.name"
> INTO .
> [(col-name, ...)]
> FORMAT  -- Only CSV format is supported in the current 
> release
> [BATCH_SIZE ]
> {noformat}
> We may want to gradually add features to this command in future to have 
> something like this:
> {noformat}
> COPY
> FROM "file.name"[CHARSET ""]
> INTO . [CREATE [IF NOT EXISTS]]
> [(col-name [] [NULLABLE] [ESCAPES], ...) [MATCH HEADER]]
> FORMAT (csv|tsv|...)
> -- CSV format options:
> [FIELDSEP='column-separators-regexp']
> [LINESEP='row-separators-regexp']
> [QUOTE='quote-chars']
> [ESCAPE='escape-char']
> [NULL='null-sequence']
> [COMMENT='single-line-comment-start-char']
> [TRIM_LINES]
> [IMPORT_EMPTY_LINES]
> [CHARSET ""]
> [ROWS -]
> --or--
> [SKIP ROWS ] [MAX ROWS ]
> [COLS -]
> --or--
> [SKIP COLS ] [MAX COLS ]
> [(MATCH | SKIP) HEADER]
> [(REPLACE|IGNORE|ABORT ON [])) DUPLICATE KEYS]
> [BATCH SIZE ( ROWS | [K|M|G|T|P])]
> [COMPRESS "codec-name" [codec options]]
> [LOCK (TABLE|ROWS)]
> [NOLOGGING]
> [BACKEND (DIRECT | STREAMER)]
> {noformat}
> h1. Implementation decisions and notes
> h2. Parsing
> * We support CSV format described in RFC 4180.
> * Custom row and column separators, quoting characters are currently hardcoded
> * Escape sequences, line comment characters are currently not supported
> * We may want to support fixed-length formats (via format descriptors) in 
> future
> * We may want to strip comments from lines (for example, starting with '#')
> * We may want to allow user to either ignore empty lines or treat them as a 
> special case of record having all default values
> * We may allow user to enable whitespace trimming from beginning and end of a 
> line
> * We may want to allow user to specify error handling strategy: e.g., only 
> one quote character is present or escape sequence is invalid.
> h2. File handling
> * File character set to be supported in future
> * Skipped/imported row number (or first/last line or skip header option), 
> skipped/imported column number (or first/last column): to be supported in 
> future
> * Line start pattern (as in MySQL): no support planned
> * We currently support only client-side import. No server-side file import.
> * We may want to support client-side stdin import in future.
> * We do not handle importing multiple files from single command
> * We don't benefit from any kind of pre-sorting pre-partitioning data on 
> client side.
> * We don't include any any metadata, such as line number from client side.
> h3. Transferring data
> * We send file data via batches. In future we will support batch size 
> (specified with rows per batch or data block 

[jira] [Comment Edited] (IGNITE-7533) Throttle writting threads according fsync progress and checkpoint writting speed instead of region fill

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347192#comment-16347192
 ] 

Dmitriy Pavlov edited comment on IGNITE-7533 at 1/31/18 5:25 PM:
-

Warning is also generated each 10secs if throttling is applied and affects 
current mark process more than 20%
{noformat}
Throttling is applied to page mark [weight=0,20, mark=7905 pages/sec, 
checkpointWrite=32945 pages/sec, estIdealMark=9869 pages/sec, curDirty=0,73, 
targetDirty=0,69, avgParkTime=203099 ns, pages: (total=373443, evicted=10891, 
written=360734, synced=300052)]
{noformat}


was (Author: dpavlov):
Warning is also generated each 10secs if throttling is applied and affects 
current mark process more than 20%
{noformat}
Throttling is applied to page mark [weight=0,27, mark speed=18299 pages/sec, 
checkpoint write speed=19016 pages/sec, est.ideal mark=18301 pages/sec, 
curDirty=0,06, targetDirty=0,06, avgParkTime=117471 ns]
{noformat}

> Throttle writting threads according fsync progress and checkpoint writting 
> speed instead of region fill
> ---
>
> Key: IGNITE-7533
> URL: https://issues.apache.org/jira/browse/IGNITE-7533
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Affects Versions: 2.3
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
> Fix For: 2.5
>
> Attachments: image (2).png, imageRatioBased.png, imageSpeedBased.png
>
>
> Throttling implemented under IGNITE-6334 is based on region fill percentage 
> (ditry pages ratio) and current checkpoint progress.
> But actual progress of writting is based on write operation complete, but not 
> on fsync() complete.
> Suppose following stage of CP is currently running: most of data is being 
> written and fsync is started. Fsync on experiments requires more time than 
> write, but throttling is disabled for that stage. There is enough time to 
> unthrottled grid to fill remaining 16% of clear pages to get sufficient 75% 
> of dirty pages and writes stops.
> Fsync progress is to be included in checkpoint progress, but actual fsync 
> progress reported by OS is irregular both on Linux and Windows. See picture, 
> green line is fsync progress, and yellow is write complete.
> Because fsync progress reported is not regular (the differences are 3-4 
> orders of magnitude) it is suggested to implement new speed based throttling 
> policy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347177#comment-16347177
 ] 

Alexander Paschenko edited comment on IGNITE-6917 at 1/31/18 5:22 PM:
--

[~kirill.shirokov], please allow me to proceed.

 

1. {{SqlParserCopySelfTest}} - no single test for positive cases, only negative 
stuff. We need to cover positive cases extensively as well.

2. {{JdbcBulkLoadContext}} - probably we don't need this class and it would be 
better to keep update counter in {{BulkLoadContext}} as the notion of counter 
is used not just in JDBC driver.

3. {{JdbcThinStatement}} -

3.1 let's catch and wrap any exception in {{sendFile}}

3.2 Odd empty line in the beginning of {{sendFile}}

4. {{CsvLineProcessorBlock}} - we don't handle cases when quoted strings in CSV 
line contain commas, do we? This clearly does not seem right and I don't see 
any tests for that either. Shouldn't we handle escaped quotes too?

5. All {{*PipelineBlock}} classes - odd empty line after class header.

6. {{StrListAppenderBlock}} - do we need list of lists here? Probably just list 
of arrays could do?

7. {{BulkLoadContextCursor}}{{}} - it's unclear to me why we return context in 
fake "row" while all other parameters should be retrieved via getters. Can we 
unify this? (E.g. add getters for everything AND return everything in fake row 
- that would be consistent.)

8. Please add more tests for behavior in case of failure - server or client 
disconnect during file processing, file open error on client, etc.

I think this is it for now. Looking forward to the fixes.


was (Author: al.psc):
[~kirill.shirokov], please allow me to proceed.

 

1. {{SqlParserCopySelfTest}} - no single test for positive cases, only negative 
stuff. We need to cover positive cases extensively as well.

2. {{JdbcBulkLoadContext}} - probably we don't need this class and it would be 
better to keep update counter in {{BulkLoadContext}} as the notion of counter 
is used not just in JDBC driver.

3. {{JdbcThinStatement}} -

3.1 let's catch and wrap any exception in {{sendFile}}

3.2 Odd empty line in the beginning of {{sendFile}}

4. {{CsvLineProcessorBlock}} - we don't handle cases when quoted strings in CSV 
line contain commas, do we? This clearly does not seem right and I don't see 
any tests for that either. Shouldn't we handle escaped quotes too?

5. All {{*PipelineBlock}} classes - odd empty line after class header.

6. {{StrListAppenderBlock}} - do we need list of lists here? Probably just list 
of arrays could do?

7. {{BulkLoadContextCursor}}{{}} - it's unclear to me why we return context in 
fake "row" while all other parameters should be retrieved via getters. Can we 
unify this? (E.g. add getters for everything AND return everything in fake row 
- that would be consistent.)

8. Please add more tests for behavior in case of failure - server or client 
disconnect during file processing, file open error on client, etc.

I think this is it for now. Looking forward for the fixes.

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then transfer data to 
> the cluster, and then execute update inside the cluster, e.g. through 
> streamer.
> First of all we need to create quick and dirty prototype to assess potential 
> performance improvement. It speedup is confirmed, we should build base 
> implementation which will accept only files. But at the same time we should 
> understand how it will evolve in future: multiple file formats (probably 
> including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
> etc..
> [1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
> h1. Proposed syntax
> Curent implementation:
> {noformat}
> COPY 
> FROM "file.name"
> INTO .
> [(col-name, ...)]
> FORMAT  -- Only CSV format is supported in the current 
> release
> [BATCH_SIZE ]
> {noformat}
> We may want to gradually add features to this command in future to have 
> something like this:
> {noformat}
> COPY
> FROM "file.name"[CHARSET ""]
> INTO . [CREATE [IF NOT EXISTS]]
> [(col-name [] 

[jira] [Commented] (IGNITE-3999) Support case insensitive search in SQL

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347216#comment-16347216
 ] 

Alexander Paschenko commented on IGNITE-3999:
-

[~aakhmedov] I've started looking at the patch. I'm in progress, meanwhile 
please ensure that:
 * all method overrides have javadoc like this: /** \{@inheritDoc} */
 * all new parameters have matching javadocs.

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Kirill Shirokov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347213#comment-16347213
 ] 

Kirill Shirokov edited comment on IGNITE-6917 at 1/31/18 5:18 PM:
--

So you have 3 lists with the same numbering :)

Ok, list #1:

> 1. JdbcAbstractUpdateStatementSelfTest - unused code

FIXED

> 2. org.apache.ignite.internal.jdbc2.JdbcBulkLoadSelfTest: createConnection - 
> unused param; testBulkLoadThrows - please use try-with-resources.

FIXED

> 3. org.apache.ignite.jdbc.thin.JdbcThinAbstractDmlStatementSelfTest:
 > 3.1 jdbcRun: please remove this method and use inherited method execute 
 > instead.

FIXED

> 3.2 testDOA: please rename into something more meaningful.

FIXED (=> testBasicStatement())

> 4. BulkLoadContextCursor:
 > 4.1 Please make field final.

FIXED

> 4.2 getAll - I changed the code to double call of singletonList just fine.

This time it worked.

> 4.3 getFieldName - I believe this method should throw NoSuchElementException 
> for all i except 0.

It looks more logical to throw IndexOutOfBoundsException (if you look at the 
existing implementations)

I don't think it's ever being called, but OK.

> 4.4 Odd empty line right after class header.

There's a lot of odd empty lines in this project, which don't seem logical or 
have nice appearance, but OK.

An example: you have a lot of empty lines around while and this is easily 
readable:
{noformat}
    private void parseColumnList(SqlLexer lex) {
    if (!lex.shift() || lex.tokenType() != 
SqlLexerTokenType.PARENTHESIS_LEFT)
    throw errorUnexpectedToken(lex, "(");

    while (true) {
    parseIndexColumn(lex);

    if (skipCommaOrRightParenthesis(lex))
    break;
    }
    }{noformat}
But there are no empty lines in the top of the method and if you have 5+ 
arguments and exceptions declared there, it's almost not readable.

> 5. SqlBulkLoadCommand: let's parse command in a way consistent with how it's 
> done in SQL usually - we clearly should not use ugly stuff like BATCH_SIZE 
> (with underscore instead of space) now that we are headed to our own parser. 
> Please rework parsing so that user has to write BATCH SIZE. (NB: yes, I'm 
> aware that we already have quite ugly stuff here, like INLINE_SIZE for 
> indexes, but I believe that this must be avoided as much as possible.) Also I 
> think we should support case when the user does not supply columns list (so 
> no empty brackets too) - currently it's a parsing error and there's a test 
> for it. (Please disregard what's been struck through.)

I also find it ugly and asked Vladimir about this, but he insisted on this ugly 
solution.

But to be optimistic about this, the approach with the underscore has some 
advantages:

1. More specific keywords (user has more freedom: more identifiers aren't 
clashing with keywords)
 2. A bit simpler code

> 6. UpdatePlan#processRow:
 > 6.1 Odd empty line after method declaration.

As in 4.4 it looks like I don't understand logic behind this. Or it's not about 
aestetics.

> 6.2. Shouldn't we rather check for equality here? row.size() < 
> colNames.length - like this

When we are importing CSV, every row can have its own number of columns. I 
think it's OK to silently skip extra columns. 
 Or we can handle this CSV-specific case in the code which is calling 
processRow().

> 7. UpdatePlanBuilder#planForBulkLoad:
 > 7.1 Odd empty line after method declaration.

FIXED with the same comment as in 4.4

> 7.2 Let's add VERY simple test case (like cache of  and bulk 
> load to it) - I think this may break on current implementation as we may 
> actually will to insert into _val column, and this case does not seem to be 
> handled - say, like this COPY from file to String () format csv (String is 
> table name, table has 3 columns all of which are system columns.)

Yes, I planned tests for auto-generated fields like _key and _val. But I think 
it makes sense to implement them as a part of this JIRA.

Although we decided to disallow allow empty or missing column list in the 
current implementation – this has to be handled as a separate JIRA.


was (Author: kirill.shirokov):
So you have 3 lists with the same numbering :)

Ok, list #1:

> 1. JdbcAbstractUpdateStatementSelfTest - unused code

FIXED

> 2. org.apache.ignite.internal.jdbc2.JdbcBulkLoadSelfTest: createConnection - 
> unused param; testBulkLoadThrows - please use try-with-resources.

FIXED

> 3. org.apache.ignite.jdbc.thin.JdbcThinAbstractDmlStatementSelfTest:
> 3.1 jdbcRun: please remove this method and use inherited method execute 
> instead.

FIXED

> 3.2 testDOA: please rename into something more meaningful.

FIXED (=> testBasicStatement())

> 4. BulkLoadContextCursor:
> 4.1 Please make field final.

FIXED

> 4.2 getAll - I changed the code to double call of singletonList just fine.

This time it worked.

> 

[jira] [Commented] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Kirill Shirokov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347213#comment-16347213
 ] 

Kirill Shirokov commented on IGNITE-6917:
-

So you have 3 lists with the same numbering :)

Ok, list #1:

> 1. JdbcAbstractUpdateStatementSelfTest - unused code

FIXED

> 2. org.apache.ignite.internal.jdbc2.JdbcBulkLoadSelfTest: createConnection - 
> unused param; testBulkLoadThrows - please use try-with-resources.

FIXED

> 3. org.apache.ignite.jdbc.thin.JdbcThinAbstractDmlStatementSelfTest:
> 3.1 jdbcRun: please remove this method and use inherited method execute 
> instead.

FIXED

> 3.2 testDOA: please rename into something more meaningful.

FIXED (=> testBasicStatement())

> 4. BulkLoadContextCursor:
> 4.1 Please make field final.

FIXED

> 4.2 getAll - I changed the code to double call of singletonList just fine.

This time it worked.

> 4.3 getFieldName - I believe this method should throw NoSuchElementException 
> for all i except 0.

It looks more logical to throw IndexOutOfBoundsException (if you look at the 
existing implementations)

I don't think it's ever being called, but OK.

> 4.4 Odd empty line right after class header.

There's a lot of odd empty lines in this project, which don't seem logical or 
have nice appearance, but OK.

An example: you have a lot of empty lines around while and this is easily 
readable:

private void parseColumnList(SqlLexer lex) {
if (!lex.shift() || lex.tokenType() != 
SqlLexerTokenType.PARENTHESIS_LEFT)
throw errorUnexpectedToken(lex, "(");

while (true) {
parseIndexColumn(lex);

if (skipCommaOrRightParenthesis(lex))
break;
}
}

But there are no empty lines in the top of the method and if you have 5+ 
arguments and exceptions declared there, it's almost not readable.

> 5. SqlBulkLoadCommand: let's parse command in a way consistent with how it's 
> done in SQL usually - we clearly should not use ugly stuff like BATCH_SIZE 
> (with underscore instead of space) now that we are headed to our own parser. 
> Please rework parsing so that user has to write BATCH SIZE. (NB: yes, I'm 
> aware that we already have quite ugly stuff here, like INLINE_SIZE for 
> indexes, but I believe that this must be avoided as much as possible.) Also I 
> think we should support case when the user does not supply columns list (so 
> no empty brackets too) - currently it's a parsing error and there's a test 
> for it. (Please disregard what's been struck through.)

I also find it ugly and asked Vladimir about this, but he insisted on this ugly 
solution.

But this approach has some advantages as well: 

1. More specific keywords (user has more freedom: more identifiers aren't 
clashing with keywords)
2. A bit simpler code

> 6. UpdatePlan#processRow:
> 6.1 Odd empty line after method declaration.

As in 4.4 it looks like I don't understand logic behind this. Or it's not about 
aestetics.

> 6.2. Shouldn't we rather check for equality here? row.size() < 
> colNames.length - like this

When we are importing CSV, every row can have its own number of columns. I 
think it's OK to silently skip extra columns. 
Or we can handle this CSV-specific case in the code which is calling 
processRow().

> 7. UpdatePlanBuilder#planForBulkLoad:
> 7.1 Odd empty line after method declaration.

FIXED with the same comment as in 4.4

> 7.2 Let's add VERY simple test case (like cache of  and bulk 
> load to it) - I think this may break on current implementation as we may 
> actually will to insert into _val column, and this case does not seem to be 
> handled - say, like this COPY from file to String () format csv (String is 
> table name, table has 3 columns all of which are system columns.)

Yes, I planned tests for auto-generated fields like _key and _val. But I think 
it makes sense to implement them as a part of this JIRA.

Although we decided to disallow allow empty or missing column list in the 
current implementation -- this has to be handled as a separate JIRA.


> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then 

[jira] [Issue Comment Deleted] (IGNITE-3999) Support case insensitive search in SQL

[Alexander Paschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Paschenko updated IGNITE-3999:

Comment: was deleted

(was: Upsource review for the patch)

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-3999) Support case insensitive search in SQL

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347197#comment-16347197
 ] 

Alexander Paschenko commented on IGNITE-3999:
-

Added UpSource review link for the patch.

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-3999) Support case insensitive search in SQL

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347196#comment-16347196
 ] 

Alexander Paschenko commented on IGNITE-3999:
-

Upsource review for the patch

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-3999) Support case insensitive search in SQL

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347191#comment-16347191
 ] 

Alexander Paschenko commented on IGNITE-3999:
-

[~aakhmedov], [~vkulichenko], I will do that, thanks

> Support case insensitive search in SQL
> --
>
> Key: IGNITE-3999
> URL: https://issues.apache.org/jira/browse/IGNITE-3999
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Amir Akhmedov
>Priority: Critical
> Fix For: 2.5
>
>
> Currently case insensitive search is possible only with the help of 
> {{lower()}} function:
> {code}
> select name from MyValue where lower(name) = 'abc_5'
> {code}
> But this will always be a full scan, even if {{name}} field is indexed.
> We need to correctly support {{VARCHAR_IGNORECASE}} H2 type in Ignite and add 
> a respective property to {{@QuerySqlField}} annotation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7533) Throttle writting threads according fsync progress and checkpoint writting speed instead of region fill

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347192#comment-16347192
 ] 

Dmitriy Pavlov commented on IGNITE-7533:


Warning is also generated each 10secs if throttling is applied and affects 
current mark process more than 20%
{noformat}
Throttling is applied for page mark [weight=0,27, mark speed=18299 pages/sec, 
checkpoint write speed=19016 pages/sec, est.ideal mark=18301 pages/sec, 
curDirty=0,06, targetDirty=0,06, avgParkTime=117471 ns]
{noformat}

> Throttle writting threads according fsync progress and checkpoint writting 
> speed instead of region fill
> ---
>
> Key: IGNITE-7533
> URL: https://issues.apache.org/jira/browse/IGNITE-7533
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Affects Versions: 2.3
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
> Fix For: 2.5
>
> Attachments: image (2).png, imageRatioBased.png, imageSpeedBased.png
>
>
> Throttling implemented under IGNITE-6334 is based on region fill percentage 
> (ditry pages ratio) and current checkpoint progress.
> But actual progress of writting is based on write operation complete, but not 
> on fsync() complete.
> Suppose following stage of CP is currently running: most of data is being 
> written and fsync is started. Fsync on experiments requires more time than 
> write, but throttling is disabled for that stage. There is enough time to 
> unthrottled grid to fill remaining 16% of clear pages to get sufficient 75% 
> of dirty pages and writes stops.
> Fsync progress is to be included in checkpoint progress, but actual fsync 
> progress reported by OS is irregular both on Linux and Windows. See picture, 
> green line is fsync progress, and yellow is write complete.
> Because fsync progress reported is not regular (the differences are 3-4 
> orders of magnitude) it is suggested to implement new speed based throttling 
> policy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7533) Throttle writting threads according fsync progress and checkpoint writting speed instead of region fill

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347192#comment-16347192
 ] 

Dmitriy Pavlov edited comment on IGNITE-7533 at 1/31/18 5:07 PM:
-

Warning is also generated each 10secs if throttling is applied and affects 
current mark process more than 20%
{noformat}
Throttling is applied to page mark [weight=0,27, mark speed=18299 pages/sec, 
checkpoint write speed=19016 pages/sec, est.ideal mark=18301 pages/sec, 
curDirty=0,06, targetDirty=0,06, avgParkTime=117471 ns]
{noformat}


was (Author: dpavlov):
Warning is also generated each 10secs if throttling is applied and affects 
current mark process more than 20%
{noformat}
Throttling is applied for page mark [weight=0,27, mark speed=18299 pages/sec, 
checkpoint write speed=19016 pages/sec, est.ideal mark=18301 pages/sec, 
curDirty=0,06, targetDirty=0,06, avgParkTime=117471 ns]
{noformat}

> Throttle writting threads according fsync progress and checkpoint writting 
> speed instead of region fill
> ---
>
> Key: IGNITE-7533
> URL: https://issues.apache.org/jira/browse/IGNITE-7533
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Affects Versions: 2.3
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
> Fix For: 2.5
>
> Attachments: image (2).png, imageRatioBased.png, imageSpeedBased.png
>
>
> Throttling implemented under IGNITE-6334 is based on region fill percentage 
> (ditry pages ratio) and current checkpoint progress.
> But actual progress of writting is based on write operation complete, but not 
> on fsync() complete.
> Suppose following stage of CP is currently running: most of data is being 
> written and fsync is started. Fsync on experiments requires more time than 
> write, but throttling is disabled for that stage. There is enough time to 
> unthrottled grid to fill remaining 16% of clear pages to get sufficient 75% 
> of dirty pages and writes stops.
> Fsync progress is to be included in checkpoint progress, but actual fsync 
> progress reported by OS is irregular both on Linux and Windows. See picture, 
> green line is fsync progress, and yellow is write complete.
> Because fsync progress reported is not regular (the differences are 3-4 
> orders of magnitude) it is suggested to implement new speed based throttling 
> policy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347177#comment-16347177
 ] 

Alexander Paschenko commented on IGNITE-6917:
-

[~kirill.shirokov], please allow me to proceed.

 

1. {{SqlParserCopySelfTest}} - no single test for positive cases, only negative 
stuff. We need to cover positive cases extensively as well.

2. {{JdbcBulkLoadContext}} - probably we don't need this class and it would be 
better to keep update counter in {{BulkLoadContext}} as the notion of counter 
is used not just in JDBC driver.

3. {{JdbcThinStatement}} -

3.1 let's catch and wrap any exception in {{sendFile}}

3.2 Odd empty line in the beginning of {{sendFile}}

4. {{CsvLineProcessorBlock}} - we don't handle cases when quoted strings in CSV 
line contain commas, do we? This clearly does not seem right and I don't see 
any tests for that either. Shouldn't we handle escaped quotes too?

5. All {{*PipelineBlock}} classes - odd empty line after class header.

6. {{StrListAppenderBlock}} - do we need list of lists here? Probably just list 
of arrays could do?

7. {{BulkLoadContextCursor}}{{}} - it's unclear to me why we return context in 
fake "row" while all other parameters should be retrieved via getters. Can we 
unify this? (E.g. add getters for everything AND return everything in fake row 
- that would be consistent.)

8. Please add more tests for behavior in case of failure - server or client 
disconnect during file processing, file open error on client, etc.

I think this is it for now. Looking forward for the fixes.

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
> only possible to execute single commands one by one. We already can batch 
> them to improve performance, but there is still big room for improvement.
> We should think of a completely new command - {{COPY}}. It will accept a file 
> (or input stream in general case) on the client side, then transfer data to 
> the cluster, and then execute update inside the cluster, e.g. through 
> streamer.
> First of all we need to create quick and dirty prototype to assess potential 
> performance improvement. It speedup is confirmed, we should build base 
> implementation which will accept only files. But at the same time we should 
> understand how it will evolve in future: multiple file formats (probably 
> including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
> etc..
> [1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
> h1. Proposed syntax
> Curent implementation:
> {noformat}
> COPY 
> FROM "file.name"
> INTO .
> [(col-name, ...)]
> FORMAT  -- Only CSV format is supported in the current 
> release
> [BATCH_SIZE ]
> {noformat}
> We may want to gradually add features to this command in future to have 
> something like this:
> {noformat}
> COPY
> FROM "file.name"[CHARSET ""]
> INTO . [CREATE [IF NOT EXISTS]]
> [(col-name [] [NULLABLE] [ESCAPES], ...) [MATCH HEADER]]
> FORMAT (csv|tsv|...)
> -- CSV format options:
> [FIELDSEP='column-separators-regexp']
> [LINESEP='row-separators-regexp']
> [QUOTE='quote-chars']
> [ESCAPE='escape-char']
> [NULL='null-sequence']
> [COMMENT='single-line-comment-start-char']
> [TRIM_LINES]
> [IMPORT_EMPTY_LINES]
> [CHARSET ""]
> [ROWS -]
> --or--
> [SKIP ROWS ] [MAX ROWS ]
> [COLS -]
> --or--
> [SKIP COLS ] [MAX COLS ]
> [(MATCH | SKIP) HEADER]
> [(REPLACE|IGNORE|ABORT ON [])) DUPLICATE KEYS]
> [BATCH SIZE ( ROWS | [K|M|G|T|P])]
> [COMPRESS "codec-name" [codec options]]
> [LOCK (TABLE|ROWS)]
> [NOLOGGING]
> [BACKEND (DIRECT | STREAMER)]
> {noformat}
> h1. Implementation decisions and notes
> h2. Parsing
> * We support CSV format described in RFC 4180.
> * Custom row and column separators, quoting characters are currently hardcoded
> * Escape sequences, line comment characters are currently not supported
> * We may want to support fixed-length formats (via format descriptors) in 
> future
> * We may want to strip comments from lines (for example, starting with '#')
> * We may want to allow user to either ignore empty lines or treat them as a 
> special case of record having all default values
> * We may allow user to enable whitespace trimming from beginning and end of a 
> line
> * We may want to allow user to specify error handling strategy: e.g., only 
> one 

[jira] [Resolved] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn resolved IGNITE-7473.

Resolution: Fixed

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.4
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347163#comment-16347163
 ] 

Pavel Tupitsyn commented on IGNITE-7473:


Weakening generic constraint to {{class}} fixed the issue.

Merged to master: {{0088405ce305a915174c53bcafcc9a1740da64c7}}.
Cherry-picked to ignite-2.4: {{aca3b8c664de8dcbbfeabbb0f8c252194d6ad1b2}}.

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.4
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7591) Backup node crash causes incorrect transaction failover recovery

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347141#comment-16347141
 ] 

Anton Vinogradov commented on IGNITE-7591:
--

[~Alexey Kuznetsov]

Sleeps never gives you any warranty.

Is it possible to update reproducer?

> Backup node crash causes incorrect transaction failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7591) Backup node crash causes incorrect transaction failover recovery

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347141#comment-16347141
 ] 

Anton Vinogradov edited comment on IGNITE-7591 at 1/31/18 4:35 PM:
---

[~Alexey Kuznetsov]

Sleeps never give you any warranty.

Is it possible to update reproducer?


was (Author: avinogradov):
[~Alexey Kuznetsov]

Sleeps never gives you any warranty.

Is it possible to update reproducer?

> Backup node crash causes incorrect transaction failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-7473:
---
Fix Version/s: (was: 2.5)
   2.4

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.4
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347088#comment-16347088
 ] 

Pavel Tupitsyn edited comment on IGNITE-7473 at 1/31/18 4:26 PM:
-

Reopening. Turns out it worked in nightly builds right after IGNITE-2662 was 
merged, and was broken after IGNITE-6615 merge.

Looks like this is caused by generic {{GetConfigurationSection}} method, may 
be removing generic constraint will fix the problem?


was (Author: ptupitsyn):
Reopening. Turns out it worked in nightly builds right after IGNITE-2662 was 
merged, and was broken after IGNITE-6615 merge.

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.5
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-7473:
---
Fix Version/s: (was: 2.4)
   2.5

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.5
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (IGNITE-7473) .NET: "Could not load file or assembly ConfigurationManager" error on Linux

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn reopened IGNITE-7473:


Reopening. Turns out it worked in nightly builds right after IGNITE-2662 was 
merged, and was broken after IGNITE-6615 merge.

> .NET: "Could not load file or assembly ConfigurationManager" error on Linux
> ---
>
> Key: IGNITE-7473
> URL: https://issues.apache.org/jira/browse/IGNITE-7473
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET, xplat
> Fix For: 2.5
>
>
> Steps to reproduce:
> {code}
> dotnet new console
> dotnet add package Apache.Ignite
> nano Program.cs  # Add Ignition.Start() line and a namespace import
> dotnet run
> {code}
> Error:
> {code}
> Unhandled Exception: System.IO.FileNotFoundException: Could not load file or 
> assembly 'System.Configuration.ConfigurationManager, Version=0.0.0.0, 
> Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the 
> file specified.
> {code}
> Works on Windows, does not work on Linux.
> {{ Version="4.4.0" />}} is required to fix the issue.
> However, {{ConfigurationManager}} is not necessary for code configuration, we 
> should be able to start without it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7591) Backup node crash causes incorrect transaction failover recovery

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347075#comment-16347075
 ] 

Andrew Mashenkov commented on IGNITE-7591:
--

You expect localPeek will return '4' after backup node stop.

I'd think correct way will be
 # to ensure thread that set '4' is the last thread that tries to get a lock. 
Sleep(1s) doen't give such guarantee.
 # wait for all transactions finished before calling localPeek. AFAIK, local 
peek ignores entry locks and may read stale value.

> Backup node crash causes incorrect transaction failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IGNITE-7387) Add Activate\Deactivate lifecycle event support.

[Evgenii Zhuravlev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgenii Zhuravlev resolved IGNITE-7387.
---
Resolution: Duplicate

> Add Activate\Deactivate lifecycle event support.
> 
>
> Key: IGNITE-7387
> URL: https://issues.apache.org/jira/browse/IGNITE-7387
> Project: Ignite
>  Issue Type: Improvement
>  Components: general
>Reporter: Andrew Mashenkov
>Priority: Major
>
> For now Ignite has a number of LifecycleEvent type that allow user to perform 
> some logic before node join\leave topology.
> However, with persistence enabled Ignite node start in non-active state and 
> AFTER_NODE_START event is useless in some cases as user may need to spin-wait 
> for activation.
> We should add AFTER_ACTIVATE and BEFORE_DEACTIVATE lifecycle event types. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IGNITE-7179) Create events for cluster activation/deactivation

[Evgenii Zhuravlev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgenii Zhuravlev resolved IGNITE-7179.
---
Resolution: Duplicate

> Create events for cluster activation/deactivation
> -
>
> Key: IGNITE-7179
> URL: https://issues.apache.org/jira/browse/IGNITE-7179
> Project: Ignite
>  Issue Type: Improvement
>  Components: general
>Affects Versions: 2.3
>Reporter: Valentin Kulichenko
>Priority: Major
>
> It would be great to have a distributed system event (similar to discovery 
> vents) fired when cluster is activated or deactivated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-6899) Adding GA Grid to Apache Ignite ML module.

[Yury Babak (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347038#comment-16347038
 ] 

Yury Babak commented on IGNITE-6899:


thread on dev list: 
http://apache-ignite-developers.2346864.n4.nabble.com/GA-Grid-Request-to-contribute-GA-library-to-Apache-Ignite-tt23920.html

> Adding GA Grid to Apache Ignite ML module.
> --
>
> Key: IGNITE-6899
> URL: https://issues.apache.org/jira/browse/IGNITE-6899
> Project: Ignite
>  Issue Type: New Feature
>  Components: ml
>Reporter: Yury Babak
>Assignee: Turik Campbell
>Priority: Major
> Fix For: 2.5
>
> Attachments: coverage.zip
>
>
> We want to add GA Grid to our ML Module.
> This is the first iteration of this integration. On this step we will simple 
> add GA Grid to the separate package in ML module.
> (i) This is a good package for GA Grid: org.apache.ignite.ml.genetic 
> (i) For GA Grid we need unit tests as well as examples



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7584) Smooth decrease WAL write speed if low work segments left

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347018#comment-16347018
 ] 

ASF GitHub Bot commented on IGNITE-7584:


GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3458

IGNITE-7584: WAL log throttle implementation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7584

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3458.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3458


commit 90befa9d3bc89833eeed42a8814e8a7f51a11448
Author: dpavlov 
Date:   2018-01-25T18:11:07Z

IGNITE-7533: Throttle writing threads according fsync progress and 
checkpoint write speed

commit 74411e7aea9744df7f7656006807aa0403ae921f
Author: dpavlov 
Date:   2018-01-29T15:51:23Z

IGNITE-7533: Throttle writing threads according fsync progress and 
checkpoint write speed

commit 4be02ec3444596ee0bc95bd55ece9ba741e729a1
Author: dpavlov 
Date:   2018-01-29T15:55:48Z

Merge branch 'master' into ignite-7533

commit f5d383ddef1b2a66470ec110f781a98aa78c5d03
Author: dpavlov 
Date:   2018-01-29T17:08:35Z

IGNITE-7533: Throttle writing threads according fsync progress and 
checkpoint write speed

commit 7c0afa374907202e45d6dcbfae88af1c3a27687f
Author: dpavlov 
Date:   2018-01-30T14:04:52Z

IGNITE-7533: Option to enable old implementation of throttling

commit 9f9c1e7955d894bbfd8a8572362d2c13177a60c6
Author: dpavlov 
Date:   2018-01-30T14:13:42Z

IGNITE-7380: Flaky test reproduction

commit 8d8aecd55a4d94c091e970ccf3bbbd72272cf325
Author: dpavlov 
Date:   2018-01-30T14:20:50Z

IGNITE-7380: Flaky test reproduction

commit cf9d42ba77133c4c6e37b1d8a7ac6d54054d1bc1
Author: dpavlov 
Date:   2018-01-30T14:40:26Z

IGNITE-7533: Preserve order of writing in fsync

commit b37f27275446a8cccafbd231c35e3605d9fd7089
Author: dpavlov 
Date:   2018-01-30T14:43:10Z

IGNITE-7380: Increase of timeout of checkpoint

commit b05ef5dae3d4e5deef6482844989077aba6f1bf2
Author: dpavlov 
Date:   2018-01-30T16:23:51Z

IGNITE-7533: Too much pages written case, no throttling in case too long 
wait. Added more delay in case low space left

commit 62685bcb363add269930af99e59b74d396870b55
Author: dpavlov 
Date:   2018-01-30T16:37:35Z

IGNITE-7380: Flaky test reproduction

commit c7ba24580199a238506ea00176aaf7ae229aa135
Author: dpavlov 
Date:   2018-01-30T17:57:15Z

IGNITE-7533: Sandbox test with progress gaps detection was added.

commit 6b111a2bef8c8afbea15d9a42114442aa2fc897d
Author: dpavlov 
Date:   2018-01-31T15:31:24Z

IGNITE-7584: WAL log throttle implementation




> Smooth decrease WAL write speed if low work segments left
> -
>
> Key: IGNITE-7584
> URL: https://issues.apache.org/jira/browse/IGNITE-7584
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Dmitriy Pavlov
>Assignee: Dmitriy Pavlov
>Priority: Major
>
> Smooth decrease of the write speed in WAL when approaching the exhaustion of 
> current segments in the working directory.
> Now archiver segment switch involves file copying and may be bottleneck of 
> speed of grid operation.
> Now if out of work segments occurs, grid put speed drop down to 0 ops/sec 
> ocurred.
> This isssue was fixed under IGNITE-7017 with providing new 'NoArchvier' mode, 
> but in case archiver based WAL is already used by end-user, it is required to 
> have opportunity to smooth decrease speed instead of drop to 0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7489) Weird FillFactor metric fluctuation.

[Ilya Kasnacheev (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347017#comment-16347017
 ] 

Ilya Kasnacheev commented on IGNITE-7489:
-

I don't know, I have tried your test and it seems to check for a very peculiar 
case where every object is of the same size. FreeList only contains one 
non-empty bucket in this case.

If you change item size to random, suddently you will only see load factor of 
0.9.

 

If you factor all allocated pages in, you only see load factor of 0.999 (this 
of course assumes that any non-data pages are full).

 

I just don't see how it would be a useful metric.

> Weird FillFactor metric fluctuation.
> 
>
> Key: IGNITE-7489
> URL: https://issues.apache.org/jira/browse/IGNITE-7489
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.1, 2.2, 2.3
>Reporter: Andrew Mashenkov
>Assignee: Ilya Kasnacheev
>Priority: Major
> Attachments: FillFactorTest.java
>
>
> For now there is no way to get Used\FreeMemory for region in bytes. 
> MemoryMetrics.getPagesFillFactor() javadoc says that the method return the 
> percentage of space that is still free and can be filled in.
> So, I'd think used memory can be calculated as 
> PhysicalMemoryPages*PageSize*FillFactor.
> I've tried to use this, but found weir thing.
>  
> PFA a repro.
> There is 2 node grid with no persistence configure. Topology is stable.
> Cache is populated with unique keys (no updates) and observe allocated data 
> pages metric grows constantly as expected.
> But the error look too large, used memory (and FillFactor as well) may 2-10+ 
> time differs.
> E.g. allocated pages, fillFactor, usedMem (bytes):(
>  node-0: 13789 0.851563 48096032
>  node-1: 14447 0.781250 46230400
> In next second:
> node-0: 13958 0.039063 2233280
>  node-1: 14624 0.347656 20824576
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7591) Backup node crash causes incorrect transaction failover recovery

[Alexey Kuznetsov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kuznetsov updated IGNITE-7591:
-
Summary: Backup node crash causes incorrect transaction failover recovery  
(was: Backup node crash causes incorrect failover recovery)

> Backup node crash causes incorrect transaction failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7591) Backup node crash causes incorrect failover recovery

[Alexey Kuznetsov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kuznetsov updated IGNITE-7591:
-
Attachment: (was: TxBackupFailoverTest.java)

> Backup node crash causes incorrect failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7591) Backup node crash causes incorrect failover recovery

[Alexey Kuznetsov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kuznetsov updated IGNITE-7591:
-
Attachment: TxBackupFailoverTest.java

> Backup node crash causes incorrect failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7592) Dynamic cache with rebalanceDelay == -1 doesn't trigger late affinity assignment even after explicit rebalance is called on every node

[Ilya Lantukh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Lantukh updated IGNITE-7592:
-
Description: 
Reproducer:
{noformat}
startGrids(NODE_COUNT);

IgniteEx ig = grid(0);

ig.cluster().active(true);

awaitPartitionMapExchange();

IgniteCache cache =
ig.createCache(
new CacheConfiguration()
.setName(CACHE_NAME)
.setCacheMode(PARTITIONED)
.setBackups(1)
.setPartitionLossPolicy(READ_ONLY_SAFE)
.setReadFromBackup(true)
.setWriteSynchronizationMode(FULL_SYNC)
.setRebalanceDelay(-1)
);

for (int i = 0; i < NODE_COUNT; i++)
grid(i).cache(CACHE_NAME).rebalance().get();

awaitPartitionMapExchange();
{noformat}

Sometimes this code will hang on the last awaitPartitionMapExchange(), though 
probability that it will happen is rather low (<10%).

  was:
Reproducer:
{noformat}
startGrids(NODE_COUNT);

IgniteEx ig = grid(0);

ig.cluster().active(true);

awaitPartitionMapExchange();

IgniteCache cache =
ig.createCache(
new CacheConfiguration()
.setName(CACHE_NAME)
.setCacheMode(PARTITIONED)
.setBackups(1)
.setPartitionLossPolicy(READ_ONLY_SAFE)
.setReadFromBackup(true)
.setWriteSynchronizationMode(FULL_SYNC)
.setRebalanceDelay(-1)
);

for (int i = 0; i < NODE_COUNT; i++)
grid(i).cache(CACHE_NAME).rebalance().get();

awaitPartitionMapExchange();
{noformat}

Sometimes this code will hang on awaitPartitionMapExchange(), though 
probability that it will happen is rather low (<10%).


> Dynamic cache with rebalanceDelay == -1 doesn't trigger late affinity 
> assignment even after explicit rebalance is called on every node
> --
>
> Key: IGNITE-7592
> URL: https://issues.apache.org/jira/browse/IGNITE-7592
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Ilya Lantukh
>Priority: Major
>
> Reproducer:
> {noformat}
> startGrids(NODE_COUNT);
> IgniteEx ig = grid(0);
> ig.cluster().active(true);
> awaitPartitionMapExchange();
> IgniteCache cache =
> ig.createCache(
> new CacheConfiguration()
> .setName(CACHE_NAME)
> .setCacheMode(PARTITIONED)
> .setBackups(1)
> .setPartitionLossPolicy(READ_ONLY_SAFE)
> .setReadFromBackup(true)
> .setWriteSynchronizationMode(FULL_SYNC)
> .setRebalanceDelay(-1)
> );
> for (int i = 0; i < NODE_COUNT; i++)
> grid(i).cache(CACHE_NAME).rebalance().get();
> awaitPartitionMapExchange();
> {noformat}
> Sometimes this code will hang on the last awaitPartitionMapExchange(), though 
> probability that it will happen is rather low (<10%).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7592) Dynamic cache with rebalanceDelay == -1 doesn't trigger late affinity assignment even after explicit rebalance is called on every node

[Ilya Lantukh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Lantukh updated IGNITE-7592:
-
Summary: Dynamic cache with rebalanceDelay == -1 doesn't trigger late 
affinity assignment even after explicit rebalance is called on every node  
(was: Cache with rebalanceDelay == -1 doesn't trigger late affinity assignment 
even after explicit rebalance is called on every node)

> Dynamic cache with rebalanceDelay == -1 doesn't trigger late affinity 
> assignment even after explicit rebalance is called on every node
> --
>
> Key: IGNITE-7592
> URL: https://issues.apache.org/jira/browse/IGNITE-7592
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Ilya Lantukh
>Priority: Major
>
> Reproducer:
> {noformat}
> startGrids(NODE_COUNT);
> IgniteEx ig = grid(0);
> ig.cluster().active(true);
> awaitPartitionMapExchange();
> IgniteCache cache =
> ig.createCache(
> new CacheConfiguration()
> .setName(CACHE_NAME)
> .setCacheMode(PARTITIONED)
> .setBackups(1)
> .setPartitionLossPolicy(READ_ONLY_SAFE)
> .setReadFromBackup(true)
> .setWriteSynchronizationMode(FULL_SYNC)
> .setRebalanceDelay(-1)
> );
> for (int i = 0; i < NODE_COUNT; i++)
> grid(i).cache(CACHE_NAME).rebalance().get();
> awaitPartitionMapExchange();
> {noformat}
> Sometimes this code will hang on awaitPartitionMapExchange(), though 
> probability that it will happen is rather low (<10%).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7591) Backup node crash causes incorrect failover recovery

[Alexey Kuznetsov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kuznetsov updated IGNITE-7591:
-
Description: 
reproduction:

Start multiple pessimistic transactions, changing the same key.Prepare message 
from backup node must be blocked(GridDhtTxPrepareResponse). Then backup node 
crashes.

All transactions must finish and keys must be committed.

But not all keys are committed. Reproducer attached.

Test is flaky.

  was:
reproduction:

Start multiple pessimistic transactions, changing the same key.Prepare message 
from backup node must be blocked(GridDhtTxPrepareResponse). Then backup node 
crashes.

All transactions must finish and keys must be committed.

But not all keys are committed. Reproducer attached.


> Backup node crash causes incorrect failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.
> Test is flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7592) Cache with rebalanceDelay == -1 doesn't trigger late affinity assignment even after explicit rebalance is called on every node

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-7592:


 Summary: Cache with rebalanceDelay == -1 doesn't trigger late 
affinity assignment even after explicit rebalance is called on every node
 Key: IGNITE-7592
 URL: https://issues.apache.org/jira/browse/IGNITE-7592
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4
Reporter: Ilya Lantukh


Reproducer:
{noformat}
startGrids(NODE_COUNT);

IgniteEx ig = grid(0);

ig.cluster().active(true);

awaitPartitionMapExchange();

IgniteCache cache =
ig.createCache(
new CacheConfiguration()
.setName(CACHE_NAME)
.setCacheMode(PARTITIONED)
.setBackups(1)
.setPartitionLossPolicy(READ_ONLY_SAFE)
.setReadFromBackup(true)
.setWriteSynchronizationMode(FULL_SYNC)
.setRebalanceDelay(-1)
);

for (int i = 0; i < NODE_COUNT; i++)
grid(i).cache(CACHE_NAME).rebalance().get();

awaitPartitionMapExchange();
{noformat}

Sometimes this code will hang on awaitPartitionMapExchange(), though 
probability that it will happen is rather low (<10%).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7591) Backup node crash causes incorrect failover

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-7591:


 Summary: Backup node crash causes incorrect failover
 Key: IGNITE-7591
 URL: https://issues.apache.org/jira/browse/IGNITE-7591
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Kuznetsov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7591) Backup node crash causes incorrect failover recovery

[Alexey Kuznetsov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kuznetsov updated IGNITE-7591:
-
 Attachment: TxBackupFailoverTest.java
Description: 
reproduction:

Start multiple pessimistic transactions, changing the same key.Prepare message 
from backup node must be blocked(GridDhtTxPrepareResponse). Then backup node 
crashes.

All transactions must finish and keys must be committed.

But not all keys are committed. Reproducer attached.
Summary: Backup node crash causes incorrect failover recovery  (was: 
Backup node crash causes incorrect failover)

> Backup node crash causes incorrect failover recovery
> 
>
> Key: IGNITE-7591
> URL: https://issues.apache.org/jira/browse/IGNITE-7591
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexey Kuznetsov
>Priority: Major
> Attachments: TxBackupFailoverTest.java
>
>
> reproduction:
> Start multiple pessimistic transactions, changing the same key.Prepare 
> message from backup node must be blocked(GridDhtTxPrepareResponse). Then 
> backup node crashes.
> All transactions must finish and keys must be committed.
> But not all keys are committed. Reproducer attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7561) .NET: Service proxy as dynamic object

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-7561:
---
Fix Version/s: 2.5

> .NET: Service proxy as dynamic object
> -
>
> Key: IGNITE-7561
> URL: https://issues.apache.org/jira/browse/IGNITE-7561
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
> Fix For: 2.5
>
>
> Currently we require user to provide an interface to interact with .NET and 
> Java services.
> We could also provide a dynamic API (based on {{dynamic}} keyword) like this:
> {code}
> IServices services = ignite.GetServices();
> dynamic prx = services.GetDynamicServiceProxy("someService");
> prx.Foo();
> prx.Bar = 1;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-7561) .NET: Service proxy as dynamic object

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346948#comment-16346948
 ] 

Pavel Tupitsyn commented on IGNITE-7561:


Done, waiting for TC: 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8_IgniteTests24Java8=pull%2F3457%2Fhead

> .NET: Service proxy as dynamic object
> -
>
> Key: IGNITE-7561
> URL: https://issues.apache.org/jira/browse/IGNITE-7561
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
> Fix For: 2.5
>
>
> Currently we require user to provide an interface to interact with .NET and 
> Java services.
> We could also provide a dynamic API (based on {{dynamic}} keyword) like this:
> {code}
> IServices services = ignite.GetServices();
> dynamic prx = services.GetDynamicServiceProxy("someService");
> prx.Foo();
> prx.Bar = 1;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-7561) .NET: Service proxy as dynamic object

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346943#comment-16346943
 ] 

Pavel Tupitsyn edited comment on IGNITE-7561 at 1/31/18 2:53 PM:
-

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/3457

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

Work with service proxies in dynamic fashion

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-7561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3457


was (Author: githubbot):
GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/3457

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

Work with service proxies in dynamic fashion

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-7561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3457


commit f5b7cfcea7183eff9921f7c94da9b673ea87fcab
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:34:48Z

IGNITE-7561 .NET: Service proxy as dynamic object

commit 59283989e6827db7148b3b80a8ec274a2f7f11d1
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:36:15Z

wip

commit 093774e6cc1473668fe2359e149a7220391cfa31
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:45:19Z

wip

commit 71616f76b4638d2f07b1ce56cda7cb4ba2ce99f8
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:46:21Z

wip

commit 6b5bc3d992e36220b7e6207dc26a03eb5e58fa60
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:52:18Z

add initial test

commit a261ef0aaa5f03687c5b9296779cac8017ddde86
Author: Pavel Tupitsyn 
Date:   2018-01-31T11:26:27Z

wip

commit 2bc60aa0ef22b2ce270da388d989207f861c5597
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:10:11Z

Refactoring proxy serializer

commit 861a5b4dfa679aa542b14845f6ead74603f15216
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:15:38Z

wip

commit 070afc45ed2be645fb5ed8436227d19d191f9855
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:17:14Z

implementing property getter

commit 1d8bb5f62f340f25bc296ead951b1e96485c9e52
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:18:23Z

wip

commit 57bb354eacdc829383cfcc5ffe306d1e2079765e
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:21:42Z

fix NPE

commit 6884bcaf6236324cd99e6f165b63c8ac46af8032
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:28:53Z

add getter prefix

commit b68b18e5e57177369c12d7e887823f9a5e48b8b9
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:30:27Z

First test works!!

commit 81d2b99a811a54769aa1d729a3225fe09dd7720d
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:32:26Z

wip tests

commit 4dbf58444f3dd70b44752a8db1bb9cb51e3e7cd8
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:38:30Z

Dynamic proxy implemented

commit 82900f6639b1d6020f2b9cd135a2600844f47b7e
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:42:13Z

Fix ToString

commit cee63d8ec3938c2060c5d600951cf263b64b42b8
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:45:53Z

wip

commit 299a0ce341c74aadc82623728bfe3a2ce7d9d25d
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:48:25Z

wip

commit c3bfdf78af7fe20af744f6b7fb3664c97ab8678a
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:56:44Z

wip tests

commit ba77ad4ff28aa95d312e71aec83955e0c17ba842
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:12:36Z

wip tests

commit ed1bad372b67d01237c6274596f80af12a8f979b
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:42:01Z

TestCallJavaServiceDynamicProxy

commit 858e49b56cf59e1ffa0a7cf3cbe8a4a2c0e22379
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:43:51Z

TestCallJavaServiceDynamicProxy done

commit 9f017b8990bf509f2bc4e5f9717800d578df2c46
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:47:01Z

All tests done

commit 65955ba37db37f3a289ad11522e31b2c72dfe733
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:49:10Z

Add overload without sticky




> .NET: Service proxy as dynamic object
> -
>
> Key: IGNITE-7561
> URL: 

[jira] [Commented] (IGNITE-7561) .NET: Service proxy as dynamic object

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346943#comment-16346943
 ] 

ASF GitHub Bot commented on IGNITE-7561:


GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/3457

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

Work with service proxies in dynamic fashion

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-7561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3457


commit f5b7cfcea7183eff9921f7c94da9b673ea87fcab
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:34:48Z

IGNITE-7561 .NET: Service proxy as dynamic object

commit 59283989e6827db7148b3b80a8ec274a2f7f11d1
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:36:15Z

wip

commit 093774e6cc1473668fe2359e149a7220391cfa31
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:45:19Z

wip

commit 71616f76b4638d2f07b1ce56cda7cb4ba2ce99f8
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:46:21Z

wip

commit 6b5bc3d992e36220b7e6207dc26a03eb5e58fa60
Author: Pavel Tupitsyn 
Date:   2018-01-31T10:52:18Z

add initial test

commit a261ef0aaa5f03687c5b9296779cac8017ddde86
Author: Pavel Tupitsyn 
Date:   2018-01-31T11:26:27Z

wip

commit 2bc60aa0ef22b2ce270da388d989207f861c5597
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:10:11Z

Refactoring proxy serializer

commit 861a5b4dfa679aa542b14845f6ead74603f15216
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:15:38Z

wip

commit 070afc45ed2be645fb5ed8436227d19d191f9855
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:17:14Z

implementing property getter

commit 1d8bb5f62f340f25bc296ead951b1e96485c9e52
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:18:23Z

wip

commit 57bb354eacdc829383cfcc5ffe306d1e2079765e
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:21:42Z

fix NPE

commit 6884bcaf6236324cd99e6f165b63c8ac46af8032
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:28:53Z

add getter prefix

commit b68b18e5e57177369c12d7e887823f9a5e48b8b9
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:30:27Z

First test works!!

commit 81d2b99a811a54769aa1d729a3225fe09dd7720d
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:32:26Z

wip tests

commit 4dbf58444f3dd70b44752a8db1bb9cb51e3e7cd8
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:38:30Z

Dynamic proxy implemented

commit 82900f6639b1d6020f2b9cd135a2600844f47b7e
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:42:13Z

Fix ToString

commit cee63d8ec3938c2060c5d600951cf263b64b42b8
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:45:53Z

wip

commit 299a0ce341c74aadc82623728bfe3a2ce7d9d25d
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:48:25Z

wip

commit c3bfdf78af7fe20af744f6b7fb3664c97ab8678a
Author: Pavel Tupitsyn 
Date:   2018-01-31T13:56:44Z

wip tests

commit ba77ad4ff28aa95d312e71aec83955e0c17ba842
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:12:36Z

wip tests

commit ed1bad372b67d01237c6274596f80af12a8f979b
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:42:01Z

TestCallJavaServiceDynamicProxy

commit 858e49b56cf59e1ffa0a7cf3cbe8a4a2c0e22379
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:43:51Z

TestCallJavaServiceDynamicProxy done

commit 9f017b8990bf509f2bc4e5f9717800d578df2c46
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:47:01Z

All tests done

commit 65955ba37db37f3a289ad11522e31b2c72dfe733
Author: Pavel Tupitsyn 
Date:   2018-01-31T14:49:10Z

Add overload without sticky




> .NET: Service proxy as dynamic object
> -
>
> Key: IGNITE-7561
> URL: https://issues.apache.org/jira/browse/IGNITE-7561
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
>
> Currently we require user to provide an interface to interact with .NET and 
> Java services.
> We could also provide a dynamic API (based on {{dynamic}} keyword) like this:
> {code}
> IServices services = ignite.GetServices();
> dynamic prx = services.GetDynamicServiceProxy("someService");
> prx.Foo();
> prx.Bar = 1;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345541#comment-16345541
 ] 

Alexander Paschenko edited comment on IGNITE-6917 at 1/31/18 2:47 PM:
--

[~kirill.shirokov], my comments:

1. {{JdbcAbstractUpdateStatementSelfTest}} - unused code

2. {{org.apache.ignite.internal.jdbc2.JdbcBulkLoadSelfTest: createConnection}} 
- unused param; {{testBulkLoadThrows}} - please use try-with-resources.

3. {{org.apache.ignite.jdbc.thin.JdbcThinAbstractDmlStatementSelfTest:}}

3.1 {{jdbcRun}}: please remove this method and use inherited method execute 
instead.

3.2 {{testDOA}}: please rename into something more meaningful.

4. {{BulkLoadContextCursor:}}

4.1 Please make field final.

4.2 {{getAll}} - I changed the code to double call of {{singletonList}} just 
fine.

4.3 {{getFieldName}} - I believe this method should throw 
{{NoSuchElementException}} for all {{i}} except {{0}}.

4.4 Odd empty line right after class header.

5. {{SqlBulkLoadCommand}}: let's parse command in a way consistent with how 
it's done in SQL usually - we clearly should not use ugly stuff like 
{{BATCH_SIZE}} (with underscore instead of space) now that we are headed to our 
own parser. Please rework parsing so that user has to write {{BATCH SIZE}}. 
(NB: yes, I'm aware that we already have quite ugly stuff here, like 
{{INLINE_SIZE}} for indexes, but I believe that this must be avoided as much as 
possible.) -Also I think we should support case when the user does not supply 
columns list (so no empty brackets too) - currently it's a parsing error and 
there's a test for it.- _(Please disregard what's been struck through.)_

6. {{UpdatePlan#processRow}}:

6.1 Odd empty line after method declaration.

6.2. Shouldn't we rather check for equality here? {{row.size() < 
colNames.length}} - like this

7. {{UpdatePlanBuilder#planForBulkLoad}}:

7.1 Odd empty line after method declaration.

7.2 Let's add VERY simple test case (like cache of  and bulk 
load to it) - I think this may break on current implementation as we may 
actually will to insert into _val column, and this case does not seem to be 
handled - say, like this {{COPY from file to String () format csv}} (String is 
table name, table has 3 columns all of which are system columns.)

 

This is not all, more comments will follow.

 

 


was (Author: al.psc):
[~kirill.shirokov], my comments:

1. {{JdbcAbstractUpdateStatementSelfTest}} - unused code

2. {{org.apache.ignite.internal.jdbc2.JdbcBulkLoadSelfTest: createConnection}} 
- unused param; {{testBulkLoadThrows}} - please use try-with-resources.

3. {{org.apache.ignite.jdbc.thin.JdbcThinAbstractDmlStatementSelfTest:}}

3.1 {{jdbcRun}}: please remove this method and use inherited method execute 
instead.

3.2 {{testDOA}}: please rename into something more meaningful.

4. {{BulkLoadContextCursor:}}

4.1 Please make field final.

4.2 {{getAll}} - I changed the code to double call of {{singletonList}} just 
fine.

4.3 {{getFieldName}} - I believe this method should throw 
{{NoSuchElementException}} for all {{i}} except {{0}}.

4.4 Odd empty line right after class header.

5. {{SqlBulkLoadCommand}}: let's parse command in a way consistent with how 
it's done in SQL usually - we clearly should not use ugly stuff like 
{{BATCH_SIZE}} (with underscore instead of space) now that we are headed to our 
own parser. Please rework parsing so that user has to write {{BATCH SIZE}}. 
(NB: yes, I'm aware that we already have quite ugly stuff here, like 
{{INLINE_SIZE}} for indexes, but I believe that this must be avoided as much as 
possible.) Also I think we should support case when the user does not supply 
columns list (so no empty brackets too) - currently it's a parsing error and 
there's a test for it.

6. {{UpdatePlan#processRow}}:

6.1 Odd empty line after method declaration.

6.2. Shouldn't we rather check for equality here? {{row.size() < 
colNames.length}} - like this

7. {{UpdatePlanBuilder#planForBulkLoad}}:

7.1 Odd empty line after method declaration.

7.2 Let's add VERY simple test case (like cache of  and bulk 
load to it) - I think this may break on current implementation as we may 
actually will to insert into _val column, and this case does not seem to be 
handled - say, like this {{COPY from file to String () format csv}} (String is 
table name, table has 3 columns all of which are system columns.)

 

This is not all, more comments will follow.

 

 

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>

[jira] [Assigned] (IGNITE-5427) Add cluster activation/deactivation events

[Evgenii Zhuravlev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgenii Zhuravlev reassigned IGNITE-5427:
-

Assignee: Evgenii Zhuravlev

> Add cluster activation/deactivation events
> --
>
> Key: IGNITE-5427
> URL: https://issues.apache.org/jira/browse/IGNITE-5427
> Project: Ignite
>  Issue Type: Improvement
>  Components: general
>Affects Versions: 2.0
>Reporter: Alexey Goncharuk
>Assignee: Evgenii Zhuravlev
>Priority: Major
> Fix For: 2.5
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7590) Change example for decision trees

[Artem Malykh (JIRA)]

Artem Malykh created IGNITE-7590:


 Summary: Change example for decision trees 
 Key: IGNITE-7590
 URL: https://issues.apache.org/jira/browse/IGNITE-7590
 Project: Ignite
  Issue Type: Bug
  Components: ml
Reporter: Artem Malykh
Assignee: Artem Malykh
 Fix For: 2.4


Change example for decision trees according to comments from [~dmagda] at 
https://issues.apache.org/jira/browse/IGNITE-7504 .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7590) Change example for decision trees

[Artem Malykh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Malykh updated IGNITE-7590:
-
Issue Type: Improvement  (was: Bug)

> Change example for decision trees 
> --
>
> Key: IGNITE-7590
> URL: https://issues.apache.org/jira/browse/IGNITE-7590
> Project: Ignite
>  Issue Type: Improvement
>  Components: ml
>Reporter: Artem Malykh
>Assignee: Artem Malykh
>Priority: Major
> Fix For: 2.4
>
>
> Change example for decision trees according to comments from [~dmagda] at 
> https://issues.apache.org/jira/browse/IGNITE-7504 .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7429) Thin client Java API - cache put/get/contains

[Alexey Kukushkin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kukushkin reassigned IGNITE-7429:


Assignee: Alexey Kukushkin

> Thin client Java API - cache put/get/contains
> -
>
> Key: IGNITE-7429
> URL: https://issues.apache.org/jira/browse/IGNITE-7429
> Project: Ignite
>  Issue Type: Sub-task
> Environment: Implement cache put/get/contains thin client Java API 
> including unit and system tests and samples.
> Cache
>        put(key, val)
>         get(key): V
>         containsKey(key): boolean
>  
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
>  Labels: data, java, thin
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7423) Thin client Java API - cache management

[Alexey Kukushkin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Kukushkin reassigned IGNITE-7423:


Assignee: Alexey Kukushkin

> Thin client Java API - cache management
> ---
>
> Key: IGNITE-7423
> URL: https://issues.apache.org/jira/browse/IGNITE-7423
> Project: Ignite
>  Issue Type: Sub-task
> Environment: Implement get/create/destroy cache thin client Java API 
> including unit and system tests and samples.
> Ignite
>     cache(name: String)
>     createCache(name): Cache
>     getOrCreateCache(name): Cache
>     destroyCache(name)
>Reporter: Alexey Kukushkin
>Assignee: Alexey Kukushkin
>Priority: Major
>  Labels: data, java, thin
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (IGNITE-7566) .NET: Build scripts: stop build when one of the steps fails

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn reassigned IGNITE-7566:
--

Assignee: Pavel Tupitsyn

> .NET: Build scripts: stop build when one of the steps fails
> ---
>
> Key: IGNITE-7566
> URL: https://issues.apache.org/jira/browse/IGNITE-7566
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
> Fix For: 2.5
>
>
> We have multiple build scripts for various platforms:
> * {{build.bat}} / {{build.ps1}} (Windows)
> * {{build.sh}} (.NET Core Linux / macOS)
> * {{build-mono.sh}} (Mono Linux / macOS)
> Make sure that we stop build scripts execution when one of the steps fails.
> For example, if Java build fails, we should show an error and stop instead of 
> continuing to build .NET.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IGNITE-7573) IgniteOfflineBaselineNodeFullApiSelfTest.testSize fails

[Alexey Goncharuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Goncharuk resolved IGNITE-7573.
--
Resolution: Fixed

Merged to master

> IgniteOfflineBaselineNodeFullApiSelfTest.testSize fails
> ---
>
> Key: IGNITE-7573
> URL: https://issues.apache.org/jira/browse/IGNITE-7573
> Project: Ignite
>  Issue Type: Bug
>  Components: persistence
>Affects Versions: 2.4
>Reporter: Alexey Goncharuk
>Assignee: Alexey Goncharuk
>Priority: Major
> Fix For: 2.5
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Kirill Shirokov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Shirokov updated IGNITE-6917:

Description: 
Inspired by Postgres [1]

Common use case - bulk data load through JDBC/ODBC interface. Currently it is 
only possible to execute single commands one by one. We already can batch them 
to improve performance, but there is still big room for improvement.

We should think of a completely new command - {{COPY}}. It will accept a file 
(or input stream in general case) on the client side, then transfer data to the 
cluster, and then execute update inside the cluster, e.g. through streamer.

First of all we need to create quick and dirty prototype to assess potential 
performance improvement. It speedup is confirmed, we should build base 
implementation which will accept only files. But at the same time we should 
understand how it will evolve in future: multiple file formats (probably 
including Hadoop formarts, e.g. Parquet), escape characters, input streams, 
etc..

[1] [https://www.postgresql.org/docs/9.6/static/sql-copy.html]
h1. Proposed syntax

Curent implementation:
{noformat}
COPY 
FROM "file.name"
INTO .
[(col-name, ...)]
FORMAT  -- Only CSV format is supported in the current 
release
[BATCH_SIZE ]
{noformat}
We may want to gradually add features to this command in future to have 
something like this:
{noformat}
COPY
FROM "file.name"[CHARSET ""]
INTO . [CREATE [IF NOT EXISTS]]
[(col-name [] [NULLABLE] [ESCAPES], ...) [MATCH HEADER]]
FORMAT (csv|tsv|...)

-- CSV format options:
[FIELDSEP='column-separators-regexp']
[LINESEP='row-separators-regexp']
[QUOTE='quote-chars']
[ESCAPE='escape-char']
[NULL='null-sequence']
[COMMENT='single-line-comment-start-char']
[TRIM_LINES]
[IMPORT_EMPTY_LINES]
[CHARSET ""]

[ROWS -]
--or--
[SKIP ROWS ] [MAX ROWS ]

[COLS -]
--or--
[SKIP COLS ] [MAX COLS ]

[(MATCH | SKIP) HEADER]
[(REPLACE|IGNORE|ABORT ON [])) DUPLICATE KEYS]
[BATCH SIZE ( ROWS | [K|M|G|T|P])]
[COMPRESS "codec-name" [codec options]]
[LOCK (TABLE|ROWS)]
[NOLOGGING]
[BACKEND (DIRECT | STREAMER)]
{noformat}

h1. Implementation decisions and notes

h2. Parsing

* We support CSV format described in RFC 4180.
* Custom row and column separators, quoting characters are currently hardcoded
* Escape sequences, line comment characters are currently not supported
* We may want to support fixed-length formats (via format descriptors) in future
* We may want to strip comments from lines (for example, starting with '#')
* We may want to allow user to either ignore empty lines or treat them as a 
special case of record having all default values
* We may allow user to enable whitespace trimming from beginning and end of a 
line
* We may want to allow user to specify error handling strategy: e.g., only one 
quote character is present or escape sequence is invalid.

h2. File handling

* File character set to be supported in future
* Skipped/imported row number (or first/last line or skip header option), 
skipped/imported column number (or first/last column): to be supported in future
* Line start pattern (as in MySQL): no support planned
* We currently support only client-side import. No server-side file import.
* We may want to support client-side stdin import in future.
* We do not handle importing multiple files from single command
* We don't benefit from any kind of pre-sorting pre-partitioning data on client 
side.
* We don't include any any metadata, such as line number from client side.

h3. Transferring data

* We send file data via batches. In future we will support batch size 
(specified with rows per batch or data block size 
per batch).
* We may want to implement data compression in future.
* We connect to single node in JDBC driver (no multi-node connections).

h3. Cache/tables/column handling

* We don't create table in the bulk load command
* We may want to have and option for reading header row, which contains column 
names to match columns
* In future we may wish to support COLUMNS (col1, _, col2, _, col3) syntax, 
where '_' marker means a skipped column (MySQL uses '@dummy' for this)

h3. Data types

* Data types are converted as if they were supplied to INSERT SQL command.
* We may want type conversion (automatic, custom using sql function, custom via 
Java code, string auto-trimming) in future.
* We will support optional null sequence ("\N") later
* We may want to allow user to specify what to do if the same record exists 
(e.g., ignore record, replace it, report error with a max. error counter before 
failing the command)
* We don't currently support any generated/autoincremented row IDs or any 
custom generators.
* We don't support any filtering/conditional expressions
* We don't support any files/recordsets/tables with multiple conversion errors 
generated during 

[jira] [Comment Edited] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346774#comment-16346774
 ] 

Alexander Paschenko edited comment on IGNITE-6917 at 1/31/18 1:13 PM:
--

[~kirill.shirokov], please let me continue with comments.

1. {{BulkLoadParameters}} - do we really need this class? It just has two 
fields. For now I see no point in it. Do you expect many more new fields to 
come?

2. {{BulkLoadParser}}:

2.1 Unused private field.

2.2 Too long line in {{processBatch}} declaration.

2.3 It seems plain wrong that arguments of {{processBatch}} are related with 
JDBC. JDBC is a connectivity interface, nothing else, and these references are 
in deeper "guts". We need to decouple these somehow - one day we might work to 
support this feature in ODBC, or somewhere else, and then it'd be inevitable. 
I'd just introduce an enum for various command types.

3. {{BulkLoadFormat}}:

3.1 Unused import

3.2 Odd empty lines in the beginning of class, in switch, etc

4. {{BulkLoadParser}} and {{BulkLoadFormat}} seem to be related 1:1 - probably 
they should be single class and implementations could be merged accordingly 
too. Can you come up with a case where we would really need to separate this 
entities? Say, one parser for few formats, or one format yielding more than one 
type of parser?

5. {{BulkLoadCsvParser}}:

{{5.1 processBatch}} method looks like it could be actually defined in parent 
class, and the logic to be changed in subclasses actually lives in 
{{parseBatch}}. This is so because command types like {{CMD_FINISHED_EOF}} are 
defined on higher level. Can we rework hierarchy accordingly?

5.2 Unused parameter in ctor ( {{params}} ), do we have tests for params at all?

5.3 Odd empty lines after class header, method declarations

6. {{BulkLoadEntryConverter}}: do we really expect more than one implementation 
of this interface? If no, let's just replace it with {{IgniteClosureX}} or 
something like that. We have more than enough interfaces and abstract classes 
to represent lambdas. *Before refactoring this, please look at pt.10.3 as well*

7. {{BulkLoadCsvFormat}}: amount of useful code in this class approaches zero, 
and this further assures that parser and its format must be one.

8. {{BulkLoadContext}}: too long comment line.

9. {{BulkLoadCacheWriter}}: do we expect more than one implementation of this 
interface?quit If no, just remove this class and let {{BulkLoadContext}} know 
about streamer. If yes, make it inherit one of Ignite's provided 
classes/interfaces for lambdas. *Before refactoring this, please look at 
pt.10.3 as well*

10. {{JdbcRequestHandler}}:

10.1 Unused import

10.2 {{processBulkLoadFileBatch}}: all other methods just catch {{Exception}}, 
please do the same here.

10.3 {{processBulkLoadFileBatch}}: it looks like doing actual job (data 
conversion, feeding to streamer) is just plain wrong. Not only it's logic that 
could be reused when we'll implement same feature for other interface like 
JDBC, it's also not the area of responsibility of JDBC request handler. This 
again shows the need of refactoring of {{BulkLoadContext}}: probably we need to 
introduce a concrete class that will *a)* wrap streamer and *b)* do conversion. 
It will just accept {{List}} and do with it whatever it wants. JDBC 
must not know about this at all, it must just detect correct context and feed 
lists of objects to that new class I've mentioned above.

10.4 {{executeQuery}}: I'd move whole stuff of {{if (fieldsCur instanceof 
BulkLoadContextCursor)}} above, before {{if (results.size() == 1)}} - it would 
decrease scope depth, make code better readable and decrease overall size of 
diff (never hurts).

10.5 {{executeQuery}}: {{filesToSendResult}} - doesn't have anything at all to 
do with files.

11. {{JdbcBulkLoadContext}}: unused field {{cmdParsingResult}} (getter has 0 
usages), not to mention that field name is very misleading - what does JDBC 
have to do with parsing?

12. {{JdbcBulkLoadBatchRequestResult}}: quite weird name, I don't think we 
should mix words "request" and "result", we never do so in JDBC. Not to mention 
that we never send {{JdbcBulkLoadBatchRequest}} *before* we get 
{{JdbcBulkLoadBatchRequestResult}} :) I'd just rename this to 
{{JdbcBulkLoadAckResult}} as it's sent as an acknowledgement of {{COPY}} 
command and never else.

This is not all, more comments will follow.


was (Author: al.psc):
[~kirill.shirokov], please let me continue with comments.

1. {{BulkLoadParameters}} - do we really need this class? It just has two 
fields. For now I see no point in it. Do you expect many more new fields to 
come?

2. {{BulkLoadParser}}:

2.1 Unused private field.

2.2 Too long line in {{processBatch}} declaration.

2.3 It seems plain wrong that arguments of {{processBatch}} are related with 
JDBC. JDBC is a connectivity interface, nothing else, and these 

[jira] [Created] (IGNITE-7589) Grid hangs on partition map exhcange during failover test with 34 physical caches (PDS + in memory)

[Ksenia Rybakova (JIRA)]

Ksenia Rybakova created IGNITE-7589:
---

 Summary: Grid hangs on partition map exhcange during failover test 
with 34 physical caches (PDS + in memory)
 Key: IGNITE-7589
 URL: https://issues.apache.org/jira/browse/IGNITE-7589
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4
Reporter: Ksenia Rybakova


Grid hangs on partition map exhcange during failover test with 34 physical 
caches (PDS + in memory) during preloading (streaming) phase. Several restarts 
completed successfully, but then grid hung after node joined.

Load config:
CacheRandomOperationBenchmark
11 client nodes, 22 server nodes
250K perloading, 500K key range
34 physical caches of different types (onheap/offheap, tx/atomic, 
partitioned/replicated, with/without eviction, pds/in memory)
2 backups

2 server nodes are being restarted periodically

 

Complete configs are attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IGNITE-7589) Grid hangs on partition map exhcange during failover test with 34 physical caches (PDS + in memory)

[Ksenia Rybakova (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ksenia Rybakova updated IGNITE-7589:

Attachment: run-load.xml
run-load.properties
ignite-base-load-config.xml

> Grid hangs on partition map exhcange during failover test with 34 physical 
> caches (PDS + in memory)
> ---
>
> Key: IGNITE-7589
> URL: https://issues.apache.org/jira/browse/IGNITE-7589
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 2.4
>Reporter: Ksenia Rybakova
>Priority: Major
> Attachments: ignite-base-load-config.xml, run-load.properties, 
> run-load.xml
>
>
> Grid hangs on partition map exhcange during failover test with 34 physical 
> caches (PDS + in memory) during preloading (streaming) phase. Several 
> restarts completed successfully, but then grid hung after node joined.
> Load config:
> CacheRandomOperationBenchmark
> 11 client nodes, 22 server nodes
> 250K perloading, 500K key range
> 34 physical caches of different types (onheap/offheap, tx/atomic, 
> partitioned/replicated, with/without eviction, pds/in memory)
> 2 backups
> 2 server nodes are being restarted periodically
>  
> Complete configs are attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7588) Deprecate CacheLocalStore annotation

[Vyacheslav Daradur (JIRA)]

Vyacheslav Daradur created IGNITE-7588:
--

 Summary: Deprecate CacheLocalStore annotation
 Key: IGNITE-7588
 URL: https://issues.apache.org/jira/browse/IGNITE-7588
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 2.3, 2.4
Reporter: Vyacheslav Daradur
Assignee: Vyacheslav Daradur
 Fix For: 2.5


We should annotate @CacheLocalStore as @Depricated, because of using 
CacheLocalStore annotation has the hidden issues like rebalancing and possible 
data consistency issues.

See [dev-list 
discussion|http://apache-ignite-developers.2346864.n4.nabble.com/Should-we-annotate-CacheLocalStore-as-Depricated-tt26490.html]
 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-6917) SQL: implement COPY command for efficient data loading

[Alexander Paschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346774#comment-16346774
 ] 

Alexander Paschenko commented on IGNITE-6917:
-

[~kirill.shirokov], please let me continue with comments.

1. {{BulkLoadParameters}} - do we really need this class? It just has two 
fields. For now I see no point in it. Do you expect many more new fields to 
come?

2. {{BulkLoadParser}}:

2.1 Unused private field.

2.2 Too long line in {{processBatch}} declaration.

2.3 It seems plain wrong that arguments of {{processBatch}} are related with 
JDBC. JDBC is a connectivity interface, nothing else, and these references are 
in deeper "guts". We need to decouple these somehow - one day we might work to 
support this feature in ODBC, or somewhere else, and then it'd be inevitable. 
I'd just introduce an enum for various command types.

3. {{BulkLoadFormat}}:

3.1 Unused import

3.2 Odd empty lines in the beginning of class, in switch, etc

4. {{BulkLoadParser}} and {{BulkLoadFormat}} seem to be related 1:1 - probably 
they should be single class and implementations could be merged accordingly 
too. Can you come up with a case where we would really need to separate this 
entities? Say, one parser for few formats, or one format yielding more than one 
type of parser?

5. {{BulkLoadCsvParser}}:

{{5.1 processBatch}} method looks like it could be actually defined in parent 
class, and the logic to be changed in subclasses actually lives in 
{{parseBatch}}. This is so because command types like {{CMD_FINISHED_EOF }}are 
defined on higher level. Can we rework hierarchy accordingly?

5.2 Unused parameter in ctor ( {{params}} ), do we have tests for params at all?

5.3 Odd empty lines after class header, method declarations

6. {{BulkLoadEntryConverter}}: do we really expect more than one implementation 
of this interface? If no, let's just replace it with {{IgniteClosureX}} or 
something like that. We have more than enough interfaces and abstract classes 
to represent lambdas. *Before refactoring this, please look at pt.10.3 as well*

7. {{BulkLoadCsvFormat}}: amount of useful code in this class approaches zero, 
and this further assures that parser and its format must be one.

8. {{BulkLoadContext}}: too long comment line.

9. {{BulkLoadCacheWriter}}: do we expect more than one implementation of this 
interface?quit If no, just remove this class and let {{BulkLoadContext}} know 
about streamer. If yes, make it inherit one of Ignite's provided 
classes/interfaces for lambdas. *Before refactoring this, please look at 
pt.10.3 as well*

10. {{JdbcRequestHandler}}:

10.1 Unused import

10.2 {{processBulkLoadFileBatch}}: all other methods just catch {{Exception}}, 
please do the same here.

10.3 {{processBulkLoadFileBatch}}: it looks like doing actual job (data 
conversion, feeding to streamer) is just plain wrong. Not only it's logic that 
could be reused when we'll implement same feature for other interface like 
JDBC, it's also not the area of responsibility of JDBC request handler. This 
again shows the need of refactoring of {{BulkLoadContext}}: probably we need to 
introduce a concrete class that will *a)* wrap streamer and *b)* do conversion. 
It will just accept {{List}} and do with it whatever it wants. JDBC 
must not know about this at all, it must just detect correct context and feed 
lists of objects to that new class I've mentioned above.

10.4 {{executeQuery}}: I'd move whole stuff of {{if (fieldsCur instanceof 
BulkLoadContextCursor)}} above, before {{if (results.size() == 1)}} - it would 
decrease scope depth, make code better readable and decrease overall size of 
diff (never hurts).

10.5 {{executeQuery}}: {{filesToSendResult}} - doesn't have anything at all to 
do with files.

11. {{JdbcBulkLoadContext}}: unused field {{cmdParsingResult}} (getter has 0 
usages), not to mention that field name is very misleading - what does JDBC 
have to do with parsing?

12. {{JdbcBulkLoadBatchRequestResult}}: quite weird name, I don't think we 
should mix words "request" and "result", we never do so in JDBC. Not to mention 
that we never send {{JdbcBulkLoadBatchRequest}} *before* we get 
{{JdbcBulkLoadBatchRequestResult}} :) I'd just rename this to 
{{JdbcBulkLoadAckResult}} as it's sent as an acknowledgement of {{COPY}} 
command and never else.

This is not all, more comments will follow.

> SQL: implement COPY command for efficient data loading
> --
>
> Key: IGNITE-6917
> URL: https://issues.apache.org/jira/browse/IGNITE-6917
> Project: Ignite
>  Issue Type: New Feature
>  Components: sql
>Reporter: Vladimir Ozerov
>Assignee: Kirill Shirokov
>Priority: Major
>  Labels: iep-1
>
> Inspired by Postgres [1]
> Common use case - bulk data load through JDBC/ODBC 

[jira] [Created] (IGNITE-7587) SQL COPY: document the command

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7587:
---

 Summary: SQL COPY: document the command
 Key: IGNITE-7587
 URL: https://issues.apache.org/jira/browse/IGNITE-7587
 Project: Ignite
  Issue Type: Improvement
  Components: documentation, sql
Affects Versions: 2.4
Reporter: Kirill Shirokov


SQL COPY command needs to be documented at readme.io.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7586) SQL COPY: add code examples

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7586:
---

 Summary: SQL COPY: add code examples
 Key: IGNITE-7586
 URL: https://issues.apache.org/jira/browse/IGNITE-7586
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.4
Reporter: Kirill Shirokov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)