[jira] [Commented] (IGNITE-19222) .NET: Add IgniteClientConfiguration.EnableClusterDiscovery
[ https://issues.apache.org/jira/browse/IGNITE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711188#comment-17711188 ] Pavel Tupitsyn commented on IGNITE-19222: - Cherry-picked to ignite-2.15: eab71555b3a1eefd096b42af52618487452e85f9 > .NET: Add IgniteClientConfiguration.EnableClusterDiscovery > -- > > Key: IGNITE-19222 > URL: https://issues.apache.org/jira/browse/IGNITE-19222 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Affects Versions: 2.14 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.15 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Cluster discovery is always enabled currently, which can be undesirable in > some use cases. Add a config property to disable discovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19222) .NET: Add IgniteClientConfiguration.EnableClusterDiscovery
[ https://issues.apache.org/jira/browse/IGNITE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711187#comment-17711187 ] Pavel Tupitsyn commented on IGNITE-19222: - Merged to master: 3ecb062d91b21fb23e14a7cbe0805468aef3c398 > .NET: Add IgniteClientConfiguration.EnableClusterDiscovery > -- > > Key: IGNITE-19222 > URL: https://issues.apache.org/jira/browse/IGNITE-19222 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Affects Versions: 2.14 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.15 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Cluster discovery is always enabled currently, which can be undesirable in > some use cases. Add a config property to disable discovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19277) Add authorization of Ignite node stop/restart operations.
Mikhail Petrov created IGNITE-19277: --- Summary: Add authorization of Ignite node stop/restart operations. Key: IGNITE-19277 URL: https://issues.apache.org/jira/browse/IGNITE-19277 Project: Ignite Issue Type: Bug Reporter: Mikhail Petrov Before https://issues.apache.org/jira/browse/IGNITE-15322 IgniteCluster#restartNodes()/stopNodes() operations were authorized by the name of the internal tasks that is used during their execution - org.apache.ignite.internal.cluster.IgniteKillTask. After https://issues.apache.org/jira/browse/IGNITE-15322 has been resolved, authorization of internal tasks is no longer performed. So IgniteCluster#restartNodes()/stopNodes() operations are not authorized at all. We must to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19277) Add authorization of Ignite node stop/restart operations.
[ https://issues.apache.org/jira/browse/IGNITE-19277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov updated IGNITE-19277: Priority: Critical (was: Major) > Add authorization of Ignite node stop/restart operations. > - > > Key: IGNITE-19277 > URL: https://issues.apache.org/jira/browse/IGNITE-19277 > Project: Ignite > Issue Type: Bug >Reporter: Mikhail Petrov >Priority: Critical > > Before https://issues.apache.org/jira/browse/IGNITE-15322 > IgniteCluster#restartNodes()/stopNodes() operations were authorized by the > name of the internal tasks that is used during their execution - > org.apache.ignite.internal.cluster.IgniteKillTask. > After https://issues.apache.org/jira/browse/IGNITE-15322 has been resolved, > authorization of internal tasks is no longer performed. So > IgniteCluster#restartNodes()/stopNodes() operations are not authorized at > all. > We must to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19277) Add authorization of Ignite Cluster Node stop/restart operations.
[ https://issues.apache.org/jira/browse/IGNITE-19277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov updated IGNITE-19277: Summary: Add authorization of Ignite Cluster Node stop/restart operations. (was: Add authorization of Ignite node stop/restart operations.) > Add authorization of Ignite Cluster Node stop/restart operations. > - > > Key: IGNITE-19277 > URL: https://issues.apache.org/jira/browse/IGNITE-19277 > Project: Ignite > Issue Type: Bug >Reporter: Mikhail Petrov >Priority: Critical > > Before https://issues.apache.org/jira/browse/IGNITE-15322 > IgniteCluster#restartNodes()/stopNodes() operations were authorized by the > name of the internal tasks that is used during their execution - > org.apache.ignite.internal.cluster.IgniteKillTask. > After https://issues.apache.org/jira/browse/IGNITE-15322 has been resolved, > authorization of internal tasks is no longer performed. So > IgniteCluster#restartNodes()/stopNodes() operations are not authorized at > all. > We must to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19030) Platforms: Add authorization of DotNet Compute tasks
[ https://issues.apache.org/jira/browse/IGNITE-19030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov updated IGNITE-19030: Priority: Major (was: Critical) > Platforms: Add authorization of DotNet Compute tasks > > > Key: IGNITE-19030 > URL: https://issues.apache.org/jira/browse/IGNITE-19030 > Project: Ignite > Issue Type: Task > Components: platforms >Reporter: Mikhail Petrov >Assignee: Mikhail Petrov >Priority: Major > Labels: .NET, cpp > Fix For: 2.15 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Prior to https://issues.apache.org/jira/browse/IGNITE-15322, all DotNet and > CPP tasks were authorized by the name of the internal Ignite system task that > executes them on the Java side. > After https://issues.apache.org/jira/browse/IGNITE-15322 has been resolved, > authorization of internal tasks is no longer performed. As a result, we do > not have the ability to control the authorization of DotNet/CPP tasks. We > need to implement it. > It is proposed to reuse the same approach that the Java implementation uses - > to authorize DotNet tasks by their FQN - to authorize CPP tasks by the name f > the corresponding binary type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19030) Platforms: Add authorization of DotNet Compute tasks
[ https://issues.apache.org/jira/browse/IGNITE-19030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov updated IGNITE-19030: Priority: Critical (was: Major) > Platforms: Add authorization of DotNet Compute tasks > > > Key: IGNITE-19030 > URL: https://issues.apache.org/jira/browse/IGNITE-19030 > Project: Ignite > Issue Type: Task > Components: platforms >Reporter: Mikhail Petrov >Assignee: Mikhail Petrov >Priority: Critical > Labels: .NET, cpp > Fix For: 2.15 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Prior to https://issues.apache.org/jira/browse/IGNITE-15322, all DotNet and > CPP tasks were authorized by the name of the internal Ignite system task that > executes them on the Java side. > After https://issues.apache.org/jira/browse/IGNITE-15322 has been resolved, > authorization of internal tasks is no longer performed. As a result, we do > not have the ability to control the authorization of DotNet/CPP tasks. We > need to implement it. > It is proposed to reuse the same approach that the Java implementation uses - > to authorize DotNet tasks by their FQN - to authorize CPP tasks by the name f > the corresponding binary type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18954) Design a language for parsing of a filter expression
[
https://issues.apache.org/jira/browse/IGNITE-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-18954:
-
Description:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{[?(@.region == 'EU')]}} -- will give all nodes, that have region equals to
'EU'
{{[?(@.dataRegion > 10 && @storage != 'HDD')]}} -- will give all nodes, that
has dataRegion > 10 and storage not equal to HDD
UPD:
Validator form the definition of done was made in the separate ticket, so we
just needed to provide the method that can say
if the provided attributes of the node are satisfy a provided filter
was:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{[?(@.region == 'EU')]}} -- will give all nodes, that have region equals to
'EU'
{{[?(@.dataRegion > 10 && @storage != 'HDD')]}} -- will give all nodes, that
has dataRegion > 10 and storage not equal to HDD
UPD:
> Design a language for parsing of a filter expression
>
>
> Key: IGNITE-18954
> URL: https://issues.apache.org/jira/browse/IGNITE-18954
> Project: Ignite
> Issue Type: Improvement
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> {*}Motivation{*}:
> We need to parse and be able
[jira] [Updated] (IGNITE-18954) Design a language for parsing of a filter expression
[
https://issues.apache.org/jira/browse/IGNITE-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-18954:
-
Description:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{[?(@.region == 'EU')]}} -- will give all nodeName, that have region equals to
'EU'
{{[?(@.dataRegion > 10 && @storage != 'HDD')]}} -- will give all nodeNames,
that has dataRegion > 10 and storage not equal to HDD
UPD:
was:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{['nodeName'][?(@.['region'] == 'EU')]}} -- will give all nodeName, that have
region equals to 'EU'
{{['nodeName'].[?(@.['dataRegion'] > 10 && @.[storage] != 'HDD')]}} -- will
give all nodeNames, that has dataRegion > 10 and storage not equal to HDD
UPD:
> Design a language for parsing of a filter expression
>
>
> Key: IGNITE-18954
> URL: https://issues.apache.org/jira/browse/IGNITE-18954
> Project: Ignite
> Issue Type: Improvement
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> {*}Motivation{*}:
> We need to parse and be able to use filters, that user set for the
> distribution zone
> *Definition of done:*
>
> * Filters are parsed and it is possible to filter
[jira] [Updated] (IGNITE-18954) Design a language for parsing of a filter expression
[
https://issues.apache.org/jira/browse/IGNITE-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-18954:
-
Description:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{[?(@.region == 'EU')]}} -- will give all nodes, that have region equals to
'EU'
{{[?(@.dataRegion > 10 && @storage != 'HDD')]}} -- will give all nodes, that
has dataRegion > 10 and storage not equal to HDD
UPD:
was:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{[?(@.region == 'EU')]}} -- will give all nodeName, that have region equals to
'EU'
{{[?(@.dataRegion > 10 && @storage != 'HDD')]}} -- will give all nodeNames,
that has dataRegion > 10 and storage not equal to HDD
UPD:
> Design a language for parsing of a filter expression
>
>
> Key: IGNITE-18954
> URL: https://issues.apache.org/jira/browse/IGNITE-18954
> Project: Ignite
> Issue Type: Improvement
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> {*}Motivation{*}:
> We need to parse and be able to use filters, that user set for the
> distribution zone
> *Definition of done:*
>
> * Filters are parsed and it is possible to filter arbitrary set of attributes
> * Validator is
[jira] [Updated] (IGNITE-18954) Design a language for parsing of a filter expression
[
https://issues.apache.org/jira/browse/IGNITE-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-18954:
-
Description:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes":{
"nodeAttributes":{
"region":{
"attribute":"US"
},
"storage":{
"attribute":"SSD"
}
}
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{['nodeName'][?(@.['region'] == 'EU')]}} -- will give all nodeName, that have
region equals to 'EU'
{{['nodeName'].[?(@.['dataRegion'] > 10 && @.[storage] != 'HDD')]}} -- will
give all nodeNames, that has dataRegion > 10 and storage not equal to HDD
UPD:
was:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes": {
"region": "EU",
"storage": "SSD",
"dataRegion" : 10
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{['nodeName'][?(@.['region'] == 'EU')]}} -- will give all nodeName, that have
region equals to 'EU'
{{['nodeName'].[?(@.['dataRegion'] > 10 && @.[storage] != 'HDD')]}} -- will
give all nodeNames, that has dataRegion > 10 and storage not equal to HDD
UPD:
> Design a language for parsing of a filter expression
>
>
> Key: IGNITE-18954
> URL: https://issues.apache.org/jira/browse/IGNITE-18954
> Project: Ignite
> Issue Type: Improvement
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> {*}Motivation{*}:
> We need to parse and be able to use filters, that user set for the
> distribution zone
> *Definition of done:*
>
> * Filters are parsed and it is possible to filter arbitrary set of attributes
> * Validator is
[jira] [Updated] (IGNITE-18954) Design a language for parsing of a filter expression
[
https://issues.apache.org/jira/browse/IGNITE-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-18954:
-
Description:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes": {
"region": "EU",
"storage": "SSD",
"dataRegion" : 10
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{['nodeName'][?(@.['region'] == 'EU')]}} -- will give all nodeName, that have
region equals to 'EU'
{{['nodeName'].[?(@.['dataRegion'] > 10 && @.[storage] != 'HDD')]}} -- will
give all nodeNames, that has dataRegion > 10 and storage not equal to HDD
UPD:
was:
{*}Motivation{*}:
We need to parse and be able to use filters, that user set for the distribution
zone
*Definition of done:*
* Filters are parsed and it is possible to filter arbitrary set of attributes
* Validator is implemented, so we can validate a string representation of a
filter expression
*Implementation details:*
Language must allow to perform simple operations, like
* “ “ – default empty filter, means that all nodes match
* () expressions with parentheses
* key1 = “A” && key2 = “B”, () && ()
* key1 = “A” || key2 = “B”, () || ()
* key1 != "A"
Attributes representation must look like key -> value pairs, like
{noformat}
(region = EU || region = US) && (storage != HDD || storage = SSD)
{noformat}
After some investigation, I propose to not implement our own language for the
filters and attributes, but reuse JSON functionality.
For nodes, user just pass JSON representation of attributes through the
{{ignite-config.json}}:
{code:java}
{
"network": {
"port": 3344
},
"rest": {
"port": 10300
},
"nodeAttributes": {
"nodeName": "node1"
"region": "EU",
"storage": "SSD",
"dataRegion" : 10
}
}
{code}
For parsing, we can use https://github.com/json-path/JsonPath, which is already
used in ignite {{sql-engine}} module, so we don't need to bring a new
dependancy.
{{JsonPath}} has a wide functionality to work with JSON, we are interested in
filtering queries. See all list of available functionality here
https://github.com/json-path/JsonPath#functions. Example of usages you can find
here https://rows.com/docs/filtering-with-jsonpath
In general, it will allow us to make all operations, that we have listed before.
Lets provide some examples with our node attributes:
{{['nodeName'][?(@.['region'] == 'EU')]}} -- will give all nodeName, that have
region equals to 'EU'
{{['nodeName'].[?(@.['dataRegion'] > 10 && @.[storage] != 'HDD')]}} -- will
give all nodeNames, that has dataRegion > 10 and storage not equal to HDD
> Design a language for parsing of a filter expression
>
>
> Key: IGNITE-18954
> URL: https://issues.apache.org/jira/browse/IGNITE-18954
> Project: Ignite
> Issue Type: Improvement
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> {*}Motivation{*}:
> We need to parse and be able to use filters, that user set for the
> distribution zone
> *Definition of done:*
>
> * Filters are parsed and it is possible to filter arbitrary set of attributes
> * Validator is implemented, so we can validate a string representation of a
[jira] [Assigned] (IGNITE-19076) Add support of deployment statuses to CLI
[ https://issues.apache.org/jira/browse/IGNITE-19076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin reassigned IGNITE-19076: -- Assignee: Mikhail Pochatkin > Add support of deployment statuses to CLI > - > > Key: IGNITE-19076 > URL: https://issues.apache.org/jira/browse/IGNITE-19076 > Project: Ignite > Issue Type: Improvement >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Major > Labels: ignite-3 > > In IGNITE-18973 introduced deployment unit statuses. Need to add statutes > support to CLI and REST -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19192) Bump OpenApi spec generator version
[
https://issues.apache.org/jira/browse/IGNITE-19192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711062#comment-17711062
]
Mikhail Pochatkin commented on IGNITE-19192:
In the progress of updating found several problems:
1. Backward incompatible in micronaut-openapi-generator.
*micronaut.openapi.property.naming.strategy* changed from *CAMEL_CASE* to
*LOWER_CAMEL_CASE*. Introduced *micronaut.openapi.field.visibility.level* with
default value *PUBLIC* that produce problem with all private fields.
2. *@Produces* annotation start processing by openapi-generator annotation
processor and generate redundant response statuses to cover all produces media
types.
For example if presented
{code:java}
@Produces({MediaType.TEXT_PLAIN, MediaType.PROBLEM_JSON})
{code}
then openapi.yml for annotated method will contain both types for 200 status
response. This is not corrent, because PROBLEM_JSON used only for 4** and 5**
responses.
3. *require* attribute deprecated and introduced new enum attribute
*requiredMode*.
4. In some cases, the name from the *@Schema* annotation is ignored, this is
important for DTO objects. Thus, if any controller method has a reference to a
DTO object as an implementation attribute of the *@Schema* annotation, the
generator resolves the DTO name as the class name instead of the annotation
name. This is counter-intuitive and as a solution the Dto suffix has been
removed from all classes and the class name must be used as the object name in
the public API specification.
> Bump OpenApi spec generator version
> ---
>
> Key: IGNITE-19192
> URL: https://issues.apache.org/jira/browse/IGNITE-19192
> Project: Ignite
> Issue Type: Improvement
> Components: rest
>Reporter: Mikhail Pochatkin
>Assignee: Mikhail Pochatkin
>Priority: Major
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently micronaut openapi generator version is 3.2.0, but latest release is
> 4.8.6. So, need to use latest version, but between these versions exists
> non-backward compatibility changes. Need to fix spec definitions and
> generator propeties for correct spec generation. Ideally, spec should not be
> changed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18879) Leaseholder candidates balancing
[
https://issues.apache.org/jira/browse/IGNITE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-18879:
--
Description:
*Motivation*
Primary replicas (leaseholders) should be evenly distributed over cluster to
balance the transactional load between nodes. As the placement driver assigns
primary replicas, balancing the primary replicas is also it's responsibility.
Naive implementation of balancing should choose a node as leaseholder candidate
in a way to save even lease distribution over all nodes. In real cluster, it
may take into account slow nodes, hot table records, etc. If lease candidate
declines LeaseGrantMessage from placement driver, the balancer should make
decision to choose another candidate for given primary replica or enforce the
previously chosen. So the balancing algorith should be pluggable, so that we
could have ability to improve/replace/compare it with others.
*Definition of done*
Introduced interface for lease candidates balancer, and a simple implementation
sustaining even lease distribution, which is used by placement driver by
default. No public or internal configuration needed on this stage.
*Implementation notes*
Lease candidates balancer should have at least 2 methods:
- {_}get(group, ignoredNodes){_}: returns candidate for the given group, a
node from ignoredNodes set can't be chosen as a candidate
- {_}considerRedirectProposal(group, candidate, proposedCandidate){_}:
processes redirect proposal for given group provided by given candidate
(previously chosen using _get_ method), proposedCandidate is the alternative
candidate. Returns candidate that should be enforced by placement driver.
was:
*Motivation*
Primary replicas (leaseholders) should be evenly distributed over cluster to
balance the transactional load between nodes. As the placement driver assigns
primary replicas, balancing the primary replicas is also it's responsibility.
Naive implementation of balancing should choose a node as leaseholder candidate
in a way to save even lease distribution over all nodes. In real cluster, it
may take into account slow nodes, hot table records, etc. If lease candidate
declines LeaseGrantMessage from placement driver, the balancer should make
decision to choose another candidate for given primary replica or enforce the
previously chosen. So the balancing algorith should be pluggable, so that we
could have ability to improve/replace/compare it with others.
*Definition of done*
Introduced interface for lease candidates balancer, and a simple implementation
sustaining even lease distribution, which is used by placement driver by
default. No public or internal configuration needed on this stage.
*Implementation notes*
Lease candidates balancer should have at least 2 methods:
- _get(group)_: returns candidate for the given group
- _considerRedirectProposal(group, candidate, proposedCandidate)_: processes
redirect proposal for given group provided by given candidate (previously
chosen using _get_ method), proposedCandidate is the alternative candidate.
Returns candidate that should be enforced by placement driver.
> Leaseholder candidates balancing
>
>
> Key: IGNITE-18879
> URL: https://issues.apache.org/jira/browse/IGNITE-18879
> Project: Ignite
> Issue Type: Improvement
>Reporter: Denis Chudov
>Priority: Major
> Labels: ignite-3
>
> *Motivation*
> Primary replicas (leaseholders) should be evenly distributed over cluster to
> balance the transactional load between nodes. As the placement driver assigns
> primary replicas, balancing the primary replicas is also it's responsibility.
> Naive implementation of balancing should choose a node as leaseholder
> candidate in a way to save even lease distribution over all nodes. In real
> cluster, it may take into account slow nodes, hot table records, etc. If
> lease candidate declines LeaseGrantMessage from placement driver, the
> balancer should make decision to choose another candidate for given primary
> replica or enforce the previously chosen. So the balancing algorith should be
> pluggable, so that we could have ability to improve/replace/compare it with
> others.
> *Definition of done*
> Introduced interface for lease candidates balancer, and a simple
> implementation sustaining even lease distribution, which is used by placement
> driver by default. No public or internal configuration needed on this stage.
> *Implementation notes*
> Lease candidates balancer should have at least 2 methods:
> - {_}get(group, ignoredNodes){_}: returns candidate for the given group, a
> node from ignoredNodes set can't be chosen as a candidate
> - {_}considerRedirectProposal(group, candidate, proposedCandidate){_}:
> processes redirect proposal for given group provided by given candidate
[jira] [Updated] (IGNITE-18871) Sql. ColocatedSortAggregate rule can't use index for grouping
[
https://issues.apache.org/jira/browse/IGNITE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Orlov updated IGNITE-18871:
--
Summary: Sql. ColocatedSortAggregate rule can't use index for grouping
(was: Sql. ColocatedSortAggregate rule can't use index for grouping.)
> Sql. ColocatedSortAggregate rule can't use index for grouping
> -
>
> Key: IGNITE-18871
> URL: https://issues.apache.org/jira/browse/IGNITE-18871
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Andrey Mashenkov
>Assignee: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Assume there is a table with a composite index on (grp0, grp1) columns.
> ColocatedSortAggregate rule can’t use index for the next query in case of
> hash distribution:
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp1, grp0
> {code}
> However, index is for the query used in case of single distribution.
> Also, index is used for the next query in both cases: single and hash
> distributions.
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp0, grp1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-18766) Incorrect id check in ClusterGroupAdapter.forNodeId
[
https://issues.apache.org/jira/browse/IGNITE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711032#comment-17711032
]
Ignite TC Bot commented on IGNITE-18766:
{panel:title=Branch: [pull/10640/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10640/head] Base: [master] : New Tests
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Basic 1{color} [[tests
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7132563]]
* {color:#013220}IgniteBasicTestSuite:
GridProjectionForCachesSelfTest.testProjectionWithBadId - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7132674buildTypeId=IgniteTests24Java8_RunAll]
> Incorrect id check in ClusterGroupAdapter.forNodeId
> ---
>
> Key: IGNITE-18766
> URL: https://issues.apache.org/jira/browse/IGNITE-18766
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.14
>Reporter: Pavel Tupitsyn
>Assignee: Denis Kuznetsov
>Priority: Minor
> Labels: ise, newbie
> Fix For: 2.15
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ClusterGroup.forNodeId checks *id* in a loop instead of *id0*:
> {code:java}
> for (UUID id0 : ids) {
> if (contains(id))
> nodeIds.add(id0);
> }
> {code}
> https://github.com/apache/ignite/blob/3de1dc44f53ea510328badf6cb6b423fe6975bd8/modules/core/src/main/java/org/apache/ignite/internal/cluster/ClusterGroupAdapter.java#L461
> The following unit test demonstrates the problem:
> {code:java}
> @Test
> public void testProjectionWithBadId() {
> ClusterNode locNode = ignite.cluster().localNode();
> ClusterGroup prj = ignite.cluster().forNodeId(UUID.randomUUID(),
> locNode.id());
> Collection nodes = prj.nodes();
> assertEquals(1, nodes.size());
> }
> {code}
> (add to GridProjectionForCachesSelfTest)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-19222) .NET: Add IgniteClientConfiguration.EnableClusterDiscovery
[
https://issues.apache.org/jira/browse/IGNITE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711030#comment-17711030
]
Ignite TC Bot commented on IGNITE-19222:
{panel:title=Branch: [pull/10642/head] Base: [master] : Possible Blockers
(4)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Snapshots{color} [[tests 1 Exit Code , Failure on metric
|https://ci.ignite.apache.org/viewLog.html?buildId=7181020]]
* IgniteSnapshotTestSuite:
IgniteSnapshotMXBeanTest.testCreateSnapshot[encryption=false, onlyPrimay=false]
- Test has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}[Check Code Style Ducktests]{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=7180962]]
{color:#d04437}Disk Page Compressions 1{color} [[tests 0 TIMEOUT , Exit Code ,
Failure on metric |https://ci.ignite.apache.org/viewLog.html?buildId=7181038]]
{panel}
{panel:title=Branch: [pull/10642/head] Base: [master] : New Tests
(223)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Platform .NET (Core Linux){color} [[tests
116|https://ci.ignite.apache.org/viewLog.html?buildId=7181007]]
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(4,1)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(3,0)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(6,2)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(5,1)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AtomicLong_RequestIsRoutedToPrimaryNode(default-grp-replicated,null,Replicated,2)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AtomicLong_RequestIsRoutedToPrimaryNode(default-grp-partitioned,null,Partitioned,0)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AtomicLong_RequestIsRoutedToPrimaryNode(custom-grp-replicated,testAtomicLong,Replicated,0)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AtomicLong_RequestIsRoutedToPrimaryNode(custom-grp-partitioned,testAtomicLong,Partitioned,1)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(2,0)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.AllKeyBasedOperations_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(1,1)
- PASSED{color}
* {color:#013220}DotNetCore:
PartitionAwarenessWithClusterDiscoveryTest.CachePut_AllPrimitiveTypes_RequestIsRoutedToPrimaryNode(Hello
World,0) - PASSED{color}
... and 105 new tests
{color:#8b}Platform .NET (Windows){color} [[tests
107|https://ci.ignite.apache.org/viewLog.html?buildId=7181008]]
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGetAsync_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(3,0)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGetAsync_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(2,0)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGetAsync_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(5,1)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGetAsync_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(4,1)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CachePut_AllPrimitiveTypes_RequestIsRoutedToPrimaryNode(1,1)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGetAsync_PrimitiveKeyType_RequestIsRoutedToPrimaryNode(6,2)
- PASSED{color}
* {color:#013220}exe:
CachePut_AllPrimitiveTypes_RequestIsRoutedToPrimaryNode(uint.MaxValue,0) -
PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CachePut_AllPrimitiveTypes_RequestIsRoutedToPrimaryNode(2,0)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGet_UserDefinedKeyType_RequestIsRoutedToPrimaryNode(1,0)
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGet_RepeatedCall_DoesNotRequestAffinityMapping
- PASSED{color}
* {color:#013220}exe:
PartitionAwarenessWithClusterDiscoveryTest.CacheGet_UserDefinedKeyType_RequestIsRoutedToPrimaryNode(3,0)
- PASSED{color}
... and 96 new tests
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=7181041buildTypeId=IgniteTests24Java8_RunAll]
> .NET: Add
[jira] [Updated] (IGNITE-19275) Creation of big amount of tables throws exception
[
https://issues.apache.org/jira/browse/IGNITE-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Pereslegin updated IGNITE-19275:
--
Labels: ignite-3 (was: )
> Creation of big amount of tables throws exception
> -
>
> Key: IGNITE-19275
> URL: https://issues.apache.org/jira/browse/IGNITE-19275
> Project: Ignite
> Issue Type: Bug
> Components: general
>Affects Versions: 3.0
>Reporter: Igor
>Priority: Critical
> Labels: ignite-3
> Attachments: image-2023-04-11-16-11-09-934.png, node_0.log.zip,
> node_1.log.zip
>
>
> h1. *Steps:*
> # Sart 2 nodes
> # Init cluster
> # Connect to cluster via jdbc
> # Create table with 5 columns.
> # Wait 30 ms.
> # Repeat points 4-5 1000 times.
> h1. *Expected behavior:*
> # Creation time is constant.
> # Tables are created.
> h1. *Actual behavior:*
> # Creation time is increasing
> # The exception is thrown| after 200+ tables are created
> h2. *Details:*
> h3. Creation time graph:
> !image-2023-04-11-16-11-09-934.png|width=522,height=323!
> h3. Exceptions:
> # From client:
> {code:java}
> Exception in thread "main" java.sql.SQLException: Exception while executing
> query [query=CREATE TABLE table_239(id INT PRIMARY KEY, column_1 VARCHAR,
> column_2 VARCHAR, column_3 VARCHAR, column_4 VARCHAR)]. Error
> message:IGN-CMN-65535 TraceId:7299edda-2a88-4448-ba6d-366fca88cfd4
> at
> org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
> at
> org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:148)
> at
> org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)
> at lunigorn.ignite3test.Test.main(Test.java:18) {code}
> 2. From server logs:
> {code:java}
> 2023-04-11 15:13:49:890 +0200
> [WARNING][%node_1%Raft-Group-Client-5][DistributedConfigurationStorage] Meta
> storage listener issue
> org.apache.ignite.lang.IgniteInternalException: IGN-SQL-32
> TraceId:d3484efb-2ca6-4098-8c17-30a878c0801e Couldn't evaluate sql schemas
> for causality token: 722
> at
> org.apache.ignite.internal.sql.engine.schema.SqlSchemaManagerImpl.lambda$new$1(SqlSchemaManagerImpl.java:131)
> at
> org.apache.ignite.internal.causality.BaseVersionedValue.lambda$notifyCompletionListeners$7(BaseVersionedValue.java:331)
> {code}
> *The nodes logs are in attachment.*
>
> h3. The code to reproduce (or use [gradle
> project|https://github.com/Lunigorn/ignite3test/tree/tables-capacity-test]):
> {code:java}
> package lunigorn.ignite3test;
> import java.sql.*;
> public class Test {
> private static final String DB_URL = "jdbc:ignite:thin://127.0.1.1:10800";
> private static final int COLUMNS_COUNT = 5;
> private static final int TABLES_COUNT = 1000;
> private static final int SLEEP = 30;
> public static void main(String[] args) throws SQLException {
> System.out.println("Test started");
> try (Connection connection = DriverManager.getConnection(DB_URL);
> Statement statement = connection.createStatement()){
> System.out.println("Connection created");
> for (int i = 0; i < TABLES_COUNT; i++) {
> String createTableQuery = createTableQuery("table_" + i,
> COLUMNS_COUNT);
> long timestampBefore = System.currentTimeMillis();
> statement.executeUpdate(createTableQuery);
> long timestampAfter = System.currentTimeMillis();
> System.out.println("Create table " + i + " took " +
> (timestampAfter - timestampBefore) + " ms");
> if (i % 50 == 0){
> int tablesCount = findTablesCount(connection);
> if (tablesCount != i+1){
> throw new IllegalStateException("Expected " + (i+1) +
> " tables in cluster, but was " + tablesCount);
> }
> System.out.println("Tables count in cluster: " +
> tablesCount);
> }
> sleep();
> }
> }
> }
> public static String createTableQuery(String tableName, int
> columnsAmount){
> StringBuilder sb = new StringBuilder();
> sb.append("CREATE TABLE ").append(tableName).append("(");
> for (int i = 0; i < columnsAmount; i++) {
> if (i == 0){
> sb.append("id INT PRIMARY KEY");
> } else {
> sb.append("column_").append(i).append(" VARCHAR");
> }
> if (i != columnsAmount - 1){
> sb.append(", ");
> }
> }
> sb.append(")");
> return sb.toString();
> }
> public static int findTablesCount(Connection connection) throws
> SQLException {
>
[jira] [Updated] (IGNITE-15605) Calcite. Unexpected result with aggregate inside subquery.
[
https://issues.apache.org/jira/browse/IGNITE-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-15605:
---
Description:
{noformat}
statement ok
CREATE TABLE integers(i INTEGER)
statement ok
INSERT INTO integers VALUES (1), (2), (3), (NULL)
query R
SELECT (SELECT SUM(i1.i)) FROM integers i1;
6.00
{noformat}
{noformat}
/subquery/scalar/test_correlated_aggregate_subquery.test_ignore
/subquery/scalar/test_varchar_correlated_subquery.test_ignore
{noformat}
{code:java}
Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to
class java.lang.Integer (java.lang.Long and java.lang.Integer are in module
java.base of loader 'bootstrap')
at SC.execute(Unknown Source)
at
org.apache.ignite.internal.sql.engine.exec.exp.ExpressionFactoryImpl$ProjectImpl.apply(ExpressionFactoryImpl.java:654)
at
org.apache.ignite.internal.sql.engine.exec.rel.StorageScanNode.push(StorageScanNode.java:197)
at
org.apache.ignite.internal.sql.engine.exec.rel.StorageScanNode$SubscriberImpl.lambda$onComplete$2(StorageScanNode.java:303)
at
org.apache.ignite.internal.sql.engine.exec.ExecutionContext.lambda$execute$0(ExecutionContext.java:315){code}
was:
{noformat}
statement ok
CREATE TABLE integers(i INTEGER)
statement ok
INSERT INTO integers VALUES (1), (2), (3), (NULL)
query R
SELECT (SELECT SUM(i1.i)) FROM integers i1;
6.00
{noformat}
{noformat}
/subquery/scalar/test_correlated_aggregate_subquery.test_ignore
/subquery/scalar/test_varchar_correlated_subquery.test_ignore
{noformat}
> Calcite. Unexpected result with aggregate inside subquery.
> --
>
> Key: IGNITE-15605
> URL: https://issues.apache.org/jira/browse/IGNITE-15605
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Reporter: Evgeny Stanilovsky
>Priority: Major
> Labels: calcite, calcite2-required, calcite3-required, ignite-3
>
> {noformat}
> statement ok
> CREATE TABLE integers(i INTEGER)
> statement ok
> INSERT INTO integers VALUES (1), (2), (3), (NULL)
> query R
> SELECT (SELECT SUM(i1.i)) FROM integers i1;
>
> 6.00
> {noformat}
> {noformat}
> /subquery/scalar/test_correlated_aggregate_subquery.test_ignore
> /subquery/scalar/test_varchar_correlated_subquery.test_ignore
> {noformat}
>
> {code:java}
> Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast
> to class java.lang.Integer (java.lang.Long and java.lang.Integer are in
> module java.base of loader 'bootstrap')
> at SC.execute(Unknown Source)
> at
> org.apache.ignite.internal.sql.engine.exec.exp.ExpressionFactoryImpl$ProjectImpl.apply(ExpressionFactoryImpl.java:654)
> at
> org.apache.ignite.internal.sql.engine.exec.rel.StorageScanNode.push(StorageScanNode.java:197)
> at
> org.apache.ignite.internal.sql.engine.exec.rel.StorageScanNode$SubscriberImpl.lambda$onComplete$2(StorageScanNode.java:303)
> at
> org.apache.ignite.internal.sql.engine.exec.ExecutionContext.lambda$execute$0(ExecutionContext.java:315){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18871) Sql. ColocatedSortAggregate rule can't use index for grouping.
[
https://issues.apache.org/jira/browse/IGNITE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Orlov updated IGNITE-18871:
--
Ignite Flags: (was: Docs Required,Release Notes Required)
> Sql. ColocatedSortAggregate rule can't use index for grouping.
> --
>
> Key: IGNITE-18871
> URL: https://issues.apache.org/jira/browse/IGNITE-18871
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Andrey Mashenkov
>Assignee: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Assume there is a table with a composite index on (grp0, grp1) columns.
> ColocatedSortAggregate rule can’t use index for the next query in case of
> hash distribution:
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp1, grp0
> {code}
> However, index is for the query used in case of single distribution.
> Also, index is used for the next query in both cases: single and hash
> distributions.
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp0, grp1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19275) Creation of big amount of tables throws exception
[
https://issues.apache.org/jira/browse/IGNITE-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor updated IGNITE-19275:
--
Description:
h1. *Steps:*
# Sart 2 nodes
# Init cluster
# Connect to cluster via jdbc
# Create table with 5 columns.
# Wait 30 ms.
# Repeat points 4-5 1000 times.
h1. *Expected behavior:*
# Creation time is constant.
# Tables are created.
h1. *Actual behavior:*
# Creation time is increasing
# The exception is thrown| after 200+ tables are created
h2. *Details:*
h3. Creation time graph:
!image-2023-04-11-16-11-09-934.png|width=522,height=323!
h3. Exceptions:
# From client:
{code:java}
Exception in thread "main" java.sql.SQLException: Exception while executing
query [query=CREATE TABLE table_239(id INT PRIMARY KEY, column_1 VARCHAR,
column_2 VARCHAR, column_3 VARCHAR, column_4 VARCHAR)]. Error
message:IGN-CMN-65535 TraceId:7299edda-2a88-4448-ba6d-366fca88cfd4
at
org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
at
org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:148)
at
org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)
at lunigorn.ignite3test.Test.main(Test.java:18) {code}
2. From server logs:
{code:java}
2023-04-11 15:13:49:890 +0200
[WARNING][%node_1%Raft-Group-Client-5][DistributedConfigurationStorage] Meta
storage listener issue
org.apache.ignite.lang.IgniteInternalException: IGN-SQL-32
TraceId:d3484efb-2ca6-4098-8c17-30a878c0801e Couldn't evaluate sql schemas for
causality token: 722
at
org.apache.ignite.internal.sql.engine.schema.SqlSchemaManagerImpl.lambda$new$1(SqlSchemaManagerImpl.java:131)
at
org.apache.ignite.internal.causality.BaseVersionedValue.lambda$notifyCompletionListeners$7(BaseVersionedValue.java:331)
{code}
*The nodes logs are in attachment.*
h3. The code to reproduce (or use [gradle
project|https://github.com/Lunigorn/ignite3test/tree/tables-capacity-test]):
{code:java}
package lunigorn.ignite3test;
import java.sql.*;
public class Test {
private static final String DB_URL = "jdbc:ignite:thin://127.0.1.1:10800";
private static final int COLUMNS_COUNT = 5;
private static final int TABLES_COUNT = 1000;
private static final int SLEEP = 30;
public static void main(String[] args) throws SQLException {
System.out.println("Test started");
try (Connection connection = DriverManager.getConnection(DB_URL);
Statement statement = connection.createStatement()){
System.out.println("Connection created");
for (int i = 0; i < TABLES_COUNT; i++) {
String createTableQuery = createTableQuery("table_" + i,
COLUMNS_COUNT);
long timestampBefore = System.currentTimeMillis();
statement.executeUpdate(createTableQuery);
long timestampAfter = System.currentTimeMillis();
System.out.println("Create table " + i + " took " +
(timestampAfter - timestampBefore) + " ms");
if (i % 50 == 0){
int tablesCount = findTablesCount(connection);
if (tablesCount != i+1){
throw new IllegalStateException("Expected " + (i+1) + "
tables in cluster, but was " + tablesCount);
}
System.out.println("Tables count in cluster: " +
tablesCount);
}
sleep();
}
}
}
public static String createTableQuery(String tableName, int columnsAmount){
StringBuilder sb = new StringBuilder();
sb.append("CREATE TABLE ").append(tableName).append("(");
for (int i = 0; i < columnsAmount; i++) {
if (i == 0){
sb.append("id INT PRIMARY KEY");
} else {
sb.append("column_").append(i).append(" VARCHAR");
}
if (i != columnsAmount - 1){
sb.append(", ");
}
}
sb.append(")");
return sb.toString();
}
public static int findTablesCount(Connection connection) throws
SQLException {
DatabaseMetaData md = connection.getMetaData();
String catalog = connection.getCatalog();
ResultSet table_rs = md.getTables(catalog, null, null, new
String[]{"TABLE"});
int count = 0;
while (table_rs.next()){
count++;
}
return count;
}
public static void sleep(){
try {
Thread.sleep(SLEEP);
} catch (InterruptedException ignored) {
}
}
}{code}
was:
h1. *Steps:*
# Sart 2 nodes
# Init cluster
# Connect to cluster via jdbc
# Create table with 5 columns.
# Wait 30 ms.
# Repeat points 4-5 1000 times.
h1. *Expected behavior:*
# Creation time is constant.
# Tables are created.
h1. *Actual behavior:*
# Creation
[jira] [Created] (IGNITE-19276) Track partitions while building indexes
Ivan Bessonov created IGNITE-19276:
--
Summary: Track partitions while building indexes
Key: IGNITE-19276
URL: https://issues.apache.org/jira/browse/IGNITE-19276
Project: Ignite
Issue Type: Improvement
Reporter: Ivan Bessonov
IGNITE-18539 states:
{code:java}
when index is created, we also create a table somewhere that looks similar to
this:
indexCompletion..0 = false
...
indexCompletion..N = false {code}
This part has not been implemented, and it's separated into another issue.
Depending on the catalog service implementation, this data must either go to
meta-storage directly, or into a corresponding section in private configuration.
h3. Things to consider
* RAFT replica listener node may die after sending last "build index" command,
thus not updating, so new replica listener must compare states of indexes in
schema with states of indexes in storage.
* One of nodes must create new schema version when all partitions are built.
Choosing such node is a hard problem, so we can simply ask all nodes to do this
operation, making it idempotent and expecting that one of them would win. This
part requires careful thinking about the specifics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-19275) Creation of big amount of tables throws exception
Igor created IGNITE-19275:
-
Summary: Creation of big amount of tables throws exception
Key: IGNITE-19275
URL: https://issues.apache.org/jira/browse/IGNITE-19275
Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 3.0
Reporter: Igor
Attachments: image-2023-04-11-16-11-09-934.png, node_0.log.zip,
node_1.log.zip
h1. *Steps:*
# Sart 2 nodes
# Init cluster
# Connect to cluster via jdbc
# Create table with 5 columns.
# Wait 30 ms.
# Repeat points 4-5 1000 times.
h1. *Expected behavior:*
# Creation time is constant.
# Tables are created.
h1. *Actual behavior:*
# Creation time is increasing
# The exception is thrown| after 200+ tables are created
h2. *Details:*
h3. Creation time graph:
!image-2023-04-11-16-11-09-934.png|width=522,height=323!
h3. Exceptions:
# From client:
{code:java}
Exception in thread "main" java.sql.SQLException: Exception while executing
query [query=CREATE TABLE table_239(id INT PRIMARY KEY, column_1 VARCHAR,
column_2 VARCHAR, column_3 VARCHAR, column_4 VARCHAR)]. Error
message:IGN-CMN-65535 TraceId:7299edda-2a88-4448-ba6d-366fca88cfd4
at
org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
at
org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:148)
at
org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)
at lunigorn.ignite3test.Test.main(Test.java:18) {code}
2. From server logs:
{code:java}
2023-04-11 15:13:49:890 +0200
[WARNING][%node_1%Raft-Group-Client-5][DistributedConfigurationStorage] Meta
storage listener issue
org.apache.ignite.lang.IgniteInternalException: IGN-SQL-32
TraceId:d3484efb-2ca6-4098-8c17-30a878c0801e Couldn't evaluate sql schemas for
causality token: 722
at
org.apache.ignite.internal.sql.engine.schema.SqlSchemaManagerImpl.lambda$new$1(SqlSchemaManagerImpl.java:131)
at
org.apache.ignite.internal.causality.BaseVersionedValue.lambda$notifyCompletionListeners$7(BaseVersionedValue.java:331)
{code}
*The nodes logs are in attachment.*
h3. The code to reproduce (or use [gradle
project|https://github.com/Lunigorn/ignite3test/tree/tables-capacity-test]):
{code:java}
package lunigorn.ignite3test;
import java.sql.*;
public class Test {
private static final String DB_URL = "jdbc:ignite:thin://127.0.1.1:10800";
private static final int COLUMNS_COUNT = 5;
private static final int TABLES_COUNT = 1000;
private static final int SLEEP = 30;
public static void main(String[] args) throws SQLException {
System.out.println("Test started");
try (Connection connection = DriverManager.getConnection(DB_URL);
Statement statement = connection.createStatement()){
System.out.println("Connection created");
for (int i = 0; i < TABLES_COUNT; i++) {
String createTableQuery = createTableQuery("table_" + i,
COLUMNS_COUNT);
long timestampBefore = System.currentTimeMillis();
statement.executeUpdate(createTableQuery);
long timestampAfter = System.currentTimeMillis();
System.out.println("Create table " + i + " took " +
(timestampAfter - timestampBefore) + " ms");
if (i % 50 == 0){
int tablesCount = findTablesCount(connection);
if (tablesCount != i+1){
throw new IllegalStateException("Expected " + (i+1) + "
tables in cluster, but was " + tablesCount);
}
System.out.println("Tables count in cluster: " +
tablesCount);
}
sleep();
}
}
}
public static String createTableQuery(String tableName, int columnsAmount){
StringBuilder sb = new StringBuilder();
sb.append("CREATE TABLE ").append(tableName).append("(");
for (int i = 0; i < columnsAmount; i++) {
if (i == 0){
sb.append("id INT PRIMARY KEY");
} else {
sb.append("column_").append(i).append(" VARCHAR");
}
if (i != columnsAmount - 1){
sb.append(", ");
}
}
sb.append(")");
return sb.toString();
}
public static int findTablesCount(Connection connection) throws
SQLException {
DatabaseMetaData md = connection.getMetaData();
String catalog = connection.getCatalog();
ResultSet table_rs = md.getTables(catalog, null, null, new
String[]{"TABLE"});
int count = 0;
while (table_rs.next()){
count++;
}
return count;
}
public static void sleep(){
try {
Thread.sleep(SLEEP);
} catch (InterruptedException ignored) {
}
}
}{code}
--
[jira] [Created] (IGNITE-19274) Sql. Jdbc side working with TIMESTAMP WITH LOCAL TIME ZONE did not take into account current tz while storing data.
Evgeny Stanilovsky created IGNITE-19274:
---
Summary: Sql. Jdbc side working with TIMESTAMP WITH LOCAL TIME
ZONE did not take into account current tz while storing data.
Key: IGNITE-19274
URL: https://issues.apache.org/jira/browse/IGNITE-19274
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 3.0.0-beta1
Reporter: Evgeny Stanilovsky
The {{TIMESTAMP WITH LOCAL TIME ZONE}} data type is a variant of {{TIMESTAMP}}
that includes a time zone offset in its value. Data stored in the database is
normalized to the database time zone (UTC) and time zone offset is not stored
as part of the column data. When the data is retrieved, it to be returned in
the user's local session time zone.
i.e:
{noformat}
CREATE TABLE timestamp(ts TIMESTAMP, t_tz TIMESTAMP WITH TIME ZONE);
SET TIME ZONE 'tz1';
INSERT INTO timestamp VALUES ('2011-01-01 01:01:01', TIMESTAMP WITH TIME ZONE
'2011-01-01 01:01:01');
SET TIME ZONE 'tz2';
INSERT INTO timestamp VALUES ('2011-01-01 01:01:01', TIMESTAMP WITH TIME ZONE
'2011-01-01 01:01:01');
...
select * from timestamp;{noformat}
returned rows need to be different in case of different tz1 and tz2 offsets but
they are equals for now. Also returned representation need to be present in
user session time zone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-19164) Improve message about requested partitions during snapshot restore
[
https://issues.apache.org/jira/browse/IGNITE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710964#comment-17710964
]
Ignite TC Bot commented on IGNITE-19164:
{panel:title=Branch: [pull/10638/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10638/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7132560buildTypeId=IgniteTests24Java8_RunAll]
> Improve message about requested partitions during snapshot restore
> --
>
> Key: IGNITE-19164
> URL: https://issues.apache.org/jira/browse/IGNITE-19164
> Project: Ignite
> Issue Type: Task
>Reporter: Ilya Shishkov
>Assignee: Julia Bakulina
>Priority: Minor
> Labels: iep-43, ise
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, during snapshot restore message is logged before requesting
> partitions from remote nodes:
> {quote}
> [2023-03-24T18:06:59,910][INFO
> ]\[disco-notifier-worker-#792%node%|#792%node%][SnapshotRestoreProcess]
> Trying to request partitions from remote nodes
> [reqId=ff682204-9554-4fbb-804c-38a79c0b286a, snapshot=snapshot_name,
> map={*{color:#FF}76e22ef5-3c76-4987-bebd-9a6222a0{color}*={*{color:#FF}-903566235{color}*=[0,2,4,6,11,12,18,98,100,170,190,194,1015],
>
> *{color:#FF}1544803905{color}*=[1,11,17,18,22,25,27,35,37,42,45,51,62,64,67,68,73,76,1017]}}]
> {quote}
> It is necessary to make this output "human readable":
> # Print messages per node instead of one message for all nodes.
> # Print node consistent id and address.
> # Print cache / group name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-16985) Design table management flow (part 1)
[ https://issues.apache.org/jira/browse/IGNITE-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-16985: -- Attachment: (was: flow.png) > Design table management flow (part 1) > - > > Key: IGNITE-16985 > URL: https://issues.apache.org/jira/browse/IGNITE-16985 > Project: Ignite > Issue Type: Task >Reporter: Alexander Lapin >Assignee: Denis Chudov >Priority: Major > Labels: ignite-3 > Attachments: VersionedValuesUpdates.svg, VersionedValuesUpdates.yuml, > flow.png, flow.puml, onTableCreate.svg, onTableCreate.yuml, onTableDrop.svg, > onTableDrop.yuml, table.svg, table.yuml > > > As a part of the issue planed: > # Draw a time diagram of all operations: createTable(), dropTable(), > table(), tables(). > # Emphases of correctness work of causality tokens: tablesByIdVv. > # Reflect cross components interactions. Components: Schema manager, SQL > manager (SqlQueryProcessor), Affinity manager (it is not dedicated for now). > > The task for this ticket is to make a detailed diagram of the current flow, > to ease the design itself. > Definition of done: > We have detailed and clear description of table manager flows and the ticket > IGNITE-18989 is enriched with details about the flaws we want to fix. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16985) Design table management flow (part 1)
[ https://issues.apache.org/jira/browse/IGNITE-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-16985: -- Attachment: flow.puml flow.png > Design table management flow (part 1) > - > > Key: IGNITE-16985 > URL: https://issues.apache.org/jira/browse/IGNITE-16985 > Project: Ignite > Issue Type: Task >Reporter: Alexander Lapin >Assignee: Denis Chudov >Priority: Major > Labels: ignite-3 > Attachments: VersionedValuesUpdates.svg, VersionedValuesUpdates.yuml, > flow.png, flow.puml, onTableCreate.svg, onTableCreate.yuml, onTableDrop.svg, > onTableDrop.yuml, table.svg, table.yuml > > > As a part of the issue planed: > # Draw a time diagram of all operations: createTable(), dropTable(), > table(), tables(). > # Emphases of correctness work of causality tokens: tablesByIdVv. > # Reflect cross components interactions. Components: Schema manager, SQL > manager (SqlQueryProcessor), Affinity manager (it is not dedicated for now). > > The task for this ticket is to make a detailed diagram of the current flow, > to ease the design itself. > Definition of done: > We have detailed and clear description of table manager flows and the ticket > IGNITE-18989 is enriched with details about the flaws we want to fix. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16985) Design table management flow (part 1)
[ https://issues.apache.org/jira/browse/IGNITE-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-16985: -- Attachment: (was: flow.puml) > Design table management flow (part 1) > - > > Key: IGNITE-16985 > URL: https://issues.apache.org/jira/browse/IGNITE-16985 > Project: Ignite > Issue Type: Task >Reporter: Alexander Lapin >Assignee: Denis Chudov >Priority: Major > Labels: ignite-3 > Attachments: VersionedValuesUpdates.svg, VersionedValuesUpdates.yuml, > flow.png, flow.puml, onTableCreate.svg, onTableCreate.yuml, onTableDrop.svg, > onTableDrop.yuml, table.svg, table.yuml > > > As a part of the issue planed: > # Draw a time diagram of all operations: createTable(), dropTable(), > table(), tables(). > # Emphases of correctness work of causality tokens: tablesByIdVv. > # Reflect cross components interactions. Components: Schema manager, SQL > manager (SqlQueryProcessor), Affinity manager (it is not dedicated for now). > > The task for this ticket is to make a detailed diagram of the current flow, > to ease the design itself. > Definition of done: > We have detailed and clear description of table manager flows and the ticket > IGNITE-18989 is enriched with details about the flaws we want to fix. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19267) Implement local Low Watermark propagation
[
https://issues.apache.org/jira/browse/IGNITE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko reassigned IGNITE-19267:
Assignee: Kirill Tkalenko
> Implement local Low Watermark propagation
> -
>
> Key: IGNITE-19267
> URL: https://issues.apache.org/jira/browse/IGNITE-19267
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> According to the IEP-91, we can delete old data, once it becomes older than
> the certain threshold. At the moment, we can consider this threshold to be
> shared between different tables, but be unique on individual nodes. It's
> called a Low Watermark (LW).
> The way the value is chosen is the following:
> * There's the {_}data availability time{_}, that can be configured by the
> user. This is a cluster configuration. It has a value of, for example, 45
> minutes. Valid values - {{{}[0, +INF){}}}.
> * There's a {_}GC frequency{_}, that's also a cluster configuration. For
> example, 5 minuter. Range of valid values should be more strict.
> * Last LW value is persisted in Vault.
> * Every 5 minutes, we assign a new "lwCandidate{{{} = now() - 45min -
> maxClockSkew"{}}}.
> ** If there are no running transactions with timestamp below
> {{{}lwCandidate{}}}, we promote the candidate into a real LW value.
> ** Otherwise, we trigger GC with timestamp of the transaction with the
> oldest timestamp (and promoting LW to that timestamp), and raising the bar
> every time* that transaction is completed. Eventually, we will reach the
> point where there are no running transactions with timestamp below
> {{{}lwCandidate{}}}.
> * it's not necessary to do it every time. But, once the timestamp of the
> oldest RO transaction is above or equal to {{{}lwCandidate{}}}, we must
> guarantee its promotion. Everything else is optimization.
> * If there's a new RO transaction with timestamp below {{{}lwCandidate{}}},
> we fail it.
> Promoted LW value cannot become smaller no matter what. All data below LW is
> considered to be invalid, maybe broken and completely invisible to user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
[ https://issues.apache.org/jira/browse/IGNITE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Timonin updated IGNITE-19273: Description: ContinuousDataLoadApplication put all keys from specified range in single transaction. Test uses 5.000.000 keys, and the transaction doesn't finish. This is a bug, and it prevents from reusing this class for other tests. was:Conti > [ducktape] Transaction doesn't finished in discovery test > - > > Key: IGNITE-19273 > URL: https://issues.apache.org/jira/browse/IGNITE-19273 > Project: Ignite > Issue Type: Improvement >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > Fix For: 2.16 > > > ContinuousDataLoadApplication put all keys from specified range in single > transaction. Test uses 5.000.000 keys, and the transaction doesn't finish. > > This is a bug, and it prevents from reusing this class for other tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
[ https://issues.apache.org/jira/browse/IGNITE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Timonin updated IGNITE-19273: Ignite Flags: (was: Docs Required,Release Notes Required) > [ducktape] Transaction doesn't finished in discovery test > - > > Key: IGNITE-19273 > URL: https://issues.apache.org/jira/browse/IGNITE-19273 > Project: Ignite > Issue Type: Improvement >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
[ https://issues.apache.org/jira/browse/IGNITE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Timonin updated IGNITE-19273: Fix Version/s: 2.16 > [ducktape] Transaction doesn't finished in discovery test > - > > Key: IGNITE-19273 > URL: https://issues.apache.org/jira/browse/IGNITE-19273 > Project: Ignite > Issue Type: Improvement >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > Fix For: 2.16 > > > Conti -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
[ https://issues.apache.org/jira/browse/IGNITE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Timonin updated IGNITE-19273: Description: Conti > [ducktape] Transaction doesn't finished in discovery test > - > > Key: IGNITE-19273 > URL: https://issues.apache.org/jira/browse/IGNITE-19273 > Project: Ignite > Issue Type: Improvement >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > > Conti -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
Maksim Timonin created IGNITE-19273: --- Summary: [ducktape] Transaction doesn't finished in discovery test Key: IGNITE-19273 URL: https://issues.apache.org/jira/browse/IGNITE-19273 Project: Ignite Issue Type: Improvement Reporter: Maksim Timonin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19273) [ducktape] Transaction doesn't finished in discovery test
[ https://issues.apache.org/jira/browse/IGNITE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Timonin reassigned IGNITE-19273: --- Assignee: Maksim Timonin > [ducktape] Transaction doesn't finished in discovery test > - > > Key: IGNITE-19273 > URL: https://issues.apache.org/jira/browse/IGNITE-19273 > Project: Ignite > Issue Type: Improvement >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19272) Uninformative jdbc failure after insertion into TIMESTAMP WITH LOCAL TIMEZONE
Evgeny Stanilovsky created IGNITE-19272:
---
Summary: Uninformative jdbc failure after insertion into TIMESTAMP
WITH LOCAL TIMEZONE
Key: IGNITE-19272
URL: https://issues.apache.org/jira/browse/IGNITE-19272
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 3.0.0-beta1
Reporter: Evgeny Stanilovsky
{code:java}
try (Statement stmt = conn.createStatement()) {
stmt.executeUpdate("CREATE TABLE timetest(id INT PRIMARY KEY,"
+ "ts TIMESTAMP, ts_tz TIMESTAMP WITH LOCAL TIME ZONE)");
stmt.executeUpdate("INSERT INTO timetest VALUES "
+ "(3, '2011-01-01 01:01:01', TIMESTAMP WITH LOCAL TIME
ZONE '2011-01-01 01:01:01')"); // <- ok
stmt.executeUpdate("INSERT INTO timetest VALUES "
+ "(1, '2011-01-01 01:01:01', '2011-01-01 01:01:01')"); //
<- failed
}
{code}
failed with no reason why:
{noformat}
java.sql.SQLException: Exception while executing query [query=INSERT INTO
timetest VALUES (3, '2011-01-01 01:01:01', '2011-01-01 01:01:01')]. Error
message:IGN-CMN-65535 TraceId:ba72bbc8-a0e3-414b-a520-6eed739b3f45 Remote query
execution
at
org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
at
org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:148)
at
org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-19222) .NET: Add IgniteClientConfiguration.EnableClusterDiscovery
[ https://issues.apache.org/jira/browse/IGNITE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710927#comment-17710927 ] Igor Sapego commented on IGNITE-19222: -- Looks good to me. > .NET: Add IgniteClientConfiguration.EnableClusterDiscovery > -- > > Key: IGNITE-19222 > URL: https://issues.apache.org/jira/browse/IGNITE-19222 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Affects Versions: 2.14 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.15 > > Time Spent: 20m > Remaining Estimate: 0h > > Cluster discovery is always enabled currently, which can be undesirable in > some use cases. Add a config property to disable discovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19251) ODBC 3.0 Enhancements
[ https://issues.apache.org/jira/browse/IGNITE-19251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Sapego updated IGNITE-19251: - Epic Name: ODBC 3.0 Enhancements (was: ODBC 3.0 Enchantments) > ODBC 3.0 Enhancements > - > > Key: IGNITE-19251 > URL: https://issues.apache.org/jira/browse/IGNITE-19251 > Project: Ignite > Issue Type: Epic > Components: odbc >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > > Enchantments for the Ignite 3 ODBC driver -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19251) ODBC 3.0 Enhancements
[ https://issues.apache.org/jira/browse/IGNITE-19251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Sapego updated IGNITE-19251: - Summary: ODBC 3.0 Enhancements (was: ODBC 3.0 Enchantments) > ODBC 3.0 Enhancements > - > > Key: IGNITE-19251 > URL: https://issues.apache.org/jira/browse/IGNITE-19251 > Project: Ignite > Issue Type: Epic > Components: odbc >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > > Enchantments for the Ignite 3 ODBC driver -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19261) Replace parser tests for create/alter zones query by the tests for ddl converter
[ https://issues.apache.org/jira/browse/IGNITE-19261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-19261: - Labels: ignite-3 (was: ) > Replace parser tests for create/alter zones query by the tests for ddl > converter > > > Key: IGNITE-19261 > URL: https://issues.apache.org/jira/browse/IGNITE-19261 > Project: Ignite > Issue Type: Task >Reporter: Kirill Gusakov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19251) ODBC 3.0 Enchantments
[ https://issues.apache.org/jira/browse/IGNITE-19251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-19251: - Labels: ignite-3 (was: ) > ODBC 3.0 Enchantments > - > > Key: IGNITE-19251 > URL: https://issues.apache.org/jira/browse/IGNITE-19251 > Project: Ignite > Issue Type: Epic > Components: odbc >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > > Enchantments for the Ignite 3 ODBC driver -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19250) ODBC 3.0 Metainformation
[ https://issues.apache.org/jira/browse/IGNITE-19250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-19250: - Labels: ignite-3 (was: ) > ODBC 3.0 Metainformation > > > Key: IGNITE-19250 > URL: https://issues.apache.org/jira/browse/IGNITE-19250 > Project: Ignite > Issue Type: Epic > Components: odbc >Reporter: Igor Sapego >Priority: Major > Labels: ignite-3 > > ODBC features that related to metadata providing and handling. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19182) Fix NPE on building indexes
[
https://issues.apache.org/jira/browse/IGNITE-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vyacheslav Koptilin updated IGNITE-19182:
-
Labels: ignite-3 (was: ignite,)
> Fix NPE on building indexes
> ---
>
> Key: IGNITE-19182
> URL: https://issues.apache.org/jira/browse/IGNITE-19182
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After the implementation of index building, fluky tests appeared:
> *
> *org.apache.ignite.internal.runner.app.ItDataSchemaSyncTest#checkSchemasCorrectUpdate*
> Stack traces of errors:
> {noformat}
> java.util.concurrent.CompletionException: java.lang.NullPointerException:
> Cannot invoke "java.util.List.isEmpty()" because "batch" is null
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.lang.NullPointerException: Cannot invoke
> "java.util.List.isEmpty()" because "batch" is null
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.getNextRowIdForNextBatch(IndexBuilder.java:250)
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.lambda$run$1(IndexBuilder.java:165)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-19271:
---
Ignite Flags: (was: Docs Required,Release Notes Required)
> Persist revision-safeTime mapping in meta-storage
> -
>
> Key: IGNITE-19271
> URL: https://issues.apache.org/jira/browse/IGNITE-19271
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> IEP-98 states:
> {code:java}
> When creating a message M telling the cluster about a schema update
> activation moment, choose the message timestamp Tm (moving safeTime forward)
> equal to Now, but assign Tu (activation moment) contained in that M to be
> Tm+DD {code}
> This is hard to achieve.
> h3. Problem
> We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
> not straightforward. This is because we have too many actors:
> * There's a {_}client{_}, that chooses Tu, because it's the only actor that
> can affect message content.
> * There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses
> Tm.
> * There's everybody else, who expect a correspondence between Tu and Tm.
> First two actors are important, because they have independent clocks, but
> must coordinate the same event. This is impossible with described protocol.
> h3. Discussion
> Let's consider these two solutions:
> # Client generates Tm.
> # Meta-storage generates Tu.
> Option 1 is out of question, there must be only a single node at any given
> moment in time, that's responsible for the linear order of time in messages.
> What about option 2? Since meta-storage doesn't know anything about commands
> semantics, it can't really generate any data. So this solution doesn't work
> either.
> h3. Solution
> Combined solution could be the following:
> * Client sends DD as part of the command (this is not a constant, user _can_
> configure it, if they really feel like doing it)
> * Meta-storage generates {{Tm}}
> * Every node, upon receiving the update, calculates {{Tu}}
> This could work, if nodes would have never been restarted. There's one
> problem that needs to be solved: recovering the values of {{Tm}} from the
> (old) data upon node restart.
> This can be achieved by persisting safeTime along with revision as a part of
> metadata, that can be retrieved back through the meta-storage service API.
> In other words:
> 1. Client sends
> {code:java}
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s{code}
> 2. Lease-holder adds meta-data to the command:
> {code:java}
> safeTime = 10:10
> {code}
> 3. Meta-storage listener writes the data:
> {code:java}
> revision = 33
> schema.latest = 5
> schema.5.data = ...
> schema.5.dd = 30s
> revision.33.safeTime = 10:10:00{code}
>
> How can you read {{{}Tu{}}}:
> * read "{{{}schema.5.dd"{}}};
> * read its revision, it's 33;
> * read a timestamp of revision 33 via specialized API;
> * add two values together.
> h3. Implications and restrictions
> There's a cleanup process in the meta-storage. It will eventually remove any
> "revision.x.safeTime" values, because corresponding revision became obsolete.
> But, we should somehow preserve timestamps of revisions that are used y
> schemas. Such behavior can be achieved, if components can reserve a revision,
> and meta-storage can't compact it unless the reservation has been revoked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-19271:
---
Description:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
h3. Problem
We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
not straightforward. This is because we have too many actors:
* There's a {_}client{_}, that chooses Tu, because it's the only actor that
can affect message content.
* There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses Tm.
* There's everybody else, who expect a correspondence between Tu and Tm.
First two actors are important, because they have independent clocks, but must
coordinate the same event. This is impossible with described protocol.
h3. Discussion
Let's consider these two solutions:
# Client generates Tm.
# Meta-storage generates Tu.
Option 1 is out of question, there must be only a single node at any given
moment in time, that's responsible for the linear order of time in messages.
What about option 2? Since meta-storage doesn't know anything about commands
semantics, it can't really generate any data. So this solution doesn't work
either.
h3. Solution
Combined solution could be the following:
* Client sends DD as part of the command (this is not a constant, user _can_
configure it, if they really feel like doing it)
* Meta-storage generates {{Tm}}
* Every node, upon receiving the update, calculates {{Tu}}
This could work, if nodes would have never been restarted. There's one problem
that needs to be solved: recovering the values of {{Tm}} from the (old) data
upon node restart.
This can be achieved by persisting safeTime along with revision as a part of
metadata, that can be retrieved back through the meta-storage service API.
In other words:
1. Client sends
{code:java}
schema.latest = 5
schema.5.data = ...
schema.5.dd = 30s{code}
2. Lease-holder adds meta-data to the command:
{code:java}
safeTime = 10:10
{code}
3. Meta-storage listener writes the data:
{code:java}
revision = 33
schema.latest = 5
schema.5.data = ...
schema.5.dd = 30s
revision.33.safeTime = 10:10:00{code}
How can you read {{{}Tu{}}}:
* read "{{{}schema.5.dd"{}}};
* read its revision, it's 33;
* read a timestamp of revision 33 via specialized API;
* add two values together.
h3. Implications and restrictions
There's a cleanup process in the meta-storage. It will eventually remove any
"revision.x.safeTime" values, because corresponding revision became obsolete.
But, we should somehow preserve timestamps of revisions that are used y
schemas. Such behavior can be achieved, if components can reserve a revision,
and meta-storage can't compact it unless the reservation has been revoked.
was:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
h3. Problem
We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
not straightforward. This is because we have too many actors:
* There's a {_}client{_}, that chooses Tu, because it's the only actor that
can affect message content.
* There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses Tm.
* There's everybody else, who expect a correspondence between Tu and Tm.
First two actors are important, because they have independent clocks, but must
coordinate the same event. This is impossible with described protocol.
h3. Discussion
Let's consider these two solutions:
# Client generates Tm.
# Meta-storage generates Tu.
Option 1 is out of question, there must be only a single node at any given
moment in time, that's responsible for the linear order of time in messages.
What about option 2? Since meta-storage doesn't know anything about commands
semantics, it can't really generate any data. So this solution doesn't work
either.
> Persist revision-safeTime mapping in meta-storage
> -
>
> Key: IGNITE-19271
> URL: https://issues.apache.org/jira/browse/IGNITE-19271
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> IEP-98 states:
> {code:java}
> When creating a message M telling the cluster about a schema update
> activation moment, choose the message timestamp Tm (moving safeTime forward)
> equal to Now, but assign Tu (activation moment) contained in
[jira] [Resolved] (IGNITE-18274) Exception on modulus operator in SQL
[
https://issues.apache.org/jira/browse/IGNITE-18274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich resolved IGNITE-18274.
Fix Version/s: (was: 3.0.0-beta2)
Resolution: Cannot Reproduce
Seems after upgrade calcite version the bug disappeared
> Exception on modulus operator in SQL
>
>
> Key: IGNITE-18274
> URL: https://issues.apache.org/jira/browse/IGNITE-18274
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Reporter: Pavel Tupitsyn
>Priority: Major
> Labels: calcite, calcite2-required, calcite3-required, ignite-3
>
> *Query*
> {code}
> select key % 2 from TBL_DOUBLE
> {code}
> *Result*
> {code}
> java.lang.AssertionError: Conversion to relational algebra failed to preserve
> datatypes:
> validated type:
> RecordType(INTEGER NOT NULL KEY % 2) NOT NULL
> converted type:
> RecordType(DECIMAL(25, 15) NOT NULL KEY % 2) NOT NULL
> rel:
> LogicalProject(KEY % 2=[MOD(CAST($0):DECIMAL(30, 15) NOT NULL, 2)])
> IgniteLogicalTableScan(table=[[PUBLIC, TBL_DOUBLE]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:492)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:607)
> at
> org.apache.ignite.internal.sql.engine.prepare.IgnitePlanner.rel(IgnitePlanner.java:222)
> at
> org.apache.ignite.internal.sql.engine.prepare.PlannerHelper.optimize(PlannerHelper.java:63)
> at
> org.apache.ignite.internal.sql.engine.prepare.PrepareServiceImpl.lambda$prepareQuery$1(PrepareServiceImpl.java:231)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> {code}
> Note that some simple queries like "select 1 % 2" work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-18899) Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to MutableRowTupleAdapter
[ https://issues.apache.org/jira/browse/IGNITE-18899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710901#comment-17710901 ] Pavel Tupitsyn commented on IGNITE-18899: - Merged to main: 708dc99bfea060fe8887df5e1e9bad7e192a98f8 > Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to > MutableRowTupleAdapter > > > Key: IGNITE-18899 > URL: https://issues.apache.org/jira/browse/IGNITE-18899 > Project: Ignite > Issue Type: Improvement > Components: thin client >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > * Client protocol uses *BinaryTuple* to exchange data (IGNITE-17297) > * When reading data from server, currently we unpack all elements of the > incoming *BinaryTuple* into *ClientTuple* > This is extra work and extra allocations. Instead, wrap the incoming > *BinaryTuple* like *MutableRowTupleAdapter* does. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19222) .NET: Add IgniteClientConfiguration.EnableClusterDiscovery
[ https://issues.apache.org/jira/browse/IGNITE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn reassigned IGNITE-19222: --- Assignee: Pavel Tupitsyn > .NET: Add IgniteClientConfiguration.EnableClusterDiscovery > -- > > Key: IGNITE-19222 > URL: https://issues.apache.org/jira/browse/IGNITE-19222 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Affects Versions: 2.14 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.15 > > > Cluster discovery is always enabled currently, which can be undesirable in > some use cases. Add a config property to disable discovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
[
https://issues.apache.org/jira/browse/IGNITE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-19271:
---
Description:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
h3. Problem
We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
not straightforward. This is because we have too many actors:
* There's a {_}client{_}, that chooses Tu, because it's the only actor that
can affect message content.
* There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses Tm.
* There's everybody else, who expect a correspondence between Tu and Tm.
First two actors are important, because they have independent clocks, but must
coordinate the same event. This is impossible with described protocol.
h3. Discussion
Let's consider these two solutions:
# Client generates Tm.
# Meta-storage generates Tu.
Option 1 is out of question, there must be only a single node at any given
moment in time, that's responsible for the linear order of time in messages.
What about option 2? Since meta-storage doesn't know anything about commands
semantics, it can't really generate any data. So this solution doesn't work
either.
was:
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
> Persist revision-safeTime mapping in meta-storage
> -
>
> Key: IGNITE-19271
> URL: https://issues.apache.org/jira/browse/IGNITE-19271
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> IEP-98 states:
> {code:java}
> When creating a message M telling the cluster about a schema update
> activation moment, choose the message timestamp Tm (moving safeTime forward)
> equal to Now, but assign Tu (activation moment) contained in that M to be
> Tm+DD {code}
> This is hard to achieve.
> h3. Problem
> We need {{{}Tu==Tm+DD{}}}. Right now, with what we have in IGNITE-19028, it's
> not straightforward. This is because we have too many actors:
> * There's a {_}client{_}, that chooses Tu, because it's the only actor that
> can affect message content.
> * There's a meta-storage {_}lease-holder{_}, or {_}leader{_}, that chooses
> Tm.
> * There's everybody else, who expect a correspondence between Tu and Tm.
> First two actors are important, because they have independent clocks, but
> must coordinate the same event. This is impossible with described protocol.
> h3. Discussion
> Let's consider these two solutions:
> # Client generates Tm.
> # Meta-storage generates Tu.
> Option 1 is out of question, there must be only a single node at any given
> moment in time, that's responsible for the linear order of time in messages.
> What about option 2? Since meta-storage doesn't know anything about commands
> semantics, it can't really generate any data. So this solution doesn't work
> either.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-19271) Persist revision-safeTime mapping in meta-storage
Ivan Bessonov created IGNITE-19271:
--
Summary: Persist revision-safeTime mapping in meta-storage
Key: IGNITE-19271
URL: https://issues.apache.org/jira/browse/IGNITE-19271
Project: Ignite
Issue Type: Improvement
Reporter: Ivan Bessonov
IEP-98 states:
{code:java}
When creating a message M telling the cluster about a schema update activation
moment, choose the message timestamp Tm (moving safeTime forward) equal to Now,
but assign Tu (activation moment) contained in that M to be Tm+DD {code}
This is hard to achieve.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-19267) Implement local Low Watermark propagation
[
https://issues.apache.org/jira/browse/IGNITE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-19267:
---
Description:
According to the IEP-91, we can delete old data, once it becomes older than the
certain threshold. At the moment, we can consider this threshold to be shared
between different tables, but be unique on individual nodes. It's called a Low
Watermark (LW).
The way the value is chosen is the following:
* There's the {_}data availability time{_}, that can be configured by the
user. This is a cluster configuration. It has a value of, for example, 45
minutes. Valid values - {{{}[0, +INF){}}}.
* There's a {_}GC frequency{_}, that's also a cluster configuration. For
example, 5 minuter. Range of valid values should be more strict.
* Last LW value is persisted in Vault.
* Every 5 minutes, we assign a new "lwCandidate{{{} = now() - 45min -
maxClockSkew"{}}}.
** If there are no running transactions with timestamp below
{{{}lwCandidate{}}}, we promote the candidate into a real LW value.
** Otherwise, we trigger GC with timestamp of the transaction with the oldest
timestamp (and promoting LW to that timestamp), and raising the bar every time*
that transaction is completed. Eventually, we will reach the point where there
are no running transactions with timestamp below {{{}lwCandidate{}}}.
* it's not necessary to do it every time. But, once the timestamp of the
oldest RO transaction is above or equal to {{{}lwCandidate{}}}, we must
guarantee its promotion. Everything else is optimization.
* If there's a new RO transaction with timestamp below {{{}lwCandidate{}}}, we
fail it.
Promoted LW value cannot become smaller no matter what. All data below LW is
considered to be invalid, maybe broken and completely invisible to user.
was:
According to the IEP-91, we can delete old data, once it becomes older than the
certain threshold. At the moment, we can consider this threshold to be shared
between different tables, but be unique on individual nodes. It's called a Low
Watermark (LW).
The way the value is chosen is the following:
* There's the {_}data availability time{_}, that can be configured by the
user. This is a cluster configuration. It has a value of, for example, 45
minutes. Valid values - {{{}[0, +INF){}}}.
* There's a {_}GC frequency{_}, that's also a cluster configuration. For
example, 5 minuter. Range of valid values should be more strict.
* Last LW value is persisted in Vault.
* Every 5 minutes, we assign a new "lwCandidate{{{} = now() - 45min -
maxClockSkew"{}}}.
** If there are no running transactions with timestamp below
{{{}lwCandidate{}}}, we promote the candidate into a real LW value.
** Otherwise, we trigger GC with timestamp of the transaction with the oldest
timestamp (and promoting LW to that timestamp), and raising the bar every time*
that transaction is completed. Eventually, we will reach the point where there
are running transactions with timestamp below {{{}lwCandidate{}}}.
* it's not necessary to do it every time. But, once the timestamp of the oldest
RO transaction is above or equal to {{{}lwCandidate{}}}, we must guarantee its
promotion. Everything else is optimization.
* If there's a new RO transaction with timestamp below {{{}lwCandidate{}}}, we
fail it.
Promoted LW value cannot become smaller no matter what. All data below LW is
considered to be invalid, maybe broken and completely invisible to user.
> Implement local Low Watermark propagation
> -
>
> Key: IGNITE-19267
> URL: https://issues.apache.org/jira/browse/IGNITE-19267
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> According to the IEP-91, we can delete old data, once it becomes older than
> the certain threshold. At the moment, we can consider this threshold to be
> shared between different tables, but be unique on individual nodes. It's
> called a Low Watermark (LW).
> The way the value is chosen is the following:
> * There's the {_}data availability time{_}, that can be configured by the
> user. This is a cluster configuration. It has a value of, for example, 45
> minutes. Valid values - {{{}[0, +INF){}}}.
> * There's a {_}GC frequency{_}, that's also a cluster configuration. For
> example, 5 minuter. Range of valid values should be more strict.
> * Last LW value is persisted in Vault.
> * Every 5 minutes, we assign a new "lwCandidate{{{} = now() - 45min -
> maxClockSkew"{}}}.
> ** If there are no running transactions with timestamp below
> {{{}lwCandidate{}}}, we promote the candidate into a real LW value.
> ** Otherwise, we trigger GC with timestamp of the transaction with the
> oldest timestamp (and promoting LW to that
[jira] [Commented] (IGNITE-19264) Simplify C++ Binary Tuple code
[ https://issues.apache.org/jira/browse/IGNITE-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710871#comment-17710871 ] Igor Sapego commented on IGNITE-19264: -- Looks good to me. > Simplify C++ Binary Tuple code > -- > > Key: IGNITE-19264 > URL: https://issues.apache.org/jira/browse/IGNITE-19264 > Project: Ignite > Issue Type: Improvement > Components: platforms >Affects Versions: 3.0.0-beta2 >Reporter: Aleksey Demakov >Assignee: Aleksey Demakov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Some binary tuple code turned out to be useless for the C++ client lib. It is > better to be removed to simplify maintenance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19264) Simplify C++ Binary Tuple code
[
https://issues.apache.org/jira/browse/IGNITE-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710870#comment-17710870
]
Aleksey Demakov commented on IGNITE-19264:
--
{color:#1d1c1d}
{color}{color:#1d1c1d}[@isapego|https://gridgain.slack.com/team/U2WS56DB2]{color},
please review my ticket.
> Simplify C++ Binary Tuple code
> --
>
> Key: IGNITE-19264
> URL: https://issues.apache.org/jira/browse/IGNITE-19264
> Project: Ignite
> Issue Type: Improvement
> Components: platforms
>Affects Versions: 3.0.0-beta2
>Reporter: Aleksey Demakov
>Assignee: Aleksey Demakov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some binary tuple code turned out to be useless for the C++ client lib. It is
> better to be removed to simplify maintenance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-18899) Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to MutableRowTupleAdapter
[ https://issues.apache.org/jira/browse/IGNITE-18899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710867#comment-17710867 ] Igor Sapego commented on IGNITE-18899: -- Looks good to me. > Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to > MutableRowTupleAdapter > > > Key: IGNITE-18899 > URL: https://issues.apache.org/jira/browse/IGNITE-18899 > Project: Ignite > Issue Type: Improvement > Components: thin client >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > * Client protocol uses *BinaryTuple* to exchange data (IGNITE-17297) > * When reading data from server, currently we unpack all elements of the > incoming *BinaryTuple* into *ClientTuple* > This is extra work and extra allocations. Instead, wrap the incoming > *BinaryTuple* like *MutableRowTupleAdapter* does. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-19270) Thin 3.0: Streamline type codes
Pavel Tupitsyn created IGNITE-19270: --- Summary: Thin 3.0: Streamline type codes Key: IGNITE-19270 URL: https://issues.apache.org/jira/browse/IGNITE-19270 Project: Ignite Issue Type: Improvement Components: thin client Affects Versions: 3.0.0-beta1 Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 3.0.0-beta2 We have too many different type codes: * org.apache.ignite.sql.ColumnType * org.apache.ignite.internal.client.proto.ClientDataType * org.apache.ignite.internal.client.proto.ClientColumnTypeConverter (uses it's own codes) We can probably replace everything with *org.apache.ignite.sql.ColumnType* and use ordinals from there. .NET and C++ tests will protect us from breaking the protocol accidentally. And we can write an additional test that explains why you can't change ordinals there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-19185) Fix index destruction after index creation started
[
https://issues.apache.org/jira/browse/IGNITE-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710860#comment-17710860
]
Semyon Danilov commented on IGNITE-19185:
-
The patch looks awesome to me!
> Fix index destruction after index creation started
> --
>
> Key: IGNITE-19185
> URL: https://issues.apache.org/jira/browse/IGNITE-19185
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It was found that if the destruction of the index occurs immediately after
> the start of its creation, then errors occur during its building:
> *org.apache.ignite.internal.sql.api.ItSqlSynchronousApiTest#ddl*
> {code:java}
> 2023-04-02 13:53:16:882 +0300 [INFO][%issat_n_0%build-index-0][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=14, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:883 +0300 [INFO][%issat_n_0%build-index-6][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=8, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:884 +0300 [INFO][vault0][IndexManager] Index dropped
> [schema=PUBLIC, index=TEST_IDX]
> 2023-04-02 13:53:16:885 +0300 [ERROR][%issat_n_0%build-index-6][IndexBuilder]
> Index build error: [table=TEST, tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa,
> partitionId=15, index=TEST_IDX, indexId=55e899e3-5404-4409-8570-aad4459c2908]
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:f9741480-3eaf-4c32-8220-66e9a9118d48 Index configuration for
> "55e899e3-5404-4409-8570-aad4459c2908" could not be found
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:f9741480-3eaf-4c32-8220-66e9a9118d48 Index configuration for
> "55e899e3-5404-4409-8570-aad4459c2908" could not be found
> at
> org.apache.ignite.internal.storage.engine.MvTableStorage.getOrCreateIndex(MvTableStorage.java:94)
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.collectRowIdBatch(IndexBuilder.java:258)
> at
> org.apache.ignite.internal.index.IndexBuilder$BuildIndexTask.lambda$run$1(IndexBuilder.java:164)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> ... 4 more
> 2023-04-02 13:53:16:883 +0300 [INFO][%issat_n_0%build-index-2][IndexBuilder]
> Start building the index: [table=TEST,
> tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa, partitionId=1, index=TEST_IDX,
> indexId=55e899e3-5404-4409-8570-aad4459c2908]
> 2023-04-02 13:53:16:887 +0300 [ERROR][%issat_n_0%build-index-6][IndexBuilder]
> Index build error: [table=TEST, tableId=8370aa11-b3ea-4e00-a546-0776ff12fffa,
> partitionId=18, index=TEST_IDX, indexId=55e899e3-5404-4409-8570-aad4459c2908]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (IGNITE-18899) Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to MutableRowTupleAdapter
[ https://issues.apache.org/jira/browse/IGNITE-18899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710856#comment-17710856 ] Igor Sapego edited comment on IGNITE-18899 at 4/11/23 9:55 AM: --- Left a few comments. was (Author: isapego): Left few comments. > Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to > MutableRowTupleAdapter > > > Key: IGNITE-18899 > URL: https://issues.apache.org/jira/browse/IGNITE-18899 > Project: Ignite > Issue Type: Improvement > Components: thin client >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Client protocol uses *BinaryTuple* to exchange data (IGNITE-17297) > * When reading data from server, currently we unpack all elements of the > incoming *BinaryTuple* into *ClientTuple* > This is extra work and extra allocations. Instead, wrap the incoming > *BinaryTuple* like *MutableRowTupleAdapter* does. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18899) Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to MutableRowTupleAdapter
[ https://issues.apache.org/jira/browse/IGNITE-18899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710856#comment-17710856 ] Igor Sapego commented on IGNITE-18899: -- Left few comments. > Thin 3.0: Refactor ClientTuple to wrap BinaryTuple similar to > MutableRowTupleAdapter > > > Key: IGNITE-18899 > URL: https://issues.apache.org/jira/browse/IGNITE-18899 > Project: Ignite > Issue Type: Improvement > Components: thin client >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > * Client protocol uses *BinaryTuple* to exchange data (IGNITE-17297) > * When reading data from server, currently we unpack all elements of the > incoming *BinaryTuple* into *ClientTuple* > This is extra work and extra allocations. Instead, wrap the incoming > *BinaryTuple* like *MutableRowTupleAdapter* does. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-18871) Sql. ColocatedSortAggregate rule can't use index for grouping.
[
https://issues.apache.org/jira/browse/IGNITE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Orlov reassigned IGNITE-18871:
-
Assignee: Konstantin Orlov
> Sql. ColocatedSortAggregate rule can't use index for grouping.
> --
>
> Key: IGNITE-18871
> URL: https://issues.apache.org/jira/browse/IGNITE-18871
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Andrey Mashenkov
>Assignee: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Assume there is a table with a composite index on (grp0, grp1) columns.
> ColocatedSortAggregate rule can’t use index for the next query in case of
> hash distribution:
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp1, grp0
> {code}
> However, index is for the query used in case of single distribution.
> Also, index is used for the next query in both cases: single and hash
> distributions.
> {code:java}
> SELECT AVG(val0) FROM test GROUP BY grp0, grp1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-19269) Introduce proper local RowId locks
Ivan Bessonov created IGNITE-19269:
--
Summary: Introduce proper local RowId locks
Key: IGNITE-19269
URL: https://issues.apache.org/jira/browse/IGNITE-19269
Project: Ignite
Issue Type: Improvement
Reporter: Ivan Bessonov
Currently, concurrency in MV storages is a bit convoluted. List of issues
include:
* Implicit assumption that one does not insert the same RowId concurrently.
There's a chance that this assumption is already violated. This could lead to
B+Tree corruptions, for example.
* Hidden locks, that cause non-trivial bugs like this:
https://issues.apache.org/jira/browse/IGNITE-19169
* MvPartitionStorage#scanVersions implementation for B+Tree reads the entire
chain at once, because we couldn't figure out how to do it properly in
concurrent environment. This may lead to drastic performance drops in the
future.
* GC code is complicated and sub-optimal. We need to implement the same
approach that's used in Ignite 2.x - read, lock, dequeue, cleanup, unlock*. We
can also cache part of the queue beforehand, making the process faster. It is
not possible in current implementation.
* Maybe something else that I couldn't remember.
h3. Proposed solution
As wise man once said, "{_}Explicit is better than implicit{_}".
I propose having an explicit concurrency model akin Ignite 2.x's. We need to
introduce 2 new methods to MV partition storage interface.
{code:java}
/**
* Synchronously locks passed RowId.
*/
void lock(RowId rowId);
/**
* Tries to synchronously lock passed RowId.
* Used to implement deadlock prevention.
*/
boolean tryLock(RowId rowId);{code}
These methods should delegate locking to the client, basically exposing the
{{ReentrantLockByRowId}} API. Expected invariants:
* Locks can only be allowed inside of the "runConsistently" closure.
* Unlock is still supposed to happen at the end of "runConsistently",
automatically. This is predicated by the way RocksDB implementation works
(batches).
Client must guarantee, that there are no deadlocks, either by sorting the data
or by using "tryLock".
* All read operations, except for "scanVersions", may be executed without
locks.
* "scanVersions" must have a lock, because it's not atomic. This is not a big
deal, because the only two usages are:
** index cleanup, we already hold a lock
** rebalance, we already hold another lock, so adding a new one is not a
problem
* "scanVersions" implementation must also be fixed. Maybe we need to add a
hint, like "loadAllVersions", but it's not required.
This approach, with proper implementation, solves all mentioned problems,
except for GC.
h3. Changes to GC
As stated before, GC flow must be changed to something like this:
{code:java}
var entry = partition.peek(lw);
if (entry == null) {
return;
}
if (!partition.tryLock(entry.rowId())) {
return; // Must exit closure to avoid deadlocks.
}
if (!partition.dequeue(entry)) {
continue; // Or return. We may or may not exit the closure
// in this case, both options are valid.
}
cleanupIndexes(entry);
continue;{code}
Right now, all storage implementations have similar construction inside of
them. All I propose is to move it outside.
h3. Tangent changes
org.apache.ignite.internal.table.distributed.raft.PartitionDataStorage is
basically a copy of the original, except for a few methods. Maybe copied
methods should exist in a shared super-interface. It's up to developer to
decide.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17892) Ignite doc: cdcEnabled parameter is in the DataRegionConfiguration
[ https://issues.apache.org/jira/browse/IGNITE-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Plekhanov updated IGNITE-17892: --- Ignite Flags: (was: Docs Required,Release Notes Required) > Ignite doc: cdcEnabled parameter is in the DataRegionConfiguration > -- > > Key: IGNITE-17892 > URL: https://issues.apache.org/jira/browse/IGNITE-17892 > Project: Ignite > Issue Type: Bug > Components: documentation >Affects Versions: 2.14 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Minor > Fix For: 2.15 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://ignite.apache.org/docs/latest/persistence/change-data-capture|https://ignite.apache.org/docs/latest/persistence/change-data-capture#ignite-node] > DataStorageConfiguration#cdcEnabled > should be: > DataRegionConfiguration#cdcEnabled -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18260) Ignite Doc: CPP thin client has the continuous query function, which needs to be marked in the overview
[ https://issues.apache.org/jira/browse/IGNITE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710827#comment-17710827 ] Aleksey Plekhanov commented on IGNITE-18260: [~liyuj] looks good to me, merged to master, cherry-picked to 2.15. Thanks for the contribution! > Ignite Doc: CPP thin client has the continuous query function, which needs to > be marked in the overview > --- > > Key: IGNITE-18260 > URL: https://issues.apache.org/jira/browse/IGNITE-18260 > Project: Ignite > Issue Type: Improvement > Components: documentation >Affects Versions: 2.13 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Trivial > Fix For: 2.15 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18260) Ignite Doc: CPP thin client has the continuous query function, which needs to be marked in the overview
[ https://issues.apache.org/jira/browse/IGNITE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710821#comment-17710821 ] YuJue Li commented on IGNITE-18260: --- [~alex_pl] please check again. > Ignite Doc: CPP thin client has the continuous query function, which needs to > be marked in the overview > --- > > Key: IGNITE-18260 > URL: https://issues.apache.org/jira/browse/IGNITE-18260 > Project: Ignite > Issue Type: Improvement > Components: documentation >Affects Versions: 2.13 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Trivial > Fix For: 2.15 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18260) Ignite Doc: CPP thin client has the continuous query function, which needs to be marked in the overview
[ https://issues.apache.org/jira/browse/IGNITE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710785#comment-17710785 ] Aleksey Plekhanov commented on IGNITE-18260: [~liyuj], can you please fix two more lines in this patch? "Async operations" was implemented for java thin client by IGNITE-7623 and "Server discovery" for java thin client was implemented by IGNITE-18591. > Ignite Doc: CPP thin client has the continuous query function, which needs to > be marked in the overview > --- > > Key: IGNITE-18260 > URL: https://issues.apache.org/jira/browse/IGNITE-18260 > Project: Ignite > Issue Type: Improvement > Components: documentation >Affects Versions: 2.13 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Trivial > Fix For: 2.15 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
