[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822359#comment-17822359
]
Ran Tao commented on CALCITE-6285:
--
Yes, it should be a bugfix.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822356#comment-17822356
]
Mihai Budiu commented on CALCITE-6285:
--
I would say "correct". "Optimize" would imply that it's too inefficient.
If this was a bug in Spark, it means it's now a bug in Calcite.
I expect this will be an easy fix.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822355#comment-17822355
]
Ran Tao commented on CALCITE-6285:
--
Since spark-3.4.2 and the latest 3.5.0 version have been fixed, I think we
should optimize the implementation of calcite, what do you think?
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822354#comment-17822354
]
Mihai Budiu commented on CALCITE-6285:
--
[~taoran] this makes sense, I will let you decide what should happen to this
issue. If you think it should be closed, please go ahead. If you think we
should fix the problem, please assign me or yourself to the issue.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822353#comment-17822353
]
Ran Tao commented on CALCITE-6285:
--
[~mbudiu]the calcite implementation is compatibled with <= spark-3.4.1.
see:
{code:java}
spark-sql (default)> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
[4,5,3,2,1]
{code}
When I implemented it, the latest version was 3.4.1, and now 3.4.2 and 3.5.0
have fixes.
More background: This function is not Spark's own native function, but is used
to benchmark Snowflake. This function has not been particularly stable. FYI:
https://issues.apache.org/jira/browse/SPARK-45078
https://lists.apache.org/thread/1p5hkql96k5qc5ww6wkd7mq6qdbgyz1n
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822351#comment-17822351
]
Mihai Budiu commented on CALCITE-6285:
--
True, but many people don't always upgrade to the latest version of the
software, sometime exactly because they don't want the behavior of programs to
change unexpectedly. So some users of Calcite may be using Spark 3.4.0.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822349#comment-17822349
]
EveyWu commented on CALCITE-6285:
--
I agree with you
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822347#comment-17822347
]
EveyWu commented on CALCITE-6285:
--
Another simpler way, can we keep consistent with the latest implementation of
spark? In practice, calculations based on spark Function are more likely to be
executed on spark engine, calcite only provides an alternative execution.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822348#comment-17822348
]
Mihai Budiu commented on CALCITE-6285:
--
https://issues.apache.org/jira/browse/SPARK-44840 suggests that this is a
bugfix.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822346#comment-17822346
]
Mihai Budiu commented on CALCITE-6285:
--
Spark didn't bump the minor version for this change, so I am inferring that
this is a bugfix rather than a change of definition. So perhaps Calcite should
follow 3.4.2.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822344#comment-17822344
]
Mihai Budiu commented on CALCITE-6285:
--
The only solution I can imagine is having two spark dialects. Or more, if they
keep changing definitions.
Frankly, this is something Spark should not have done, they should have created
a new function.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822343#comment-17822343
]
EveyWu commented on CALCITE-6285:
--
I think the Spark version update causes the problem, -1 for which the function
inserts a new element after the current last element.
The original definition [3.4.0,
|https://spark.apache.org/docs/3.4.0/api/sql/index.html#array_insert]Updated
version definition
[3.4.2|https://spark.apache.org/docs/3.4.2/api/sql/index.html#array_insert]
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822341#comment-17822341
]
Mihai Budiu commented on CALCITE-6285:
--
Is this the image you intended to attach?
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
> Attachments: image-2024-03-01-10-31-55-853.png
>
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6285) Function ARRAY_INSERT produces an incorrect result for negative indices
[
https://issues.apache.org/jira/browse/CALCITE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822256#comment-17822256
]
Mihai Budiu commented on CALCITE-6285:
--
[~taoran] you seem to have added these tests not that long ago.
> Function ARRAY_INSERT produces an incorrect result for negative indices
> ---
>
> Key: CALCITE-6285
> URL: https://issues.apache.org/jira/browse/CALCITE-6285
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Mihai Budiu
>Priority: Minor
>
> Here is a test taken from the Spark documentation page:
> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert
> {code}
> SELECT array_insert(array(5, 3, 2, 1), -4, 4);
> [5,4,3,2,1]
> {code}
> The result produced by Calcite is:
> [4,5,3,2,1]
> The strange thing is that there are tests for negative indices. I wonder if
> the original tests are wrong, or the behavior of this function in Spark was
> changed since the tests were written.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
