[jira] [Commented] (CALCITE-4194) Materialized recognition failed when query contains cast type conversion
[
https://issues.apache.org/jira/browse/CALCITE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185422#comment-17185422
]
Julian Hyde commented on CALCITE-4194:
--
Am I correct in saying that this matching is only valid because the cast to
BIGINT is non-lossy? If so, the fix must include a check whether the cast is
lossy.
> Materialized recognition failed when query contains cast type conversion
>
>
> Key: CALCITE-4194
> URL: https://issues.apache.org/jira/browse/CALCITE-4194
> Project: Calcite
> Issue Type: Wish
>Reporter: xzh_dz
>Priority: Major
>
> There is a semantic equivalent case, which should be able to materialize and
> identify successfully, but failed. This case can be reproduced as below.
> {code:java}
> // MaterializedViewSubstitutionVisitorTest
> @Test void testCastMvMatch() {
> sql(
> "select \"empid\", count(\"deptno\") from \"emps\" where \"empid\" > 10
> group by \"empid\"",
> "select \"empid\", count(cast(\"deptno\" as bigint)) from \"emps\"
> where \"empid\" > 10 group by \"empid\"")
> .ok();
> }
> {code}
> Here is the error message：
>
> {code:java}
> java.lang.AssertionError: Materialized view failed to be matched by optimized
> results:java.lang.AssertionError: Materialized view failed to be matched by
> optimized results: at
> org.apache.calcite.test.AbstractMaterializedViewTest.checkMaterialize(AbstractMaterializedViewTest.java:112)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest.access$000(AbstractMaterializedViewTest.java:64)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest$Sql.ok(AbstractMaterializedViewTest.java:227)
> at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
> at
> org.apache.calcite.util.ImmutableBeans.lambda$makeDef$3(ImmutableBeans.java:275)
> at
> org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:446)
> at com.sun.proxy.$Proxy12.ok(Unknown Source) at
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest.testCastMvMatch(MaterializedViewSubstitutionVisitorTest.java:124)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
>
[jira] [Created] (CALCITE-4196) Avatica server responds with HTTP/401 prior to consuming all data written by client
Josh Elser created CALCITE-4196:
---
Summary: Avatica server responds with HTTP/401 prior to consuming
all data written by client
Key: CALCITE-4196
URL: https://issues.apache.org/jira/browse/CALCITE-4196
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: avatica-1.18.0
First off, big thanks to [~krisden] for pointing me to HIVE-22231 which was the
similar problem on the Hive side.
Symptoms: the client, when sending a large HTTP request to the Avatica server
which is configured for SPNEGO authentication, e.g. an ExecuteBatchRequest with
100's to 1000's of rows to execute, will receive an HTTP/401 response as a part
of the normal SPNEGO negotiation (described in
[https://tools.ietf.org/html/rfc4559#section-5]). The client will observe an
error similar to the following, indicate "Broken pipe".
{noformat}
2020-08-24 17:21:54,512 DEBUG http.wire: http-outgoing-1 >> "[write] I/O error:
Broken pipe (Write failed)"
2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection:
http-outgoing-1: Close connection
2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection:
http-outgoing-1: Shutdown connection
2020-08-24 17:21:54,512 DEBUG execchain.MainClientExec: Connection discarded
2020-08-24 17:21:54,512 DEBUG conn.PoolingHttpClientConnectionManager:
Connection released: [id: 1][route: {}->http://avatica-server:8765][total kept
alive: 0; route allocated: 0 of 25; total allocated: 0 of 100]
2020-08-24 17:21:54,512 INFO execchain.RetryExec: I/O exception
(java.net.SocketException) caught when processing request to
{}->http://avatica-server:8765: Broken pipe (Write failed)
2020-08-24 17:21:54,512 DEBUG execchain.RetryExec: Broken pipe (Write failed)
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at
org.apache.calcite.avatica.org.apache.http.impl.conn.LoggingOutputStream.write(LoggingOutputStream.java:74)
at
org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at
org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
at
org.apache.calcite.avatica.org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at
org.apache.calcite.avatica.org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112)
at
org.apache.calcite.avatica.org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at
org.apache.calcite.avatica.org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
at
org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at
org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.calcite.avatica.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.calcite.avatica.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
org.apache.calcite.avatica.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.calcite.avatica.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.calcite.avatica.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.calcite.avatica.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:129)
at
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient$1.run(DoAsAvaticaHttpClient.java:39)
at
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient$1.run(DoAsAvaticaHttpClient.java:37)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient.send(DoAsAvaticaHttpClient.java:37)
at
org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
at
org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:117)
at
org.apache.calcite.avatica.remote.RemoteMeta$20.call(RemoteMeta.java:430)
at
org.apache.calcite.avatica.remote.RemoteMeta$20.call(RemoteMeta.java:427)
at
[jira] [Commented] (CALCITE-4180) Where to set elasticsearch username/password?
[
https://issues.apache.org/jira/browse/CALCITE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185362#comment-17185362
]
Rui Wang commented on CALCITE-4180:
---
I see. [~geiguanbing]are you interested in contribute authentication code to
Calcite?
> Where to set elasticsearch username/password?
> -
>
> Key: CALCITE-4180
> URL: https://issues.apache.org/jira/browse/CALCITE-4180
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: fa
>Priority: Major
>
>
> {code:java}
> Properties config = new Properties();
> config.put("lex", "JAVA");
> String sql = "select * from index";
> try (Connection con =
> DriverManager.getConnection("jdbc:calcite:model=src/main/resources/es.json",
> config)) {
> try (Statement stmt = con.createStatement()) {
> try (ResultSet rs = stmt.executeQuery(sql)) {
> printRs(rs);
> }
> }
> }
> {code}
>
>
> es.json
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "elasticsearch",
> "schemas": [
> {
> "type": "custom",
> "name": "elasticsearch",
> "factory":
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
> "operand": {
> "coordinates": "{'192.168.133.104': 9200}",
> "jdbcUser": "elastic",
> "jdbcPassword": "elastic"
> }
> }
> ]
> }
> {code}
> and throw Exception
> {code:java}
> {
> "error":{
> "root_cause":[
> {
> "type":"security_exception",
> "reason":"missing authentication token for REST request
> [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security"
> charset="UTF-8""
> }
> }
> ],
> "type":"security_exception",
> "reason":"missing authentication token for REST request [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security" charset="UTF-8""
> }
> },
> "status":401
> }
> {code}
> Where to set Elasticsearch username/password?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4195) Cast between types with different collators must be evaluated as not monotonic
[ https://issues.apache.org/jira/browse/CALCITE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-4195: --- Summary: Cast between types with different collators must be evaluated as not monotonic (was: Cast from character to character with a different collator must be evaluated as not monotonic) > Cast between types with different collators must be evaluated as not monotonic > -- > > Key: CALCITE-4195 > URL: https://issues.apache.org/jira/browse/CALCITE-4195 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.24.0 >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Minor > Labels: pull-request-available > Fix For: 1.26.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Problem related to the feature CALCITE-3951. > A cast operation from a character type to another character type that has a > different collator must be evaluated as not monotonic. Currently this is not > the case. > This has an impact e.g. in SortProjectTransposeRule, since we can incorrectly > transpose a sort after a projection than contains such a cast, and we end up > with a plan that will return a different result than the original one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4195) Cast from character to character with a different collator must be evaluated as not monotonic
[ https://issues.apache.org/jira/browse/CALCITE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-4195: Labels: pull-request-available (was: ) > Cast from character to character with a different collator must be evaluated > as not monotonic > - > > Key: CALCITE-4195 > URL: https://issues.apache.org/jira/browse/CALCITE-4195 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.24.0 >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Minor > Labels: pull-request-available > Fix For: 1.26.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Problem related to the feature CALCITE-3951. > A cast operation from a character type to another character type that has a > different collator must be evaluated as not monotonic. Currently this is not > the case. > This has an impact e.g. in SortProjectTransposeRule, since we can incorrectly > transpose a sort after a projection than contains such a cast, and we end up > with a plan that will return a different result than the original one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4195) Cast from character to character with a different collator must be evaluated as not monotonic
[ https://issues.apache.org/jira/browse/CALCITE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L reassigned CALCITE-4195: -- Assignee: Ruben Q L > Cast from character to character with a different collator must be evaluated > as not monotonic > - > > Key: CALCITE-4195 > URL: https://issues.apache.org/jira/browse/CALCITE-4195 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.24.0 >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Minor > Fix For: 1.26.0 > > > Problem related to the feature CALCITE-3951. > A cast operation from a character type to another character type that has a > different collator must be evaluated as not monotonic. Currently this is not > the case. > This has an impact e.g. in SortProjectTransposeRule, since we can incorrectly > transpose a sort after a projection than contains such a cast, and we end up > with a plan that will return a different result than the original one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4195) Cast from character to character with a different collator must be evaluated as not monotonic
[ https://issues.apache.org/jira/browse/CALCITE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-4195: --- Summary: Cast from character to character with a different collator must be evaluated as not monotonic (was: Cast from character to character with a different collator is not monotonic) > Cast from character to character with a different collator must be evaluated > as not monotonic > - > > Key: CALCITE-4195 > URL: https://issues.apache.org/jira/browse/CALCITE-4195 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.24.0 >Reporter: Ruben Q L >Priority: Minor > Fix For: 1.26.0 > > > Problem related to the feature CALCITE-3951. > A cast operation from a character type to another character type that has a > different collator must be evaluated as not monotonic. Currently this is not > the case. > This has an impact e.g. in SortProjectTransposeRule, since we can incorrectly > transpose a sort after a projection than contains such a cast, and we end up > with a plan that will return a different result than the original one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4195) Cast from character to character with a different collator is not monotonic
Ruben Q L created CALCITE-4195: -- Summary: Cast from character to character with a different collator is not monotonic Key: CALCITE-4195 URL: https://issues.apache.org/jira/browse/CALCITE-4195 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.24.0 Reporter: Ruben Q L Fix For: 1.26.0 Problem related to the feature CALCITE-3951. A cast operation from a character type to another character type that has a different collator must be evaluated as not monotonic. Currently this is not the case. This has an impact e.g. in SortProjectTransposeRule, since we can incorrectly transpose a sort after a projection than contains such a cast, and we end up with a plan that will return a different result than the original one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4180) Where to set elasticsearch username/password?
[
https://issues.apache.org/jira/browse/CALCITE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185114#comment-17185114
]
fa edited comment on CALCITE-4180 at 8/26/20, 11:23 AM:
I found the reason at
org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory#connect
{code:java}
private static RestClient connect(List hosts, String pathPrefix) {
Objects.requireNonNull(hosts, "hosts or coordinates");
Preconditions.checkArgument(!hosts.isEmpty(), "no ES hosts specified");
RestClientBuilder builder = RestClient.builder(hosts.toArray(new
HttpHost[hosts.size()]));
if (pathPrefix != null && !pathPrefix.isEmpty()) {
builder.setPathPrefix(pathPrefix);
}
return builder.build();
}{code}
As shown in the code above, there is nothing authentication in that.
was (Author: geiguanbing):
I found the reason at
`org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory#connect`
```java
private static RestClient connect(List hosts, String pathPrefix) {
Objects.requireNonNull(hosts, "hosts or coordinates");
Preconditions.checkArgument(!hosts.isEmpty(), "no ES hosts specified");
RestClientBuilder builder = RestClient.builder(hosts.toArray(new
HttpHost[hosts.size()]));
if (pathPrefix != null && !pathPrefix.isEmpty()) {
builder.setPathPrefix(pathPrefix);
}
return builder.build();
}
```
As shown in the code above, there is nothing authentication in that.
> Where to set elasticsearch username/password?
> -
>
> Key: CALCITE-4180
> URL: https://issues.apache.org/jira/browse/CALCITE-4180
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: fa
>Priority: Major
>
>
> {code:java}
> Properties config = new Properties();
> config.put("lex", "JAVA");
> String sql = "select * from index";
> try (Connection con =
> DriverManager.getConnection("jdbc:calcite:model=src/main/resources/es.json",
> config)) {
> try (Statement stmt = con.createStatement()) {
> try (ResultSet rs = stmt.executeQuery(sql)) {
> printRs(rs);
> }
> }
> }
> {code}
>
>
> es.json
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "elasticsearch",
> "schemas": [
> {
> "type": "custom",
> "name": "elasticsearch",
> "factory":
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
> "operand": {
> "coordinates": "{'192.168.133.104': 9200}",
> "jdbcUser": "elastic",
> "jdbcPassword": "elastic"
> }
> }
> ]
> }
> {code}
> and throw Exception
> {code:java}
> {
> "error":{
> "root_cause":[
> {
> "type":"security_exception",
> "reason":"missing authentication token for REST request
> [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security"
> charset="UTF-8""
> }
> }
> ],
> "type":"security_exception",
> "reason":"missing authentication token for REST request [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security" charset="UTF-8""
> }
> },
> "status":401
> }
> {code}
> Where to set Elasticsearch username/password?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4191) Improve the logic of creating aggregate calls
[
https://issues.apache.org/jira/browse/CALCITE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185096#comment-17185096
]
Liya Fan commented on CALCITE-4191:
---
[~Chunwei Lei] Thanks for your feedback.
It seems the indirect way always generates the AggregateCall along with the
Aggregate rel node. For scenarios where we only need the AggregateCall objects
(e.g. Window#getAggregateCalls), it is more convenient to use the
Aggregate#create methods.
> Improve the logic of creating aggregate calls
> -
>
> Key: CALCITE-4191
> URL: https://issues.apache.org/jira/browse/CALCITE-4191
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> According to the current code base, the only way to create \{{AggregateCall}}
> objects is by calling one of the two \{{AggregateCall#create}} methods (other
> create methods are deprecated).
> The two {{create}} methods have 9 and 11 parameters, respectively, 3 of which
> are booleans and 2 are ints. We find this makes the code less readable and
> error-prone, as some bugs are caused by specifying the wrong parameters.
> In this issue, we improve the related logic by the builder pattern, which
> results in the following benefits:
> 1. By creating the objects by the builder pattern, there is no need to
> maintain multiple overrides of the {{create}} methods.
> 2. There is no need to maintain multiple overrides of the {{copy}} methods,
> either.
> 3. The code becomes more readable and less error-prone, as it is less like to
> specify the wrong parameter.
> 4. Creating {{AggregateCall}} objects becomes easier, as the user does not
> have specify the default parameters repeatedly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4194) Materialized recognition failed when query contains cast type conversion
xzh_dz created CALCITE-4194:
---
Summary: Materialized recognition failed when query contains cast
type conversion
Key: CALCITE-4194
URL: https://issues.apache.org/jira/browse/CALCITE-4194
Project: Calcite
Issue Type: Wish
Reporter: xzh_dz
There is a semantic equivalent case, which should be able to materialize and
identify successfully, but failed. This case can be reproduced as below.
{code:java}
// MaterializedViewSubstitutionVisitorTest
@Test void testCastMvMatch() {
sql(
"select \"empid\", count(\"deptno\") from \"emps\" where \"empid\" > 10
group by \"empid\"",
"select \"empid\", count(cast(\"deptno\" as bigint)) from \"emps\" where
\"empid\" > 10 group by \"empid\"")
.ok();
}
{code}
Here is the error message：
{code:java}
java.lang.AssertionError: Materialized view failed to be matched by optimized
results:java.lang.AssertionError: Materialized view failed to be matched by
optimized results: at
org.apache.calcite.test.AbstractMaterializedViewTest.checkMaterialize(AbstractMaterializedViewTest.java:112)
at
org.apache.calcite.test.AbstractMaterializedViewTest.access$000(AbstractMaterializedViewTest.java:64)
at
org.apache.calcite.test.AbstractMaterializedViewTest$Sql.ok(AbstractMaterializedViewTest.java:227)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at
org.apache.calcite.util.ImmutableBeans.lambda$makeDef$3(ImmutableBeans.java:275)
at
org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:446)
at com.sun.proxy.$Proxy12.ok(Unknown Source) at
org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest.testCastMvMatch(MaterializedViewSubstitutionVisitorTest.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at
[jira] [Updated] (CALCITE-4193) Implement new sort operator: ExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-4193: --- Description: Sometimes we need to sort a big volume of data which does not fit into memory. In this situation EnumerableSort will cause an OutOfMemoryError. The solution for such a scenario will be using a different sorting algorithm: [External Sort|https://en.wikipedia.org/wiki/External_sorting]. The goal of the current ticket is to implement a new operator (ExternalSort) to provide this feature. In principle, the Enumerable convention is not suitable for this operator (i.e. a theoretical EnumerableExternalSort), since this convention was designed for in-memory data and we have no mechanism to serialize / deserialize values. As suggested by [~julianhyde], we ought to be putting our effort into building a convention with a byte-oriented data format. was: Sometimes we need to sort a big volume of data which does not fit into memory. In this situation EnumerableSort will cause an OutOfMemoryError. The solution for such a scenario will be using a different sorting algorithm: [External Sort|https://en.wikipedia.org/wiki/External_sorting]. The goal of the current ticket is to implement a new operator (EnumerableExternalSort) to provide this feature. > Implement new sort operator: ExternalSort > - > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. The > solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. The goal of > the current ticket is to implement a new operator (ExternalSort) to provide > this feature. > In principle, the Enumerable convention is not suitable for this operator > (i.e. a theoretical EnumerableExternalSort), since this convention was > designed for in-memory data and we have no mechanism to serialize / > deserialize values. As suggested by [~julianhyde], we ought to be putting our > effort into building a convention with a byte-oriented data format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[
https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184985#comment-17184985
]
Ruben Q L commented on CALCITE-4193:
{quote}The Enumerable convention was designed for in-memory data, on the JVM
heap. It wasn’t intended for external algorithms.
In particular, we have no mechanism to serialize and deserialize values, and no
on-disk format, and no scheduler that can handle asynchronous i/o.
{quote}
You are right [~julianhyde], I had not looked at the implementation details. I
am going to adapt the ticket's description accordingly.
> Implement new sort operator: EnumerableExternalSort
> ---
>
> Key: CALCITE-4193
> URL: https://issues.apache.org/jira/browse/CALCITE-4193
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Ruben Q L
>Priority: Major
>
> Sometimes we need to sort a big volume of data which does not fit into
> memory. In this situation EnumerableSort will cause an OutOfMemoryError.
> The solution for such a scenario will be using a different sorting algorithm:
> [External Sort|https://en.wikipedia.org/wiki/External_sorting].
> The goal of the current ticket is to implement a new operator
> (EnumerableExternalSort) to provide this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4193) Implement new sort operator: ExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Q L updated CALCITE-4193: --- Summary: Implement new sort operator: ExternalSort (was: Implement new sort operator: EnumerableExternalSort) > Implement new sort operator: ExternalSort > - > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4192) When the relNode has passed the AggregateProjectMergeRule rule, it will be wrong to calculate the column blood relationship through RelMdColumnOrigins tool
[
https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184981#comment-17184981
]
lifangzheng commented on CALCITE-4192:
--
hi， [~danny0405]， thanks for advising.
I mean when a relNode has optimized by AggregateProjectMergeRule, the
RelMdColumnOrigins#getColumnOrigins(Aggregate, RelMetadataQuery, int) get the
wrong column origins.
one case like this:
{code:java}
@Test void testColumnOriginAfterAggProjectMergeRule() throws Exception {
final String sql = "select count(ename), SAL from emp group by SAL";
final RelNode rel = tester.convertSqlToRel(sql).rel;
final HepProgramBuilder programBuilder = HepProgram.builder();
programBuilder.addRuleInstance(AggregateProjectMergeRule.Config.DEFAULT.toRule());
final HepPlanner planner = new HepPlanner(programBuilder.build());
planner.setRoot(rel);
RelNode finalRel = planner.findBestExp();
Set origins =
RelMetadataQuery.instance().getColumnOrigins(finalRel, 1);
assertThat(origins.size(), equalTo(1));
RelColumnOrigin columnOrigin = origins.iterator().next();
assertThat(columnOrigin.getOriginColumnOrdinal(), equalTo(5));
assertThat(columnOrigin.getOriginTable().getRowType().getFieldNames().get(5),
equalTo("SAL"));
}
{code}
> When the relNode has passed the AggregateProjectMergeRule rule, it will be
> wrong to calculate the column blood relationship through RelMdColumnOrigins
> tool
> ---
>
> Key: CALCITE-4192
> URL: https://issues.apache.org/jira/browse/CALCITE-4192
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.25.0
>Reporter: lifangzheng
>Priority: Major
> Labels: pull-request-available
> Fix For: next
>
> Attachments: image-2020-08-25-15-53-19-575.png,
> image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> !image-2020-08-25-15-53-19-575.png|width=415,height=280!
>
> !image-2020-08-25-15-55-31-702.png|width=423,height=270!
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184980#comment-17184980 ] Danny Chen commented on CALCITE-4193: - > We ought to be putting our effort into building a convention with a > byte-oriented data format. That's interesting, makes Calcite like other engines like Spark or Flink. > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4183) FilterSetOpTransposeRule constructor should allow for user defined Filter and SetOp classes
[
https://issues.apache.org/jira/browse/CALCITE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184978#comment-17184978
]
Julian Hyde commented on CALCITE-4183:
--
There's no general rule, besides use your discretion. I added
{{withOperandFor}} where creating the operands was moderately complex (e.g.
there were predicates) and/or where there are multiple instances of a rule in
the code base, some with logical classes and some not.
I certainly wouldn't deprecate {{withOperandFor}} (because then we'd have to
stop using it internally) but maybe some comments would help.
I'm wary of creating a "best practice" that then becomes a rule and forces
authors of rules to write a lot of boilerplate code. I've already had push-back
that creating a rule now requires creating two classes - a sub-class of
{{RelRule}} and a {{Config}} interface - where you previously needed only one.
Good catch that there are no tests for {{FilterSetOpTranspose}}.
{{CoreRules.FILTER_SET_OP_TRANSPOSE}} is used in 3 tests in {{RelOptRulesTest}}
but it's not the main thing being tested in any of those. We could use one or
two tests, but it is sufficient if those tests use the default instance, so
they won't need to call {{withOperandFor}} or {{withOperandSupplier}}.
> FilterSetOpTransposeRule constructor should allow for user defined Filter and
> SetOp classes
> ---
>
> Key: CALCITE-4183
> URL: https://issues.apache.org/jira/browse/CALCITE-4183
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.24.0
>Reporter: Sean Broeder
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 168h
> Time Spent: 20m
> Remaining Estimate: 167h 50m
>
> The current FilterSetOpTransposeRule constructor simply takes a
> RelBuilderFactory. Adding an additional constructor would allow more
> flexibility for user specific Filter and SetOp classes and follows behavior
> already allowed for several other rules.
> For example:
> {code}
> /**
> * Creates a FilterSetOpTransposeRule that uses a generic
> * {@link Filter} and {@link SetOp}
> * @param filterClass filter class
> * @param setOpClass setOp class
> * @param relBuilderFactory builder factory for relational expressions
> */
> public FilterSetOpTransposeRule(Class filterClass,
> Class setOpClass,
> RelBuilderFactory relBuilderFactory) {
>super(
>operand(filterClass,
>operand(setOpClass, any())),
>relBuilderFactory, null);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4192) When the relNode has passed the AggregateProjectMergeRule rule, it will be wrong to calculate the column blood relationship through RelMdColumnOrigins tool
[ https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-4192: Labels: pull-request-available (was: ) > When the relNode has passed the AggregateProjectMergeRule rule, it will be > wrong to calculate the column blood relationship through RelMdColumnOrigins > tool > --- > > Key: CALCITE-4192 > URL: https://issues.apache.org/jira/browse/CALCITE-4192 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.25.0 >Reporter: lifangzheng >Priority: Major > Labels: pull-request-available > Fix For: next > > Attachments: image-2020-08-25-15-53-19-575.png, > image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png > > Time Spent: 10m > Remaining Estimate: 0h > > !image-2020-08-25-15-53-19-575.png|width=415,height=280! > > !image-2020-08-25-15-55-31-702.png|width=423,height=270! > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184971#comment-17184971 ] Rui Wang commented on CALCITE-4193: --- >Calcite does not know how to serialize values. So this operator will not know >either. The values may not even be serializable. Yes, you are right. I didn't consider this. Then I agree that Calcite needs to define how serialize and deserialize types/java classes. > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184971#comment-17184971 ] Rui Wang edited comment on CALCITE-4193 at 8/26/20, 6:52 AM: - >Calcite does not know how to serialize values. So this operator will not know >either. The values may not even be serializable. Yes, you are right. I didn't consider this. Then I agree that Calcite needs to define how to serialize and deserialize types/java classes. was (Author: amaliujia): >Calcite does not know how to serialize values. So this operator will not know >either. The values may not even be serializable. Yes, you are right. I didn't consider this. Then I agree that Calcite needs to define how serialize and deserialize types/java classes. > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4183) FilterSetOpTransposeRule constructor should allow for user defined Filter and SetOp classes
[
https://issues.apache.org/jira/browse/CALCITE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated CALCITE-4183:
-
Description:
The current FilterSetOpTransposeRule constructor simply takes a
RelBuilderFactory. Adding an additional constructor would allow more
flexibility for user specific Filter and SetOp classes and follows behavior
already allowed for several other rules.
For example:
{code}
/**
* Creates a FilterSetOpTransposeRule that uses a generic
* {@link Filter} and {@link SetOp}
* @param filterClass filter class
* @param setOpClass setOp class
* @param relBuilderFactory builder factory for relational expressions
*/
public FilterSetOpTransposeRule(Class filterClass,
Class setOpClass,
RelBuilderFactory relBuilderFactory) {
super(
operand(filterClass,
operand(setOpClass, any())),
relBuilderFactory, null);
}
{code}
was:
The current FilterSetOpTransposeRule constructor simply takes a
RelBuilderFactory. Adding an additional constructor would allow more
flexibility for user specific Filter and SetOp classes and follows behavior
already allowed for several other rules.
For example:
/**
* Creates a FilterSetOpTransposeRule that uses a generic
* {@link Filter} and {@link SetOp}
* @param filterClass filter class
* @param setOpClass setOp class
* @param relBuilderFactory builder factory for relational expressions
*/
public FilterSetOpTransposeRule(Class filterClass,
Class setOpClass,
RelBuilderFactory relBuilderFactory) {
super(
operand(filterClass,
operand(setOpClass, any())),
relBuilderFactory, null);
}
> FilterSetOpTransposeRule constructor should allow for user defined Filter and
> SetOp classes
> ---
>
> Key: CALCITE-4183
> URL: https://issues.apache.org/jira/browse/CALCITE-4183
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.24.0
>Reporter: Sean Broeder
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 168h
> Time Spent: 20m
> Remaining Estimate: 167h 50m
>
> The current FilterSetOpTransposeRule constructor simply takes a
> RelBuilderFactory. Adding an additional constructor would allow more
> flexibility for user specific Filter and SetOp classes and follows behavior
> already allowed for several other rules.
> For example:
> {code}
> /**
> * Creates a FilterSetOpTransposeRule that uses a generic
> * {@link Filter} and {@link SetOp}
> * @param filterClass filter class
> * @param setOpClass setOp class
> * @param relBuilderFactory builder factory for relational expressions
> */
> public FilterSetOpTransposeRule(Class filterClass,
> Class setOpClass,
> RelBuilderFactory relBuilderFactory) {
>super(
>operand(filterClass,
>operand(setOpClass, any())),
>relBuilderFactory, null);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4192) When the relNode has passed the AggregateProjectMergeRule rule, it will be wrong to calculate the column blood relationship through RelMdColumnOrigins tool
[ https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184968#comment-17184968 ] Danny Chen commented on CALCITE-4192: - Thanks [~cuglifangzheng] for reporting this, you may need to describe the your problem more accurately, without pasting the pictures. For example, you can give the test case (or sql) which triggers the issue, and the right logical plan you expect it to have. > When the relNode has passed the AggregateProjectMergeRule rule, it will be > wrong to calculate the column blood relationship through RelMdColumnOrigins > tool > --- > > Key: CALCITE-4192 > URL: https://issues.apache.org/jira/browse/CALCITE-4192 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.25.0 >Reporter: lifangzheng >Priority: Major > Fix For: next > > Attachments: image-2020-08-25-15-53-19-575.png, > image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png > > > !image-2020-08-25-15-53-19-575.png|width=415,height=280! > > !image-2020-08-25-15-55-31-702.png|width=423,height=270! > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[
https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184964#comment-17184964
]
Julian Hyde commented on CALCITE-4193:
--
{quote}Will it work if hide such details under
EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.?
{quote}
The API will work. But it's not what the architecture was intended for, so the
implementation will have problems (bugs, performance issues).
{quote}In this case, Calcite does not care about how serialize and deserialize
values, etc. is done.
{quote}
Calcite does not know how to serialize values. So this operator will not know
either. The values may not even be serializable.
> Implement new sort operator: EnumerableExternalSort
> ---
>
> Key: CALCITE-4193
> URL: https://issues.apache.org/jira/browse/CALCITE-4193
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Ruben Q L
>Priority: Major
>
> Sometimes we need to sort a big volume of data which does not fit into
> memory. In this situation EnumerableSort will cause an OutOfMemoryError.
> The solution for such a scenario will be using a different sorting algorithm:
> [External Sort|https://en.wikipedia.org/wiki/External_sorting].
> The goal of the current ticket is to implement a new operator
> (EnumerableExternalSort) to provide this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4191) Improve the logic of creating aggregate calls
[
https://issues.apache.org/jira/browse/CALCITE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184958#comment-17184958
]
Julian Hyde commented on CALCITE-4191:
--
[~Chunwei Lei], There is already a way - but it's indirect. Use a
{{RelBuilder}}. As an example of code that does this, see
[AggregateExtractProjectRule|https://github.com/apache/calcite/blob/885a3da76cfd59171624dc569e83e93a5bdaffe3/core/src/main/java/org/apache/calcite/rel/rules/AggregateExtractProjectRule.java#L126]:
{code:java}
final List newAggCallList = new ArrayList<>();
for (AggregateCall aggCall : aggregate.getAggCallList()) {
final ImmutableList args =
relBuilder.fields(
Mappings.apply2(mapping, aggCall.getArgList()));
final RexNode filterArg = aggCall.filterArg < 0 ? null
: relBuilder.field(Mappings.apply(mapping, aggCall.filterArg));
newAggCallList.add(
relBuilder.aggregateCall(aggCall.getAggregation(), args)
.distinct(aggCall.isDistinct())
.filter(filterArg)
.approximate(aggCall.isApproximate())
.sort(relBuilder.fields(aggCall.collation))
.as(aggCall.name));
}
final RelBuilder.GroupKey groupKey =
relBuilder.groupKey(newGroupSet, newGroupSets);
relBuilder.aggregate(groupKey, newAggCallList);
call.transformTo(relBuilder.build());
{code}
Those {{RelBuilder.AggCall}} objects are basically builders.
{{relBuilder.aggregate}} converts them into {{AggregateCall}} instances.
> Improve the logic of creating aggregate calls
> -
>
> Key: CALCITE-4191
> URL: https://issues.apache.org/jira/browse/CALCITE-4191
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> According to the current code base, the only way to create \{{AggregateCall}}
> objects is by calling one of the two \{{AggregateCall#create}} methods (other
> create methods are deprecated).
> The two {{create}} methods have 9 and 11 parameters, respectively, 3 of which
> are booleans and 2 are ints. We find this makes the code less readable and
> error-prone, as some bugs are caused by specifying the wrong parameters.
> In this issue, we improve the related logic by the builder pattern, which
> results in the following benefits:
> 1. By creating the objects by the builder pattern, there is no need to
> maintain multiple overrides of the {{create}} methods.
> 2. There is no need to maintain multiple overrides of the {{copy}} methods,
> either.
> 3. The code becomes more readable and less error-prone, as it is less like to
> specify the wrong parameter.
> 4. Creating {{AggregateCall}} objects becomes easier, as the user does not
> have specify the default parameters repeatedly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184954#comment-17184954 ] Rui Wang edited comment on CALCITE-4193 at 8/26/20, 6:23 AM: - >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? Calcite does not care about how serialize and deserialize values, etc. is done? was (Author: amaliujia): >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184954#comment-17184954 ] Rui Wang edited comment on CALCITE-4193 at 8/26/20, 6:23 AM: - >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? In this case, Calcite does not care about how serialize and deserialize values, etc. is done. was (Author: amaliujia): >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? Calcite does not care about how serialize and deserialize values, etc. is done? > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184954#comment-17184954 ] Rui Wang commented on CALCITE-4193: --- >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4193) Implement new sort operator: EnumerableExternalSort
[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184952#comment-17184952 ] Julian Hyde commented on CALCITE-4193: -- The Enumerable convention was designed for in-memory data, on the JVM heap. It wasn’t intended for external algorithms. In particular, we have no mechanism to serialize and deserialize values, and no on-disk format, and no scheduler that can handle asynchronous i/o. We ought to be putting our effort into building a convention with a byte-oriented data format. > Implement new sort operator: EnumerableExternalSort > --- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ruben Q L >Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
