xudong963 edited a comment on pull request #1067:
URL: https://github.com/apache/arrow-datafusion/pull/1067#issuecomment-932890819
Though the PR fix the bug mentioned by the issue, there are still some bugs.
I am confused and need help.
When I ran the `cargo test`, I found there were four tests that couldn't
pass. For example `tests::ballista_round_trip::q7`
The following is the SQL executed in `tests::ballista_round_trip::q7`
```sql
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l_shipdate) as l_year,
l_extendedprice * (1 - l_discount) as volume
from
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2
where
s_suppkey = l_suppkey
and o_orderkey = l_orderkey
and c_custkey = o_custkey
and s_nationkey = n1.n_nationkey
and c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
)
and l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year;
```
Then I printed some logs to find the potential problems.
First of all, I printed the plan at
https://github.com/apache/arrow-datafusion/blob/master/benchmarks/src/bin/tpch.rs#L1092
```
Sort: #all_nations.o_year ASC NULLS FIRST
Projection: #all_nations.o_year, #SUM(CASE WHEN all_nations.nation =
Utf8("BRAZIL") THEN all_nations.volume ELSE Int64(0) END) /
#SUM(all_nations.volume) AS mkt_share
Aggregate: groupBy=[[#all_nations.o_year]], aggr=[[SUM(CASE WHEN
#all_nations.nation = Utf8("BRAZIL") THEN #all_nations.volume ELSE Int64(0)
END), SUM(#all_nations.volume)]]
Projection: datepart(Utf8("YEAR"), #orders.o_orderdate) AS o_year,
#lineitem.l_extendedprice * Int64(1) - #lineitem.l_discount AS volume,
#n2.n_name AS nation
Filter: #region.r_name = Utf8("AMERICA") AND #orders.o_orderdate
BETWEEN CAST(Utf8("1995-01-01") AS Date32) AND CAST(Utf8("1996-12-31") AS
Date32) AND #part.p_type = Utf8("ECONOMY ANODIZED STEEL")
Join: #n1.n_regionkey = #region.r_regionkey
Join: #supplier.s_nationkey = #n2.n_nationkey
Join: #customer.c_nationkey = #n1.n_nationkey
Join: #orders.o_custkey = #customer.c_custkey
Join: #lineitem.l_orderkey = #orders.o_orderkey
Join: #part.p_partkey = #lineitem.l_partkey,
#supplier.s_suppkey = #lineitem.l_suppkey
CrossJoin:
TableScan: part projection=None
TableScan: supplier projection=None
TableScan: lineitem projection=None
TableScan: orders projection=None
TableScan: customer projection=None
TableScan: n1 projection=None
TableScan: n2 projection=None
TableScan: region projection=None
```
It's OK?
Then I tried to find problems in
https://github.com/apache/arrow-datafusion/blob/master/benchmarks/src/bin/tpch.rs#L1094
because the test panicked at here(`Logical Plan Proto` to `Logical Plan`)
Finally, I found the panic at building aggregate logical plan
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/logical_plan/builder.rs#L470
The all_expr in
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/logical_plan/builder.rs#L465
is the following
```
Chain { a: Some(Iter([#all_nations.o_year])), b: Some(Iter([SUM(CASE WHEN
#all_nations.nation = Utf8("BRAZIL") THEN #all_nations.volume ELSE Int64(0)
END), SUM(#all_nations.volume)])) }
```
But the plan(aggregation input) and the plan schema are the followings
```
Projection: datepart(Utf8("YEAR"), #orders.o_orderdate) AS o_year,
#lineitem.l_extendedprice * Int64(1) - #lineitem.l_discount AS volume,
#n2.n_name AS nation
Filter: #region.r_name = Utf8("AMERICA") AND #orders.o_orderdate BETWEEN
CAST(Utf8("1995-01-01") AS Date32) AND CAST(Utf8("1996-12-31") AS Date32) AND
#part.p_type = Utf8("ECONOMY ANODIZED STEEL")
Join: #n1.n_regionkey = #region.r_regionkey
Join: #supplier.s_nationkey = #n2.n_nationkey
Join: #customer.c_nationkey = #n1.n_nationkey
Join: #orders.o_custkey = #customer.c_custkey
Join: #lineitem.l_orderkey = #orders.o_orderkey
Join: #part.p_partkey = #lineitem.l_partkey,
#supplier.s_suppkey = #lineitem.l_suppkey
CrossJoin:
TableScan: part projection=None
TableScan: supplier projection=None
TableScan: lineitem projection=None
TableScan: orders projection=None
TableScan: customer projection=None
TableScan: n1 projection=None
TableScan: n2 projection=None
TableScan: region projection=None
```
```
DFSchema { fields: [DFField { qualifier: None, field: Field { name:
"o_year", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: None } }, DFField { qualifier: None, field: Field { name: "volume",
data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false,
metadata: None } }, DFField { qualifier: Some("n2"), field: Field { name:
"nation", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false,
metadata: None } }] }
```
So there will be conflict in
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/logical_plan/dfschema.rs#L150
I have two questions
1. Why
https://github.com/apache/arrow-datafusion/blob/master/benchmarks/src/bin/tpch.rs#L1092
is ok and get a plan, the plan is correct?
2. Why do `Logical Plan Proto` to `Logical Plan` get errors, now I don't
know how to fix it.
PTAL and give me some help @alamb @houqp @Dandandan. Thanks very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
