alamb commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678766915
@jorgecarleitao -- another thing I can think of would be to postpone the
UDF resolution until the type coercion logical optimizer pass.
So in other words, when converting
alamb commented on pull request #8031:
URL: https://github.com/apache/arrow/pull/8031#issuecomment-678779883
FYI @jorgecarleitao and @andygrove
This is an automated message from the Apache Git Service.
To respond to the
alamb commented on a change in pull request #8031:
URL: https://github.com/apache/arrow/pull/8031#discussion_r475224655
##
File path: rust/datafusion/src/execution/context.rs
##
@@ -189,16 +190,7 @@ impl ExecutionContext {
/// Register a scalar UDF
pub fn
andygrove closed pull request #8029:
URL: https://github.com/apache/arrow/pull/8029
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
github-actions[bot] commented on pull request #8033:
URL: https://github.com/apache/arrow/pull/8033#issuecomment-678790001
https://issues.apache.org/jira/browse/ARROW-9837
This is an automated message from the Apache Git
jorgecarleitao commented on a change in pull request #8034:
URL: https://github.com/apache/arrow/pull/8034#discussion_r475253287
##
File path: rust/datafusion/src/execution/physical_plan/mod.rs
##
@@ -74,6 +85,15 @@ impl Partitioning {
}
}
+/// Distribution schemes
github-actions[bot] commented on pull request #8030:
URL: https://github.com/apache/arrow/pull/8030#issuecomment-678764184
https://issues.apache.org/jira/browse/ARROW-9835
This is an automated message from the Apache Git
andygrove commented on a change in pull request #8029:
URL: https://github.com/apache/arrow/pull/8029#discussion_r475225864
##
File path: rust/datafusion/src/execution/physical_plan/sort.rs
##
@@ -61,44 +61,28 @@ impl ExecutionPlan for SortExec {
jorgecarleitao commented on a change in pull request #8031:
URL: https://github.com/apache/arrow/pull/8031#discussion_r475229968
##
File path: rust/datafusion/src/execution/context.rs
##
@@ -477,9 +466,9 @@ impl ExecutionConfig {
/// Execution context for registering data
wqc200 opened a new pull request #8033:
URL: https://github.com/apache/arrow/pull/8033
Select @@version;
@@version is a variable, and if we want to get its value, we should get it
from outside the system,
This is an
alamb commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678780754
> When you mean data_type you mean the arguments' types or the return_type?
I was referring to `Expr::ScalarFunction::data_type`:
andygrove commented on pull request #8029:
URL: https://github.com/apache/arrow/pull/8029#issuecomment-678784408
> This is an impressive simplification and improvement. Really great work,
@andygrove !
>
> I went through it and could not find any issue with it, only benefits.
>
github-actions[bot] commented on pull request #8032:
URL: https://github.com/apache/arrow/pull/8032#issuecomment-678788344
https://issues.apache.org/jira/browse/ARROW-9836
This is an automated message from the Apache Git
alamb commented on a change in pull request #8029:
URL: https://github.com/apache/arrow/pull/8029#discussion_r475205751
##
File path: rust/datafusion/src/execution/context.rs
##
@@ -350,72 +350,64 @@ impl ExecutionContext {
}
/// Execute a physical plan and collect
jorgecarleitao commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678768666
When you mean `data_type` you mean the arguments' types or the `return_type`?
This is an automated message
github-actions[bot] commented on pull request #8031:
URL: https://github.com/apache/arrow/pull/8031#issuecomment-678780263
https://issues.apache.org/jira/browse/ARROW-9815
This is an automated message from the Apache Git
alamb commented on a change in pull request #8030:
URL: https://github.com/apache/arrow/pull/8030#discussion_r475225101
##
File path: rust/datafusion/src/execution/context.rs
##
@@ -489,15 +489,13 @@ impl SchemaProvider for ExecutionContextState {
alamb opened a new pull request #8031:
URL: https://github.com/apache/arrow/pull/8031
Inspired by the conversation on
https://github.com/apache/arrow/pull/8018/files, I have been bothered by the
use of Arc/Mutex and the resulting code complication in ExecutionContext and
LogicalPlanning.
jorgecarleitao commented on a change in pull request #8030:
URL: https://github.com/apache/arrow/pull/8030#discussion_r475227407
##
File path: rust/datafusion/src/execution/context.rs
##
@@ -489,15 +489,13 @@ impl SchemaProvider for ExecutionContextState {
jorgecarleitao commented on pull request #8034:
URL: https://github.com/apache/arrow/pull/8034#issuecomment-678807425
It looks fantastic! Super excited to see this!
Gave it a quick look, and my understanding so far:
* Physical nodes have requirements
jorgecarleitao commented on pull request #8027:
URL: https://github.com/apache/arrow/pull/8027#issuecomment-678760761
Well, thank you to @andygrove , that took the initiative and did the hard
work!
This is an automated
jorgecarleitao commented on pull request #8024:
URL: https://github.com/apache/arrow/pull/8024#issuecomment-678770336
@alamb , thanks a lot for that insight.
I may have been using the wrong notation here.
I think that we have each columns' type during logical planning: the
andygrove commented on pull request #8034:
URL: https://github.com/apache/arrow/pull/8034#issuecomment-678809178
> It looks fantastic! Super excited to see this!
>
> Gave it a quick look, and my understanding so far:
>
> * Physical nodes have requirements
jorgecarleitao opened a new pull request #8030:
URL: https://github.com/apache/arrow/pull/8030
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
andygrove closed pull request #8028:
URL: https://github.com/apache/arrow/pull/8028
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
andygrove closed pull request #8024:
URL: https://github.com/apache/arrow/pull/8024
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
github-actions[bot] commented on pull request #8034:
URL: https://github.com/apache/arrow/pull/8034#issuecomment-678805666
https://issues.apache.org/jira/browse/ARROW-9464
This is an automated message from the Apache Git
andygrove opened a new pull request #8034:
URL: https://github.com/apache/arrow/pull/8034
This PR adds the first physical optimization rule, to insert explicit
MergeExec nodes into the physical plan when operators require a single
partition of input (such as GlobalLimitExec, SortExec,
andygrove commented on pull request #8034:
URL: https://github.com/apache/arrow/pull/8034#issuecomment-678805566
@alamb @jorgecarleitao I'm pretty excited about this PR. This is a good
example of how we can write optimizer rules against a trait-based plan.
jorgecarleitao commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678781518
The code you pointed to reads `return_type: DataType`. I will assume you
mean the return type declared in `Expr::ScalarFunctions`.
Two minds thinking alike: I was
jorgecarleitao opened a new pull request #8032:
URL: https://github.com/apache/arrow/pull/8032
See associated issue and document for details.
The gist is that currently, users call UDFs through
```
df.select(scalar_functions(“sqrt”, vec![col(“a”)], DataType::Float64))
jorgecarleitao commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678739758
This is currently failing for an interesting reason, and I need some help in
decision making.
TL;DR options:
1. wait for #8024 and stop using the type
jorgecarleitao commented on a change in pull request #8033:
URL: https://github.com/apache/arrow/pull/8033#discussion_r475270454
##
File path: rust/datafusion/src/variable/system.rs
##
@@ -0,0 +1,18 @@
+use crate::logicalplan::ScalarValue;
+use crate::error::Result;
+use
liyafan82 commented on pull request #7837:
URL: https://github.com/apache/arrow/pull/7837#issuecomment-678870459
Merging this. Thanks to all reviews for the good comments.
This is an automated message from the Apache Git
jorgecarleitao edited a comment on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678825604
I pushed a new commit for this. Essentially, the new commit moves the type
coercion of UDFs to the physical plan, thus aligning this code base to the
current master
jorgecarleitao commented on a change in pull request #8033:
URL: https://github.com/apache/arrow/pull/8033#discussion_r475270612
##
File path: rust/datafusion/src/logicalplan.rs
##
@@ -713,6 +718,9 @@ impl fmt::Debug for Expr {
match self {
sagnikc-dremio opened a new pull request #8035:
URL: https://github.com/apache/arrow/pull/8035
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
jorgecarleitao commented on pull request #7984:
URL: https://github.com/apache/arrow/pull/7984#issuecomment-678829586
Closing as wont fix.
This is an automated message from the Apache Git Service.
To respond to the message,
jorgecarleitao closed pull request #7984:
URL: https://github.com/apache/arrow/pull/7984
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
liyafan82 closed pull request #7837:
URL: https://github.com/apache/arrow/pull/7837
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
emkornfield commented on a change in pull request #8023:
URL: https://github.com/apache/arrow/pull/8023#discussion_r475337736
##
File path: cpp/src/parquet/encryption.h
##
@@ -47,15 +47,15 @@ using ColumnPathToEncryptionPropertiesMap =
class PARQUET_EXPORT
jorgecarleitao commented on a change in pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#discussion_r475266986
##
File path: rust/datafusion/src/execution/physical_plan/planner.rs
##
@@ -306,32 +305,25 @@ impl DefaultPhysicalPlanner {
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-678839259
@emkornfield Sorry for the late reply. And yes I was planing to try adding a
`Bits.java` based implementation to this PR, but I may not be able to work on
it
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-678839259
@emkornfield Sorry for the late reply. And yes I was planing to try adding a
`Bits.java` based implementation to this PR, but I may not be able to working
on it instantly
jorgecarleitao commented on a change in pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#discussion_r475266854
##
File path: rust/datafusion/src/execution/physical_plan/math_expressions.rs
##
@@ -103,56 +103,3 @@ pub fn scalar_functions() -> Vec {
jorgecarleitao commented on pull request #7967:
URL: https://github.com/apache/arrow/pull/7967#issuecomment-678825604
I pushed a new commit for this. Essentially, the new commit moves the type
coercion of UDFs to the physical plan, thus aligning this code base to the
current master after
github-actions[bot] commented on pull request #8035:
URL: https://github.com/apache/arrow/pull/8035#issuecomment-678885179
https://issues.apache.org/jira/browse/ARROW-9641
This is an automated message from the Apache Git
emkornfield commented on a change in pull request #8023:
URL: https://github.com/apache/arrow/pull/8023#discussion_r475351570
##
File path: cpp/src/parquet/test_encryption_util.h
##
@@ -65,5 +66,36 @@ inline std::string data_file(const char* file) {
return ss.str();
}
48 matches
Mail list logo
