Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
deniskuzZ merged PR #5191: URL: https://github.com/apache/hive/pull/5191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2075398541 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2068777158 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2068473576 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2068080068 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2067366526 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2064466439 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
zhangbutao commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2061588240 > LGTM +1, pending tests note: mv_iceberg_orc5 fails in master as well, looks like flaky `mv_iceberg_orc5.q` is flaky. Please see https://issues.apache.org/jira/browse/HIVE-28203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
deniskuzZ commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1568783021
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java:
##
@@ -108,4 +114,14 @@ public String getDisplayString(String[] children) {
return getStandardDisplayString("lower", children);
}
+ @Override
+ public StatEstimator getStatEstimator() {
+return new StatEstimator() {
+ @Override
+ public Optional estimate(List argStats) {
+return Optional.of(argStats.get(0).clone());
Review Comment:
could it be an empty list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
deniskuzZ commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1568783021
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java:
##
@@ -108,4 +114,14 @@ public String getDisplayString(String[] children) {
return getStandardDisplayString("lower", children);
}
+ @Override
+ public StatEstimator getStatEstimator() {
+return new StatEstimator() {
+ @Override
+ public Optional estimate(List argStats) {
+return Optional.of(argStats.get(0).clone());
Review Comment:
could it be an empty list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
okumin commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1568759804
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -1736,6 +1749,13 @@ private static boolean
isWideningCast(ExprNodeGenericFuncDesc engfd) {
engfd.getTypeInfo());
}
+ private static boolean isColStatsReusableFunc(ExprNodeGenericFuncDesc engfd)
{
+GenericUDF udf = engfd.getGenericUDF();
+Class udfClass =
+(udf instanceof GenericUDFBridge) ?
((GenericUDFBridge)udf).getUdfClass() : udf.getClass();
+return udfClass == GenericUDFLower.class || udfClass ==
GenericUDFUpper.class;
Review Comment:
I didn't know the interface and looks super nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2060743845 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
ngsg commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1568322246
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -1736,6 +1749,13 @@ private static boolean
isWideningCast(ExprNodeGenericFuncDesc engfd) {
engfd.getTypeInfo());
}
+ private static boolean isColStatsReusableFunc(ExprNodeGenericFuncDesc engfd)
{
+GenericUDF udf = engfd.getGenericUDF();
+Class udfClass =
+(udf instanceof GenericUDFBridge) ?
((GenericUDFBridge)udf).getUdfClass() : udf.getClass();
+return udfClass == GenericUDFLower.class || udfClass ==
GenericUDFUpper.class;
Review Comment:
Thanks for sharing the examples. I moved the estimation logic to UDF side by
using `StatEstimatorProvider` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2060305300 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
okumin commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1565852758
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -1736,6 +1749,13 @@ private static boolean
isWideningCast(ExprNodeGenericFuncDesc engfd) {
engfd.getTypeInfo());
}
+ private static boolean isColStatsReusableFunc(ExprNodeGenericFuncDesc engfd)
{
+GenericUDF udf = engfd.getGenericUDF();
+Class udfClass =
+(udf instanceof GenericUDFBridge) ?
((GenericUDFBridge)udf).getUdfClass() : udf.getClass();
+return udfClass == GenericUDFLower.class || udfClass ==
GenericUDFUpper.class;
Review Comment:
I like this PR. I have also observed that an estimation was wrong, and Map
Join was disabled.
We implement some estimation APIs on the UDF side. I'll share examples with
you. This is not a strong suggestion, and the current implementation seems to
be a good starting point. We may finally need a pluggable I/F to empower custom
UDFs more.
-
https://github.com/apache/hive/blob/rel/release-4.0.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NDV.java
-
https://github.com/apache/hive/blob/rel/release-4.0.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java#L63-L81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2049058390 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues  [13 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
