[jira] [Resolved] (TRAFODION-2602) Update Trafodion web site with new 2.1.0 release
[ https://issues.apache.org/jira/browse/TRAFODION-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2602. Resolution: Fixed Note that PR 1078 had a typo in it, using the wrong JIRA, so it placed a lot of comments into this JIRA. > Update Trafodion web site with new 2.1.0 release > > > Key: TRAFODION-2602 > URL: https://issues.apache.org/jira/browse/TRAFODION-2602 > Project: Apache Trafodion > Issue Type: Improvement > Components: documentation >Affects Versions: 2.2-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Now that release 2.1.0-incubating is out, the web site needs to be updated. > The new artifacts need to be on the download page, and there needs to be a > new section with the 2.1.0 manuals. The "in development" manual section will > switch over to 2.2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2602) Update Trafodion web site with new 2.1.0 release
[ https://issues.apache.org/jira/browse/TRAFODION-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991760#comment-15991760 ] Hans Zeller commented on TRAFODION-2602: The release notes from the wiki, https://cwiki.apache.org/confluence/display/TRAFODION/Release+2.1, need to go into the site as well. > Update Trafodion web site with new 2.1.0 release > > > Key: TRAFODION-2602 > URL: https://issues.apache.org/jira/browse/TRAFODION-2602 > Project: Apache Trafodion > Issue Type: Improvement > Components: documentation >Affects Versions: 2.2-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Now that release 2.1.0-incubating is out, the web site needs to be updated. > The new artifacts need to be on the download page, and there needs to be a > new section with the 2.1.0 manuals. The "in development" manual section will > switch over to 2.2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2602) Update Trafodion web site with new 2.1.0 release
Hans Zeller created TRAFODION-2602: -- Summary: Update Trafodion web site with new 2.1.0 release Key: TRAFODION-2602 URL: https://issues.apache.org/jira/browse/TRAFODION-2602 Project: Apache Trafodion Issue Type: Improvement Components: documentation Affects Versions: 2.2-incubating Reporter: Hans Zeller Assignee: Hans Zeller Fix For: 2.2-incubating Now that release 2.1.0-incubating is out, the web site needs to be updated. The new artifacts need to be on the download page, and there needs to be a new section with the 2.1.0 manuals. The "in development" manual section will switch over to 2.2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2593) Update Apache Incubator logo on the web site
[ https://issues.apache.org/jira/browse/TRAFODION-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971649#comment-15971649 ] Hans Zeller commented on TRAFODION-2593: If you don't see it, your browser probably has the image file cached. In that case, go to the following URL and then hit refresh or F5: http://trafodion.incubator.apache.org/images/logos/egg-logo.png > Update Apache Incubator logo on the web site > > > Key: TRAFODION-2593 > URL: https://issues.apache.org/jira/browse/TRAFODION-2593 > Project: Apache Trafodion > Issue Type: Improvement > Components: documentation >Affects Versions: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: any > > > The Apache Incubator has designed a new logo, see > http://incubator.staging.apache.org/guides/press-kit.html. Our site uses the > incubator logo in many places, so it needs to be updated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2593) Update Apache Incubator logo on the web site
[ https://issues.apache.org/jira/browse/TRAFODION-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2593. -- > Update Apache Incubator logo on the web site > > > Key: TRAFODION-2593 > URL: https://issues.apache.org/jira/browse/TRAFODION-2593 > Project: Apache Trafodion > Issue Type: Improvement > Components: documentation >Affects Versions: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: any > > > The Apache Incubator has designed a new logo, see > http://incubator.staging.apache.org/guides/press-kit.html. Our site uses the > incubator logo in many places, so it needs to be updated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (TRAFODION-2593) Update Apache Incubator logo on the web site
[ https://issues.apache.org/jira/browse/TRAFODION-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2593 started by Hans Zeller. -- > Update Apache Incubator logo on the web site > > > Key: TRAFODION-2593 > URL: https://issues.apache.org/jira/browse/TRAFODION-2593 > Project: Apache Trafodion > Issue Type: Improvement > Components: documentation >Affects Versions: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: any > > > The Apache Incubator has designed a new logo, see > http://incubator.staging.apache.org/guides/press-kit.html. Our site uses the > incubator logo in many places, so it needs to be updated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2593) Update Apache Incubator logo on the web site
Hans Zeller created TRAFODION-2593: -- Summary: Update Apache Incubator logo on the web site Key: TRAFODION-2593 URL: https://issues.apache.org/jira/browse/TRAFODION-2593 Project: Apache Trafodion Issue Type: Improvement Components: documentation Affects Versions: any Reporter: Hans Zeller Assignee: Hans Zeller Fix For: any The Apache Incubator has designed a new logo, see http://incubator.staging.apache.org/guides/press-kit.html. Our site uses the incubator logo in many places, so it needs to be updated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2592) Give better diagnostics when HBase is not available while Trafodion starts
[ https://issues.apache.org/jira/browse/TRAFODION-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2592. Resolution: Fixed > Give better diagnostics when HBase is not available while Trafodion starts > -- > > Key: TRAFODION-2592 > URL: https://issues.apache.org/jira/browse/TRAFODION-2592 > Project: Apache Trafodion > Issue Type: Bug > Components: foundation >Affects Versions: 1.1 (pre-incubation) > Environment: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When Trafodion gets started with sqstart and HBase is down, Trafodion may > hang or fail with difficult to understand error messages. We should give an > easy to understand diagnostics that pinpoints the HBase status as the source > of the problem. > Narendra has written a tool that does just that, I'll add a call to this tool > to the sqstart method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2592) Give better diagnostics when HBase is not available while Trafodion starts
[ https://issues.apache.org/jira/browse/TRAFODION-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2592. -- > Give better diagnostics when HBase is not available while Trafodion starts > -- > > Key: TRAFODION-2592 > URL: https://issues.apache.org/jira/browse/TRAFODION-2592 > Project: Apache Trafodion > Issue Type: Bug > Components: foundation >Affects Versions: 1.1 (pre-incubation) > Environment: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When Trafodion gets started with sqstart and HBase is down, Trafodion may > hang or fail with difficult to understand error messages. We should give an > easy to understand diagnostics that pinpoints the HBase status as the source > of the problem. > Narendra has written a tool that does just that, I'll add a call to this tool > to the sqstart method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2592) Give better diagnostics when HBase is not available while Trafodion starts
Hans Zeller created TRAFODION-2592: -- Summary: Give better diagnostics when HBase is not available while Trafodion starts Key: TRAFODION-2592 URL: https://issues.apache.org/jira/browse/TRAFODION-2592 Project: Apache Trafodion Issue Type: Bug Components: foundation Affects Versions: 1.1 (pre-incubation) Environment: any Reporter: Hans Zeller Assignee: Hans Zeller Fix For: 2.2-incubating When Trafodion gets started with sqstart and HBase is down, Trafodion may hang or fail with difficult to understand error messages. We should give an easy to understand diagnostics that pinpoints the HBase status as the source of the problem. Narendra has written a tool that does just that, I'll add a call to this tool to the sqstart method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (TRAFODION-2592) Give better diagnostics when HBase is not available while Trafodion starts
[ https://issues.apache.org/jira/browse/TRAFODION-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2592 started by Hans Zeller. -- > Give better diagnostics when HBase is not available while Trafodion starts > -- > > Key: TRAFODION-2592 > URL: https://issues.apache.org/jira/browse/TRAFODION-2592 > Project: Apache Trafodion > Issue Type: Bug > Components: foundation >Affects Versions: 1.1 (pre-incubation) > Environment: any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When Trafodion gets started with sqstart and HBase is down, Trafodion may > hang or fail with difficult to understand error messages. We should give an > easy to understand diagnostics that pinpoints the HBase status as the source > of the problem. > Narendra has written a tool that does just that, I'll add a call to this tool > to the sqstart method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2569) Improve handling of index hints
[ https://issues.apache.org/jira/browse/TRAFODION-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2569. -- > Improve handling of index hints > --- > > Key: TRAFODION-2569 > URL: https://issues.apache.org/jira/browse/TRAFODION-2569 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > We allow index hints of this form: > select ... from table1 <<+index TRAFODION.SCH.IX1>> ... > In many cases, the compiler does not choose the index specified in the hint, > however. A few improvements I am planning to do, if I can pull it off: > * if we need an index join, make sure we get it using this index > * allow hints in update statements > * bind the names in the hint, so they don't have to be specified as > upper-case 3-part names. > * issue a warning if the table name in the index does not specify an index -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2569) Improve handling of index hints
[ https://issues.apache.org/jira/browse/TRAFODION-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2569. Resolution: Fixed > Improve handling of index hints > --- > > Key: TRAFODION-2569 > URL: https://issues.apache.org/jira/browse/TRAFODION-2569 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > We allow index hints of this form: > select ... from table1 <<+index TRAFODION.SCH.IX1>> ... > In many cases, the compiler does not choose the index specified in the hint, > however. A few improvements I am planning to do, if I can pull it off: > * if we need an index join, make sure we get it using this index > * allow hints in update statements > * bind the names in the hint, so they don't have to be specified as > upper-case 3-part names. > * issue a warning if the table name in the index does not specify an index -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TRAFODION-2582) Two timestamp minus and cannot cast to interval value
[ https://issues.apache.org/jira/browse/TRAFODION-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller reassigned TRAFODION-2582: -- Assignee: Hans Zeller > Two timestamp minus and cannot cast to interval value > - > > Key: TRAFODION-2582 > URL: https://issues.apache.org/jira/browse/TRAFODION-2582 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.1-incubating >Reporter: Yuan Liu >Assignee: Hans Zeller > Fix For: any > > > SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06 > 11:45:00') as interval second) from (values(1)); > (EXPR) > -- > 0.00 > --- 1 row(s) selected. > SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06 > 11:45:00') as interval minute) from (values(1)); > (EXPR) > -- > 0 > >>select timestamp'2017-04-06 11:50:00' - timestamp'2017-04-03 11:45:00' from > >>dual; > (EXPR) > - > 3 > --- 1 row(s) selected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2582) Two timestamp minus and cannot cast to interval value
[ https://issues.apache.org/jira/browse/TRAFODION-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2582: --- Component/s: (was: Build Infrastructure) sql-cmp > Two timestamp minus and cannot cast to interval value > - > > Key: TRAFODION-2582 > URL: https://issues.apache.org/jira/browse/TRAFODION-2582 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.1-incubating >Reporter: Yuan Liu > Fix For: any > > > SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06 > 11:45:00') as interval second) from (values(1)); > (EXPR) > -- > 0.00 > --- 1 row(s) selected. > SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06 > 11:45:00') as interval minute) from (values(1)); > (EXPR) > -- > 0 > >>select timestamp'2017-04-06 11:50:00' - timestamp'2017-04-03 11:45:00' from > >>dual; > (EXPR) > - > 3 > --- 1 row(s) selected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2582) Two timestamp minus and cannot cast to interval value
[
https://issues.apache.org/jira/browse/TRAFODION-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958199#comment-15958199
]
Hans Zeller commented on TRAFODION-2582:
The result we would have expected would show the difference between the
timestamps, which is 300 seconds or 5 minutes for the first two queries.
The problem (if it is indeed a bug, as I believe) is in BiArith::bindNode. When
we see this operation of a difference between two timestamps, we get to the
following code:
{noformat}
else if ((naType0->getTypeQualifier() == NA_DATETIME_TYPE) &&
(naType1->getTypeQualifier() == NA_DATETIME_TYPE) &&
(getOperatorType() == ITM_MINUS))
{
// Column of DATE datatype is internally created as TIMESTAMP(0).
// timestamp(0) - date = diff in days
// timestamp(0) - timestamp(0) = diff in days
// date - date= diff in days
{noformat}
After some more checks, we cast both timestamp(0) values to a date. Therefore,
the result will be the difference in days.
That looks wrong to me. I probably need to consult with some other folks before
making this change, but it does not look right to me.
> Two timestamp minus and cannot cast to interval value
> -
>
> Key: TRAFODION-2582
> URL: https://issues.apache.org/jira/browse/TRAFODION-2582
> Project: Apache Trafodion
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 2.1-incubating
>Reporter: Yuan Liu
> Fix For: any
>
>
> SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06
> 11:45:00') as interval second) from (values(1));
> (EXPR)
> --
> 0.00
> --- 1 row(s) selected.
> SQL>select cast((timestamp'2017-04-06 11:50:00' - timestamp'2017-04-06
> 11:45:00') as interval minute) from (values(1));
> (EXPR)
> --
> 0
> >>select timestamp'2017-04-06 11:50:00' - timestamp'2017-04-03 11:45:00' from
> >>dual;
> (EXPR)
> -
> 3
> --- 1 row(s) selected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TRAFODION-2581) Combine JVM startup code of SQL executor and UDR server
Hans Zeller created TRAFODION-2581: -- Summary: Combine JVM startup code of SQL executor and UDR server Key: TRAFODION-2581 URL: https://issues.apache.org/jira/browse/TRAFODION-2581 Project: Apache Trafodion Issue Type: Improvement Components: sql-general Reporter: Hans Zeller Assignee: Hans Zeller Priority: Minor This was discussed in https://github.com/apache/incubator-trafodion/pull/1045. Ideally we would be using JavaObjectInterface::createJVM() to create the JVM in the UDR server. This may require an enhancement to pass in additional options to this method. For now we have some duplicated code, which is probably OK in the short term. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2557) Update TMUDF wiki to explain "trusted" flavor of TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2557. Resolution: Fixed Updated the wiki on 3/29/2017. https://cwiki.apache.org/confluence/display/TRAFODION/Tutorial%3A+The+object-oriented+UDF+interface > Update TMUDF wiki to explain "trusted" flavor of TMUDFs > --- > > Key: TRAFODION-2557 > URL: https://issues.apache.org/jira/browse/TRAFODION-2557 > Project: Apache Trafodion > Issue Type: Sub-task >Reporter: Hans Zeller >Assignee: Hans Zeller > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2557) Update TMUDF wiki to explain "trusted" flavor of TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2557. -- > Update TMUDF wiki to explain "trusted" flavor of TMUDFs > --- > > Key: TRAFODION-2557 > URL: https://issues.apache.org/jira/browse/TRAFODION-2557 > Project: Apache Trafodion > Issue Type: Sub-task >Reporter: Hans Zeller >Assignee: Hans Zeller > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2578) Internal error 2235 with UDFs and subquery unnesting
Hans Zeller created TRAFODION-2578: -- Summary: Internal error 2235 with UDFs and subquery unnesting Key: TRAFODION-2578 URL: https://issues.apache.org/jira/browse/TRAFODION-2578 Project: Apache Trafodion Issue Type: Bug Components: sql-cmp Affects Versions: 1.3-incubating Reporter: Hans Zeller Assignee: Hans Zeller Fix For: 2.2-incubating I've seen this error when compiling query plans with TMUDFs: *** ERROR[2235] Compiler Internal Error: An unknown error, originated from file ../optimizer/RelRoutine.cpp at line 465. The issue is in method RelExpr::getMoreOutputsIfPossible(), which is used during semantic query optimization to get additional key columns from RelExprs. This calls TableMappingUDF::getPotentialOutputValuesAsVEGs(), which is stubbed out with an assert. We either need to make RelExpr::getMoreOutputsIfPossible() virtual and override it for TableMappingUDF, or implement TableMappingUDF::getPotentialOutputValuesAsVEGs(). Right now I'm leaning towards the former. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2573) Support Index hints in a DML statement
[ https://issues.apache.org/jira/browse/TRAFODION-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2573: --- Issue Type: Sub-task (was: Improvement) Parent: TRAFODION-2569 > Support Index hints in a DML statement > -- > > Key: TRAFODION-2573 > URL: https://issues.apache.org/jira/browse/TRAFODION-2573 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmp >Affects Versions: any >Reporter: Suresh Subbiah >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Add support for index hints to be embedded in an SQL DML statement. > Examples are > set schema sch; > create table t1 (a int not null primary key, b int, c int) salt using 2 > partitions ; > create index ix1 on t1(b) ; > create index ix2 on t1(c) ; > create index ix3 on t1(b) salt like table; > create index ix4 on t1(c) salt like table; > select * from t1 <<+ index TRAFODION.SCH.IX1>> where b > 10 ; > select * from t1 <<+ index TRAFODION.SCH.IX3>> where b > 10 ; > select a,b from t1 <<+ index TRAFODION.SCH.IX1>> where b > 10 ; > select a,b from t1 <<+ index TRAFODION.SCH.IX3>> where b > 10 ; > update t1 <<+ index TRAFODION.SCH.IX1>> set c = 5 where b > 10; > update t1 <<+ index TRAFODION.SCH.IX1>> set c = 5 where b = 10; > update t1 <<+ index TRAFODION.SCH.IX3>> set c = 5 where b > 10; > update t1 <<+ index TRAFODION.SCH.IX3>> set c = 5 where b = 10; > delete from t1 <<+ index TRAFODION.SCH.IX2>> where c > 10; > delete from t1 <<+ index TRAFODION.SCH.IX2>> where c = 10; > delete from t1 <<+ index TRAFODION.SCH.IX4>> where c > 10; > delete from t1 <<+ index TRAFODION.SCH.IX4>> where c = 10; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2534) UDRs need the ability to define a CLASSPATH or similar concept
[ https://issues.apache.org/jira/browse/TRAFODION-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2534. -- > UDRs need the ability to define a CLASSPATH or similar concept > -- > > Key: TRAFODION-2534 > URL: https://issues.apache.org/jira/browse/TRAFODION-2534 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > Right now, Java UDRs can only load classes from their own library (a jar > file). In many cases, we need to be able to load additional classes. The > current solution is to unpack all the jars in a single directory, then create > a single jar with all the needed files in it and register that as the > library. It works, but is very ugly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2534) UDRs need the ability to define a CLASSPATH or similar concept
[ https://issues.apache.org/jira/browse/TRAFODION-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2534. Resolution: Fixed > UDRs need the ability to define a CLASSPATH or similar concept > -- > > Key: TRAFODION-2534 > URL: https://issues.apache.org/jira/browse/TRAFODION-2534 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > Right now, Java UDRs can only load classes from their own library (a jar > file). In many cases, we need to be able to load additional classes. The > current solution is to unpack all the jars in a single directory, then create > a single jar with all the needed files in it and register that as the > library. It works, but is very ugly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2569) Improve handling of index hints
Hans Zeller created TRAFODION-2569: -- Summary: Improve handling of index hints Key: TRAFODION-2569 URL: https://issues.apache.org/jira/browse/TRAFODION-2569 Project: Apache Trafodion Issue Type: Improvement Components: sql-cmp Affects Versions: 2.0-incubating Reporter: Hans Zeller Assignee: Hans Zeller Fix For: 2.2-incubating We allow index hints of this form: select ... from table1 <<+index TRAFODION.SCH.IX1>> ... In many cases, the compiler does not choose the index specified in the hint, however. A few improvements I am planning to do, if I can pull it off: * if we need an index join, make sure we get it using this index * allow hints in update statements * bind the names in the hint, so they don't have to be specified as upper-case 3-part names. * issue a warning if the table name in the index does not specify an index -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2562) User ids for isolated UDRs
[ https://issues.apache.org/jira/browse/TRAFODION-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2562: --- Component/s: installer > User ids for isolated UDRs > -- > > Key: TRAFODION-2562 > URL: https://issues.apache.org/jira/browse/TRAFODION-2562 > Project: Apache Trafodion > Issue Type: Sub-task > Components: installer, sql-cmu >Affects Versions: 2.0-incubating >Reporter: Hans Zeller > > In order to implement "isolated" UDRs, we need to have a user id for the > tdm_udrserv process that executes UDRs. Right now this process runs under the > same user id as the Trafodion engine, which means that the system > administrator has to trust the UDR writer to a great degree. Running UDRs > with a user id that has no access to HBase and HDFS and to the internal > resources of the Trafodion engine would reduce the required trust by a great > deal. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TRAFODION-2562) User ids for isolated UDRs
[ https://issues.apache.org/jira/browse/TRAFODION-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948174#comment-15948174 ] Hans Zeller edited comment on TRAFODION-2562 at 3/30/17 12:55 AM: -- Maybe we need more subtasks, but here is a list of things we probably will need: * The installer should create at least 1 such id initially. This is the easiest place, since the installer has the needed privileges to create user ids on all nodes of the cluster. Ideally we would allow a list of ids as installer options. * We need to keep track of these ids in the metadata. * A library should be associated - optionally - with such an id. My proposal would be that we do this at the library level, not at the UDR level. * We need DDL commands to create such an id, or at least a DDL command to register a Linux user id as an isolated user id. Also a command to unregister the id (may have to drop the Linux id separately). * For each isolated user id we may need a copy of the tdm_udrserv executable owned by that id, with the setuid flag set, so that when the Trafodion engine starts this program it runs under the correct id. was (Author: hzeller): Maybe we need more subtasks, but here is a list of things we probably will need: * The installer should create at least 1 such id initially. This is the easiest place, since the installer has the needed privileges to create user ids on all nodes of the cluster. Ideally we would allow a list of ids as installer options. * We need to keep track of these ids in the metadata. * A library should be associated - optionally - with such an id. My proposal would be that we do this at the library level, not at the UDR level. * We need DDL commands to create such an id, or at least a DDL command to register a Linux user id as an isolated user id. Also a command to unregister the id (may have to drop the Linux id separately). > User ids for isolated UDRs > -- > > Key: TRAFODION-2562 > URL: https://issues.apache.org/jira/browse/TRAFODION-2562 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmu >Affects Versions: 2.0-incubating >Reporter: Hans Zeller > > In order to implement "isolated" UDRs, we need to have a user id for the > tdm_udrserv process that executes UDRs. Right now this process runs under the > same user id as the Trafodion engine, which means that the system > administrator has to trust the UDR writer to a great degree. Running UDRs > with a user id that has no access to HBase and HDFS and to the internal > resources of the Trafodion engine would reduce the required trust by a great > deal. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2562) User ids for isolated UDRs
[ https://issues.apache.org/jira/browse/TRAFODION-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948174#comment-15948174 ] Hans Zeller commented on TRAFODION-2562: Maybe we need more subtasks, but here is a list of things we probably will need: * The installer should create at least 1 such id initially. This is the easiest place, since the installer has the needed privileges to create user ids on all nodes of the cluster. Ideally we would allow a list of ids as installer options. * We need to keep track of these ids in the metadata. * A library should be associated - optionally - with such an id. My proposal would be that we do this at the library level, not at the UDR level. * We need DDL commands to create such an id, or at least a DDL command to register a Linux user id as an isolated user id. Also a command to unregister the id (may have to drop the Linux id separately). > User ids for isolated UDRs > -- > > Key: TRAFODION-2562 > URL: https://issues.apache.org/jira/browse/TRAFODION-2562 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmu >Affects Versions: 2.0-incubating >Reporter: Hans Zeller > > In order to implement "isolated" UDRs, we need to have a user id for the > tdm_udrserv process that executes UDRs. Right now this process runs under the > same user id as the Trafodion engine, which means that the system > administrator has to trust the UDR writer to a great degree. Running UDRs > with a user id that has no access to HBase and HDFS and to the internal > resources of the Trafodion engine would reduce the required trust by a great > deal. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TRAFODION-2564) Move TMUDF compiler interface calls to tdm_udrserv
[ https://issues.apache.org/jira/browse/TRAFODION-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller reassigned TRAFODION-2564: -- Assignee: Hans Zeller > Move TMUDF compiler interface calls to tdm_udrserv > -- > > Key: TRAFODION-2564 > URL: https://issues.apache.org/jira/browse/TRAFODION-2564 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-exe >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > Right now, when the Trafodion compiler invokes the compiler interface of a > TMUDF, it goes through an executor CLI call, and the TMUDF is invoked in the > same process. We should introduce a message interface, like we do for the > runtime call, to allow executing all of the TMUDF code in the tdm_udrserv > process. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2564) Move TMUDF compiler interface calls to tdm_udrserv
Hans Zeller created TRAFODION-2564: -- Summary: Move TMUDF compiler interface calls to tdm_udrserv Key: TRAFODION-2564 URL: https://issues.apache.org/jira/browse/TRAFODION-2564 Project: Apache Trafodion Issue Type: Sub-task Components: sql-exe Affects Versions: 2.0-incubating Reporter: Hans Zeller Right now, when the Trafodion compiler invokes the compiler interface of a TMUDF, it goes through an executor CLI call, and the TMUDF is invoked in the same process. We should introduce a message interface, like we do for the runtime call, to allow executing all of the TMUDF code in the tdm_udrserv process. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2563) Communication protocol for tdm_udrserv for isolated UDRs
Hans Zeller created TRAFODION-2563: -- Summary: Communication protocol for tdm_udrserv for isolated UDRs Key: TRAFODION-2563 URL: https://issues.apache.org/jira/browse/TRAFODION-2563 Project: Apache Trafodion Issue Type: Sub-task Components: sql-exe Affects Versions: 2.0-incubating Reporter: Hans Zeller When running the tdm_udrserv process under a different user id, we will need to use a different communication protocol to it. This subtask covers defining, testing and implementing this protocol. One option could be the socket IPC protocol that is defined in core/sql/common/Ipc.h. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2562) User ids for isolated UDRs
Hans Zeller created TRAFODION-2562: -- Summary: User ids for isolated UDRs Key: TRAFODION-2562 URL: https://issues.apache.org/jira/browse/TRAFODION-2562 Project: Apache Trafodion Issue Type: Sub-task Components: sql-cmu Affects Versions: 2.0-incubating Reporter: Hans Zeller In order to implement "isolated" UDRs, we need to have a user id for the tdm_udrserv process that executes UDRs. Right now this process runs under the same user id as the Trafodion engine, which means that the system administrator has to trust the UDR writer to a great degree. Running UDRs with a user id that has no access to HBase and HDFS and to the internal resources of the Trafodion engine would reduce the required trust by a great deal. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2561) UDRs for non-trusted users
Hans Zeller created TRAFODION-2561: -- Summary: UDRs for non-trusted users Key: TRAFODION-2561 URL: https://issues.apache.org/jira/browse/TRAFODION-2561 Project: Apache Trafodion Issue Type: New Feature Components: sql-general Affects Versions: 2.0-incubating Reporter: Hans Zeller Right now, only trusted users can create Trafodion User-defined Routines, called UDRs in short (this includes stored procedures in Java, scalar UDFs and table-mapping UDFs). This is because these UDRs run under the user id of the Trafodion engine. It would be good to have an "isolated" flavor where these routines run under some other user id that poses a much smaller security risk, as it is done in other DMBSs, like IBM DB2 for Linux, Unix and Windows. This JIRA is the parent of multiple subtasks to achieve this goal. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (TRAFODION-2557) Update TMUDF wiki to explain "trusted" flavor of TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2557 started by Hans Zeller. -- > Update TMUDF wiki to explain "trusted" flavor of TMUDFs > --- > > Key: TRAFODION-2557 > URL: https://issues.apache.org/jira/browse/TRAFODION-2557 > Project: Apache Trafodion > Issue Type: Sub-task >Reporter: Hans Zeller >Assignee: Hans Zeller > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TRAFODION-2557) Update TMUDF wiki to explain "trusted" flavor of TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller reassigned TRAFODION-2557: -- Assignee: Hans Zeller > Update TMUDF wiki to explain "trusted" flavor of TMUDFs > --- > > Key: TRAFODION-2557 > URL: https://issues.apache.org/jira/browse/TRAFODION-2557 > Project: Apache Trafodion > Issue Type: Sub-task >Reporter: Hans Zeller >Assignee: Hans Zeller > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2557) Update TMUDF wiki to explain "trusted" flavor of TMUDFs
Hans Zeller created TRAFODION-2557: -- Summary: Update TMUDF wiki to explain "trusted" flavor of TMUDFs Key: TRAFODION-2557 URL: https://issues.apache.org/jira/browse/TRAFODION-2557 Project: Apache Trafodion Issue Type: Sub-task Reporter: Hans Zeller -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2558) Update wiki for scalar UDFs to explain "trusted" flavor
Hans Zeller created TRAFODION-2558: -- Summary: Update wiki for scalar UDFs to explain "trusted" flavor Key: TRAFODION-2558 URL: https://issues.apache.org/jira/browse/TRAFODION-2558 Project: Apache Trafodion Issue Type: Sub-task Reporter: Hans Zeller -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2556) Change Stored Procedures Guide to document trusted SPJs
Hans Zeller created TRAFODION-2556: -- Summary: Change Stored Procedures Guide to document trusted SPJs Key: TRAFODION-2556 URL: https://issues.apache.org/jira/browse/TRAFODION-2556 Project: Apache Trafodion Issue Type: Sub-task Reporter: Hans Zeller Update the Stored Procedures in Java (SPJs) Guide to explain that SPJs are trusted at this time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2555) Document security implications of UDRs more clearly
Hans Zeller created TRAFODION-2555: -- Summary: Document security implications of UDRs more clearly Key: TRAFODION-2555 URL: https://issues.apache.org/jira/browse/TRAFODION-2555 Project: Apache Trafodion Issue Type: Bug Reporter: Hans Zeller Assignee: Hans Zeller Right now, our manuals don't make it clear enough that Trafodion UDRs (User-defined Routines, that is a general term for UDFs and stored procedures) are "trusted". "Trusted" in this context means that they run as the Trafodion user id and therefore the code can bypass any security check and access any data stored in the Trafodion cluster. This is similar to trusted UDRs in other database systems like Oracle or DB2. Trafodion currently does not support the "isolated" flavor (called "fenced" in DB2). We need to add this information to the documentation we have. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2546) Handle get/set methods for null values of non-primitive Java types
Hans Zeller created TRAFODION-2546: -- Summary: Handle get/set methods for null values of non-primitive Java types Key: TRAFODION-2546 URL: https://issues.apache.org/jira/browse/TRAFODION-2546 Project: Apache Trafodion Issue Type: Bug Components: sql-general Affects Versions: 2.0-incubating Reporter: Hans Zeller Assignee: Hans Zeller In JDBC, the get/set method for non-primitive types like getString, setTime, etc. use Java nulls to represent SQL NULL values. The Trafodion TMUDF interface should do the same. See, for example, https://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html#getTime(java.lang.String,%20java.util.Calendar). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2543) Reduce the three SQL jar files built by Trafodion down to one
Hans Zeller created TRAFODION-2543: -- Summary: Reduce the three SQL jar files built by Trafodion down to one Key: TRAFODION-2543 URL: https://issues.apache.org/jira/browse/TRAFODION-2543 Project: Apache Trafodion Issue Type: Improvement Components: sql-general Affects Versions: 2.0-incubating Environment: Any Reporter: Hans Zeller Right now, we have three different pom.xml files in the core/sql directory and we build three different jars, one for Apache, CDH and HDP each. However, the SQL code in the client should be the same for each of these distributions. It may depend on some code that's in the HBase-trx jar file, but hopefully this should not require us to have three SQL jars as well. These build steps are in file core/sql/nskgmake/Makerules.mk. What I would like to request is the following: * Remove files core/sql/pom.xml.apache and core/sql/pom.xml.hdp. * Remove references to Cloudera from core/sql/pom.xml and consolidate this file with the two removed ones. * Publish a single SQL jar file with name trafodion-sql-x.y.z.jar * If there are any dependencies to the hbase-trx.*.jar files, use Java reflection to handle them - if possible. * Do only a single Maven build from core/sql/nskgmake/Makerules.mk * Publish this single jar file to the Apache Maven repository next time we do a Trafodion release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2182) JVM exception for TMUDF
[ https://issues.apache.org/jira/browse/TRAFODION-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15920294#comment-15920294 ] Hans Zeller commented on TRAFODION-2182: David reminded me of this issue today - sorry for not paying enough attention. The issue is probably with the custom class loader that the Trafodion UDR mechanism uses. This class loader just looks in the jar that is the Trafodion library for the UDR. The class loader also has the ability to include the Trafodion class path in its search, but I'm not sure this is turned on and I'm also not sure whether this is a good idea from a security point of view. So, I think we may have the following options for the class path of a UDR: * Search in the library jar (done currently) * Also search the Trafodion class path (may or may not be done currently, probably ok for trusted UDRs but may not be ok for isolated UDRs) * Add a directory like $TRAF_HOME/udr/external_libs and all the jars in that directory to the class path for UDRs. This would allow a user to install needed jars in a known location. * In the long term I would also like to add a concept of a "UDR Environment". This would be created with a DDL statement and it would register a tar file. A library could specify the name of an environment, let's call it myenv. When we execute the UDR, we would untar the environment file into $TRAF_HOME/env/myenv and add all the jars in $TRAF_HOME/env/myenv/lib to the classpath of the custom class loader (TBD is how to define the order of the jars). See also https://docs.oracle.com/cd/B19306_01/java.102/b14187/chtwo.htm#BABGJCAJ for how Oracle solves the class loading problems for UDRs. See also TRAFODION-2534. > JVM exception for TMUDF > --- > > Key: TRAFODION-2182 > URL: https://issues.apache.org/jira/browse/TRAFODION-2182 > Project: Apache Trafodion > Issue Type: Bug > Components: foundation >Affects Versions: 2.1-incubating > Environment: TMUDF >Reporter: David Wong >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > This involved jar file containing Lucene components for TMUDF. When run the > TMUDF, get... > *** ERROR[11224] The Java virtual machine raised an exception. Details: > java.lang.IllegalArgumentException: An SPI class of type > org.apache.lucene.codecs.Codec with name 'Lucene60' does not exist. You need > to add the corresponding JAR file supporting this SPI to your classpath. The > current classpath supports the following names: [] [2016-08-23 16:56:47] > Add jar to CLASSPATH in $MY_SQROOT/etc/ms.env is a workaround (didn't need to > bounce anything). Exception does not occur. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2534) UDRs need the ability to define a CLASSPATH or similar concept
[ https://issues.apache.org/jira/browse/TRAFODION-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2534: --- Description: Right now, Java UDRs can only load classes from their own library (a jar file). In many cases, we need to be able to load additional classes. The current solution is to unpack all the jars in a single directory, then create a single jar with all the needed files in it and register that as the library. It works, but is very ugly. (was: Right now, Java UDRs can only load classes from their own library (a jar file). In many cases, we need to be able to load additional classes.) > UDRs need the ability to define a CLASSPATH or similar concept > -- > > Key: TRAFODION-2534 > URL: https://issues.apache.org/jira/browse/TRAFODION-2534 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > Right now, Java UDRs can only load classes from their own library (a jar > file). In many cases, we need to be able to load additional classes. The > current solution is to unpack all the jars in a single directory, then create > a single jar with all the needed files in it and register that as the > library. It works, but is very ugly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2534) UDRs need the ability to define a CLASSPATH or similar concept
[ https://issues.apache.org/jira/browse/TRAFODION-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15916930#comment-15916930 ] Hans Zeller commented on TRAFODION-2534: See also https://docs.oracle.com/cd/B19306_01/java.102/b14187/chtwo.htm#BABGJCAJ and https://www.javacodegeeks.com/2012/10/introduction-to-postgresql-pljava.html for how this works in other database systems. > UDRs need the ability to define a CLASSPATH or similar concept > -- > > Key: TRAFODION-2534 > URL: https://issues.apache.org/jira/browse/TRAFODION-2534 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > Right now, Java UDRs can only load classes from their own library (a jar > file). In many cases, we need to be able to load additional classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2534) UDRs need the ability to define a CLASSPATH or similar concept
Hans Zeller created TRAFODION-2534: -- Summary: UDRs need the ability to define a CLASSPATH or similar concept Key: TRAFODION-2534 URL: https://issues.apache.org/jira/browse/TRAFODION-2534 Project: Apache Trafodion Issue Type: Improvement Components: sql-general Affects Versions: 2.0-incubating Reporter: Hans Zeller Assignee: Hans Zeller Right now, Java UDRs can only load classes from their own library (a jar file). In many cases, we need to be able to load additional classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2517) Allow scalar UDFs with delimited identifiers
[ https://issues.apache.org/jira/browse/TRAFODION-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2517. -- > Allow scalar UDFs with delimited identifiers > > > Key: TRAFODION-2517 > URL: https://issues.apache.org/jira/browse/TRAFODION-2517 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Right now, the Trafodion parser does not accept delimited identifiers in > scalar UDF calls. Note that it does allow creation of those UDFs. > To reproduce, change one of the scalar UDFs in core/sql/regress/udr/TEST107 > to a delimited identifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2517) Allow scalar UDFs with delimited identifiers
[ https://issues.apache.org/jira/browse/TRAFODION-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2517. Resolution: Fixed > Allow scalar UDFs with delimited identifiers > > > Key: TRAFODION-2517 > URL: https://issues.apache.org/jira/browse/TRAFODION-2517 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Right now, the Trafodion parser does not accept delimited identifiers in > scalar UDF calls. Note that it does allow creation of those UDFs. > To reproduce, change one of the scalar UDFs in core/sql/regress/udr/TEST107 > to a delimited identifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2477. -- > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2477. Resolution: Fixed > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2517) Allow scalar UDFs with delimited identifiers
Hans Zeller created TRAFODION-2517: -- Summary: Allow scalar UDFs with delimited identifiers Key: TRAFODION-2517 URL: https://issues.apache.org/jira/browse/TRAFODION-2517 Project: Apache Trafodion Issue Type: Improvement Components: sql-cmp Affects Versions: 2.0-incubating Reporter: Hans Zeller Assignee: Hans Zeller Fix For: 2.2-incubating Right now, the Trafodion parser does not accept delimited identifiers in scalar UDF calls. Note that it does allow creation of those UDFs. To reproduce, change one of the scalar UDFs in core/sql/regress/udr/TEST107 to a delimited identifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (TRAFODION-2517) Allow scalar UDFs with delimited identifiers
[ https://issues.apache.org/jira/browse/TRAFODION-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2517 started by Hans Zeller. -- > Allow scalar UDFs with delimited identifiers > > > Key: TRAFODION-2517 > URL: https://issues.apache.org/jira/browse/TRAFODION-2517 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Right now, the Trafodion parser does not accept delimited identifiers in > scalar UDF calls. Note that it does allow creation of those UDFs. > To reproduce, change one of the scalar UDFs in core/sql/regress/udr/TEST107 > to a delimited identifier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TRAFODION-2515) Question mark is used instead of Unicode replacement character
[ https://issues.apache.org/jira/browse/TRAFODION-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller reassigned TRAFODION-2515: -- Assignee: Hans Zeller > Question mark is used instead of Unicode replacement character > -- > > Key: TRAFODION-2515 > URL: https://issues.apache.org/jira/browse/TRAFODION-2515 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller >Priority: Minor > > When we convert text to a character set and encounter an invalid character, > we should translate it into the "replacement character" of that character > set. For ASCII and ISO-8859-1, we just use a question mark, since there is > not special replacement character. When we convert to Unicode, however, we > should use U+FFFD as the replacement character (often displayed as a black > diamond with a question mark inside). > Test case: > cqd TRANSLATE_ERROR 'off'; > select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from > (values(0))x; > The source value is an invalid bit pattern followed by "A" (0041). Right now > the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". > With the correct replacement character, the result should be EFBFBD41, with > EFBFBD being the UTF-8 encoding of U+FFFD. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891232#comment-15891232 ] Hans Zeller edited comment on TRAFODION-2477 at 3/1/17 10:45 PM: - Right now we don't use the correct replacement character for Unicode. See TRAFODION-2515. was (Author: hzeller): Right now we don't use the correct replacement character for Unicode. > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891232#comment-15891232 ] Hans Zeller commented on TRAFODION-2477: Right now we don't use the correct replacement character for Unicode. > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2515) Question mark is used instead of Unicode replacement character
[ https://issues.apache.org/jira/browse/TRAFODION-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891230#comment-15891230 ] Hans Zeller commented on TRAFODION-2515: The fix for TRAFODION-2477 makes this problem a bit more common, since we allow insertion of replacement characters instead of giving an error. The test case in this JIRA also depends on TRAFODION-2477. > Question mark is used instead of Unicode replacement character > -- > > Key: TRAFODION-2515 > URL: https://issues.apache.org/jira/browse/TRAFODION-2515 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Priority: Minor > > When we convert text to a character set and encounter an invalid character, > we should translate it into the "replacement character" of that character > set. For ASCII and ISO-8859-1, we just use a question mark, since there is > not special replacement character. When we convert to Unicode, however, we > should use U+FFFD as the replacement character (often displayed as a black > diamond with a question mark inside). > Test case: > cqd TRANSLATE_ERROR 'off'; > select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from > (values(0))x; > The source value is an invalid bit pattern followed by "A" (0041). Right now > the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". > With the correct replacement character, the result should be EFBFBD41, with > EFBFBD being the UTF-8 encoding of U+FFFD. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2515) Question mark is used instead of Unicode replacement character
[ https://issues.apache.org/jira/browse/TRAFODION-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2515: --- Summary: Question mark is used instead of Unicode replacement character (was: Question mark instead of Unicode replacement character is used) > Question mark is used instead of Unicode replacement character > -- > > Key: TRAFODION-2515 > URL: https://issues.apache.org/jira/browse/TRAFODION-2515 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Priority: Minor > > When we convert text to a character set and encounter an invalid character, > we should translate it into the "replacement character" of that character > set. For ASCII and ISO-8859-1, we just use a question mark, since there is > not special replacement character. When we convert to Unicode, however, we > should use U+FFFD as the replacement character (often displayed as a black > diamond with a question mark inside). > Test case: > cqd TRANSLATE_ERROR 'off'; > select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from > (values(0))x; > The source value is an invalid bit pattern followed by "A" (0041). Right now > the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". > With the correct replacement character, the result should be EFBFBD41, with > EFBFBD being the UTF-8 encoding of U+FFFD. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2515) Question mark instead of Unicode replacement character is used
[ https://issues.apache.org/jira/browse/TRAFODION-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891191#comment-15891191 ] Hans Zeller commented on TRAFODION-2515: The problem I saw in debugging is that we call unicodeTocset() for this invalid UCS-2/UTF-16 string. This in turn calls UTF16ToLocale() without passing a substitution character. In UTF16ToLocale, we call csc_get_subst_char(), but that method expects the substitution character as an input. Why that is I don't know, given that we also pass the character set to this method, so it could look up the substitution character easily (or maybe that would violate some design principle, if the needed lookup table is in a higher layer). > Question mark instead of Unicode replacement character is used > -- > > Key: TRAFODION-2515 > URL: https://issues.apache.org/jira/browse/TRAFODION-2515 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Priority: Minor > > When we convert text to a character set and encounter an invalid character, > we should translate it into the "replacement character" of that character > set. For ASCII and ISO-8859-1, we just use a question mark, since there is > not special replacement character. When we convert to Unicode, however, we > should use U+FFFD as the replacement character (often displayed as a black > diamond with a question mark inside). > Test case: > cqd TRANSLATE_ERROR 'off'; > select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from > (values(0))x; > The source value is an invalid bit pattern followed by "A" (0041). Right now > the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". > With the correct replacement character, the result should be EFBFBD41, with > EFBFBD being the UTF-8 encoding of U+FFFD. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2515) Question mark instead of Unicode replacement character is used
[ https://issues.apache.org/jira/browse/TRAFODION-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2515: --- Description: When we convert text to a character set and encounter an invalid character, we should translate it into the "replacement character" of that character set. For ASCII and ISO-8859-1, we just use a question mark, since there is not special replacement character. When we convert to Unicode, however, we should use U+FFFD as the replacement character (often displayed as a black diamond with a question mark inside). Test case: cqd TRANSLATE_ERROR 'off'; select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from (values(0))x; The source value is an invalid bit pattern followed by "A" (0041). Right now the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". With the correct replacement character, the result should be EFBFBD41, with EFBFBD being the UTF-8 encoding of U+FFFD. was:When we convert text to a character set and encounter an invalid character, we should translate it into the "replacement character" of that character set. For ASCII and ISO-8859-1, we just use a question mark, since there is not special replacement character. When we convert to Unicode, however, we should use U+FFFD as the replacement character (often displayed as a black diamond with a question mark inside). > Question mark instead of Unicode replacement character is used > -- > > Key: TRAFODION-2515 > URL: https://issues.apache.org/jira/browse/TRAFODION-2515 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-general >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Priority: Minor > > When we convert text to a character set and encounter an invalid character, > we should translate it into the "replacement character" of that character > set. For ASCII and ISO-8859-1, we just use a question mark, since there is > not special replacement character. When we convert to Unicode, however, we > should use U+FFFD as the replacement character (often displayed as a black > diamond with a question mark inside). > Test case: > cqd TRANSLATE_ERROR 'off'; > select converttohex(TRANSLATE(_ucs2 X'D8340041' using UCS2toUTF8)) from > (values(0))x; > The source value is an invalid bit pattern followed by "A" (0041). Right now > the result shows 3F41 as the output, as Unicode or ASCII text this is "?A". > With the correct replacement character, the result should be EFBFBD41, with > EFBFBD being the UTF-8 encoding of U+FFFD. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2515) Question mark instead of Unicode replacement character is used
Hans Zeller created TRAFODION-2515: -- Summary: Question mark instead of Unicode replacement character is used Key: TRAFODION-2515 URL: https://issues.apache.org/jira/browse/TRAFODION-2515 Project: Apache Trafodion Issue Type: Bug Components: sql-general Affects Versions: 2.0-incubating Reporter: Hans Zeller Priority: Minor When we convert text to a character set and encounter an invalid character, we should translate it into the "replacement character" of that character set. For ASCII and ISO-8859-1, we just use a question mark, since there is not special replacement character. When we convert to Unicode, however, we should use U+FFFD as the replacement character (often displayed as a black diamond with a question mark inside). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-52) LP Blueprint: cmp-trafodion-to-hbase-mapping - Make the mapping of Trafodion tables to HBase tables and column families more flexible
[ https://issues.apache.org/jira/browse/TRAFODION-52?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-52. -- Resolution: Duplicate > LP Blueprint: cmp-trafodion-to-hbase-mapping - Make the mapping of Trafodion > tables to HBase tables and column families more flexible > - > > Key: TRAFODION-52 > URL: https://issues.apache.org/jira/browse/TRAFODION-52 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: Hans Zeller >Priority: Minor > > Right now, each Trafodion table is mapped to an HBase table with one column > family. This could be a problem in the future, if a user creates very many > Trafodion tables. It is known that HBase is not designed to have a very high > number of tables and/or regions, due to the high memory footprint that is > caused by every region. > One solution could be to allow multiple Trafodion tables to share a single > HBase table, and to use a 2 byte table id as the key prefix to identify the > table. This would work only for small tables that don't need to be split into > regions and for tables that wouldn't create a hot spot. > One the other side, it may be desirable to separate frequently used columns > in large tables from those that are less frequently used. That could be > achieved by using more than one column family. > Both features - sharing common HBase tables and utilizing multiple column > families - would require new Trafodion SQL syntax. Given the complexity of > the DDL code, it would be a significant effort to implement these features. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2477 started by Hans Zeller. -- > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2499) TMUDF sometimes does not pass errors from its input table up to the caller
[
https://issues.apache.org/jira/browse/TRAFODION-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2499.
Resolution: Fixed
Fix Version/s: 2.2-incubating
> TMUDF sometimes does not pass errors from its input table up to the caller
> --
>
> Key: TRAFODION-2499
> URL: https://issues.apache.org/jira/browse/TRAFODION-2499
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-exe
>Affects Versions: 2.0-incubating
> Environment: Any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.2-incubating
>
>
> This only happens in very limited circumstances. In the test case below, we
> get an error while processing the input table of the UDF. In this case it's a
> duplicate key error. But, the caller of the UDF gets this error message:
> {noformat}
> *** ERROR[8810] Executor ran into an internal failure and returned an error
> without populating the diagnostics area. This error is being injected to
> indicate that.
> Here is the UDF code (file SimpleFunc.java):
> import org.trafodion.sql.udr.*;
> public class SimpleFunc extends UDR {
> // empty constructor
> public SimpleFunc() {}
>
> @Override
> public void describeParamsAndColumns(UDRInvocationInfo info)
>throws UDRException
> {
> info.out().addLongColumn("ROWS_IN", false);
> }
>
> @Override
> public void processData(UDRInvocationInfo info,
> UDRPlanInfo plan)
> throws UDRException
> {
> long rowCount = 0;
> try {
> while (getNextRow(info)) {
> rowCount++;
>
> } // while processData rows
> info.out().setLong(0, rowCount);
> emitRow(info);
> } // try
> catch (Exception ex) {
> throw new UDRException (38110, "processData exception:
> " + ex.getMessage());
> }
> }
>
> }
> Here is how to compile file SimpleFunc.java:
> $JAVA_HOME/bin/javac SimpleFunc.java
> jar cvf simple_func.jar SimpleFunc.*
> Here are the SQL statements:
> -- create library
> create library simple_func file '/simple_func.jar';
> -- create UDF
> drop TABLE_MAPPING FUNCTION simple_func;
> CREATE TABLE_MAPPING FUNCTION simple_func()
> EXTERNAL NAME 'SimpleFunc'
> LIBRARY simple_func
> LANGUAGE JAVA
> ;
> drop table tgt;
> create table tgt (
> pksmallint not null no default,
> val varchar(100),
> primary key (pk)
> )
> attributes aligned format
> ;
> insert into tgt values (1,'x');
> cqd udr_debug_flags '16';
> prepare s from
> SELECT *
> FROM UDF(simple_func(TABLE(
>select pk, val
>from (INSERT INTO tgt values ( 1, 'DAVID')) t
> )))
> ;
> execute s;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889485#comment-15889485 ] Hans Zeller commented on TRAFODION-2477: My plan is to do two things: First, raise errors when we encounter an invalid UCS-2 code point in UCS-2 to UTF-8 conversion. This could lead to new errors in places where we didn't see them before. Second, I'm planning to add two new CQDs: TRANSLATE_ERROR (default is ON) can be set to OFF to allow invalid code points in translation and replace them with a replacement character. A second CQD, TRANSLATE_ERROR_UNICODE_TO_UNICODE (default ON) is just for Unicode to Unicode translations, which should not normally cause any errors. If the newly raised errors cause trouble, they can be turned off, but they no longer cause additional characters to be lost. The invalid characters (invalid UTF-16 surrogate pairs) will be translated into the "replacement character". Luckily, convDoIt has everything that's needed already, all we have to do is set a flag. > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2499) TMUDF sometimes does not pass errors from its input table up to the caller
[
https://issues.apache.org/jira/browse/TRAFODION-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889112#comment-15889112
]
Hans Zeller commented on TRAFODION-2499:
The problem is in ExUdrTcb::insertUpQueueEntry(). If the diags area passed to
this method is the same as the diags area in the down queue entry, then we
actually clear all the errors in it by mistake. I'll add a check for this
situation.
> TMUDF sometimes does not pass errors from its input table up to the caller
> --
>
> Key: TRAFODION-2499
> URL: https://issues.apache.org/jira/browse/TRAFODION-2499
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-exe
>Affects Versions: 2.0-incubating
> Environment: Any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
>
> This only happens in very limited circumstances. In the test case below, we
> get an error while processing the input table of the UDF. In this case it's a
> duplicate key error. But, the caller of the UDF gets this error message:
> {noformat}
> *** ERROR[8810] Executor ran into an internal failure and returned an error
> without populating the diagnostics area. This error is being injected to
> indicate that.
> Here is the UDF code (file SimpleFunc.java):
> import org.trafodion.sql.udr.*;
> public class SimpleFunc extends UDR {
> // empty constructor
> public SimpleFunc() {}
>
> @Override
> public void describeParamsAndColumns(UDRInvocationInfo info)
>throws UDRException
> {
> info.out().addLongColumn("ROWS_IN", false);
> }
>
> @Override
> public void processData(UDRInvocationInfo info,
> UDRPlanInfo plan)
> throws UDRException
> {
> long rowCount = 0;
> try {
> while (getNextRow(info)) {
> rowCount++;
>
> } // while processData rows
> info.out().setLong(0, rowCount);
> emitRow(info);
> } // try
> catch (Exception ex) {
> throw new UDRException (38110, "processData exception:
> " + ex.getMessage());
> }
> }
>
> }
> Here is how to compile file SimpleFunc.java:
> $JAVA_HOME/bin/javac SimpleFunc.java
> jar cvf simple_func.jar SimpleFunc.*
> Here are the SQL statements:
> -- create library
> create library simple_func file '/simple_func.jar';
> -- create UDF
> drop TABLE_MAPPING FUNCTION simple_func;
> CREATE TABLE_MAPPING FUNCTION simple_func()
> EXTERNAL NAME 'SimpleFunc'
> LIBRARY simple_func
> LANGUAGE JAVA
> ;
> drop table tgt;
> create table tgt (
> pksmallint not null no default,
> val varchar(100),
> primary key (pk)
> )
> attributes aligned format
> ;
> insert into tgt values (1,'x');
> cqd udr_debug_flags '16';
> prepare s from
> SELECT *
> FROM UDF(simple_func(TABLE(
>select pk, val
>from (INSERT INTO tgt values ( 1, 'DAVID')) t
> )))
> ;
> execute s;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (TRAFODION-2499) TMUDF sometimes does not pass errors from its input table up to the caller
[
https://issues.apache.org/jira/browse/TRAFODION-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller reassigned TRAFODION-2499:
--
Assignee: Hans Zeller
> TMUDF sometimes does not pass errors from its input table up to the caller
> --
>
> Key: TRAFODION-2499
> URL: https://issues.apache.org/jira/browse/TRAFODION-2499
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-exe
>Affects Versions: 2.0-incubating
> Environment: Any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
>
> This only happens in very limited circumstances. In the test case below, we
> get an error while processing the input table of the UDF. In this case it's a
> duplicate key error. But, the caller of the UDF gets this error message:
> {noformat}
> *** ERROR[8810] Executor ran into an internal failure and returned an error
> without populating the diagnostics area. This error is being injected to
> indicate that.
> Here is the UDF code (file SimpleFunc.java):
> import org.trafodion.sql.udr.*;
> public class SimpleFunc extends UDR {
> // empty constructor
> public SimpleFunc() {}
>
> @Override
> public void describeParamsAndColumns(UDRInvocationInfo info)
>throws UDRException
> {
> info.out().addLongColumn("ROWS_IN", false);
> }
>
> @Override
> public void processData(UDRInvocationInfo info,
> UDRPlanInfo plan)
> throws UDRException
> {
> long rowCount = 0;
> try {
> while (getNextRow(info)) {
> rowCount++;
>
> } // while processData rows
> info.out().setLong(0, rowCount);
> emitRow(info);
> } // try
> catch (Exception ex) {
> throw new UDRException (38110, "processData exception:
> " + ex.getMessage());
> }
> }
>
> }
> Here is how to compile file SimpleFunc.java:
> $JAVA_HOME/bin/javac SimpleFunc.java
> jar cvf simple_func.jar SimpleFunc.*
> Here are the SQL statements:
> -- create library
> create library simple_func file '/simple_func.jar';
> -- create UDF
> drop TABLE_MAPPING FUNCTION simple_func;
> CREATE TABLE_MAPPING FUNCTION simple_func()
> EXTERNAL NAME 'SimpleFunc'
> LIBRARY simple_func
> LANGUAGE JAVA
> ;
> drop table tgt;
> create table tgt (
> pksmallint not null no default,
> val varchar(100),
> primary key (pk)
> )
> attributes aligned format
> ;
> insert into tgt values (1,'x');
> cqd udr_debug_flags '16';
> prepare s from
> SELECT *
> FROM UDF(simple_func(TABLE(
>select pk, val
>from (INSERT INTO tgt values ( 1, 'DAVID')) t
> )))
> ;
> execute s;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2499) TMUDF sometimes does not pass errors from its input table up to the caller
[
https://issues.apache.org/jira/browse/TRAFODION-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881910#comment-15881910
]
Hans Zeller commented on TRAFODION-2499:
This problem was originally found by David Wong.
> TMUDF sometimes does not pass errors from its input table up to the caller
> --
>
> Key: TRAFODION-2499
> URL: https://issues.apache.org/jira/browse/TRAFODION-2499
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-exe
>Affects Versions: 2.0-incubating
> Environment: Any
>Reporter: Hans Zeller
>
> This only happens in very limited circumstances. In the test case below, we
> get an error while processing the input table of the UDF. In this case it's a
> duplicate key error. But, the caller of the UDF gets this error message:
> {noformat}
> *** ERROR[8810] Executor ran into an internal failure and returned an error
> without populating the diagnostics area. This error is being injected to
> indicate that.
> Here is the UDF code (file SimpleFunc.java):
> import org.trafodion.sql.udr.*;
> public class SimpleFunc extends UDR {
> // empty constructor
> public SimpleFunc() {}
>
> @Override
> public void describeParamsAndColumns(UDRInvocationInfo info)
>throws UDRException
> {
> info.out().addLongColumn("ROWS_IN", false);
> }
>
> @Override
> public void processData(UDRInvocationInfo info,
> UDRPlanInfo plan)
> throws UDRException
> {
> long rowCount = 0;
> try {
> while (getNextRow(info)) {
> rowCount++;
>
> } // while processData rows
> info.out().setLong(0, rowCount);
> emitRow(info);
> } // try
> catch (Exception ex) {
> throw new UDRException (38110, "processData exception:
> " + ex.getMessage());
> }
> }
>
> }
> Here is how to compile file SimpleFunc.java:
> $JAVA_HOME/bin/javac SimpleFunc.java
> jar cvf simple_func.jar SimpleFunc.*
> Here are the SQL statements:
> -- create library
> create library simple_func file '/simple_func.jar';
> -- create UDF
> drop TABLE_MAPPING FUNCTION simple_func;
> CREATE TABLE_MAPPING FUNCTION simple_func()
> EXTERNAL NAME 'SimpleFunc'
> LIBRARY simple_func
> LANGUAGE JAVA
> ;
> drop table tgt;
> create table tgt (
> pksmallint not null no default,
> val varchar(100),
> primary key (pk)
> )
> attributes aligned format
> ;
> insert into tgt values (1,'x');
> cqd udr_debug_flags '16';
> prepare s from
> SELECT *
> FROM UDF(simple_func(TABLE(
>select pk, val
>from (INSERT INTO tgt values ( 1, 'DAVID')) t
> )))
> ;
> execute s;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TRAFODION-2499) TMUDF sometimes does not pass errors from its input table up to the caller
Hans Zeller created TRAFODION-2499:
--
Summary: TMUDF sometimes does not pass errors from its input table
up to the caller
Key: TRAFODION-2499
URL: https://issues.apache.org/jira/browse/TRAFODION-2499
Project: Apache Trafodion
Issue Type: Bug
Components: sql-exe
Affects Versions: 2.0-incubating
Environment: Any
Reporter: Hans Zeller
This only happens in very limited circumstances. In the test case below, we get
an error while processing the input table of the UDF. In this case it's a
duplicate key error. But, the caller of the UDF gets this error message:
{noformat}
*** ERROR[8810] Executor ran into an internal failure and returned an error
without populating the diagnostics area. This error is being injected to
indicate that.
Here is the UDF code (file SimpleFunc.java):
import org.trafodion.sql.udr.*;
public class SimpleFunc extends UDR {
// empty constructor
public SimpleFunc() {}
@Override
public void describeParamsAndColumns(UDRInvocationInfo info)
throws UDRException
{
info.out().addLongColumn("ROWS_IN", false);
}
@Override
public void processData(UDRInvocationInfo info,
UDRPlanInfo plan)
throws UDRException
{
long rowCount = 0;
try {
while (getNextRow(info)) {
rowCount++;
} // while processData rows
info.out().setLong(0, rowCount);
emitRow(info);
} // try
catch (Exception ex) {
throw new UDRException (38110, "processData exception:
" + ex.getMessage());
}
}
}
Here is how to compile file SimpleFunc.java:
$JAVA_HOME/bin/javac SimpleFunc.java
jar cvf simple_func.jar SimpleFunc.*
Here are the SQL statements:
-- create library
create library simple_func file '/simple_func.jar';
-- create UDF
drop TABLE_MAPPING FUNCTION simple_func;
CREATE TABLE_MAPPING FUNCTION simple_func()
EXTERNAL NAME 'SimpleFunc'
LIBRARY simple_func
LANGUAGE JAVA
;
drop table tgt;
create table tgt (
pksmallint not null no default,
val varchar(100),
primary key (pk)
)
attributes aligned format
;
insert into tgt values (1,'x');
cqd udr_debug_flags '16';
prepare s from
SELECT *
FROM UDF(simple_func(TABLE(
select pk, val
from (INSERT INTO tgt values ( 1, 'DAVID')) t
)))
;
execute s;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (TRAFODION-1987) Enable min/max optimization for TYPE 1 repartitioned joins
[ https://issues.apache.org/jira/browse/TRAFODION-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-1987: --- Fix Version/s: (was: 2.1-incubating) 2.2-incubating > Enable min/max optimization for TYPE 1 repartitioned joins > -- > > Key: TRAFODION-1987 > URL: https://issues.apache.org/jira/browse/TRAFODION-1987 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 1.3-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > Right now, min/max optimization for hash joins (making min and max values of > equi-join columns available to the outer child tree of a hash join) is > limited to operators in the same plan fragment. In other words, an > ESP_EXCHANGE between a scan and a hash join makes it impossible to do this > optimization. We could be a bit less restrictive: If the hash table of the > hash join isn't partitioned, either because it runs in a single instance, or > because its right (inner) child is replicated, then we can do this > optimization across exchanges. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2182) JVM exception for TMUDF
[ https://issues.apache.org/jira/browse/TRAFODION-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2182: --- Fix Version/s: (was: 2.1-incubating) 2.2-incubating > JVM exception for TMUDF > --- > > Key: TRAFODION-2182 > URL: https://issues.apache.org/jira/browse/TRAFODION-2182 > Project: Apache Trafodion > Issue Type: Bug > Components: foundation >Affects Versions: 2.1-incubating > Environment: TMUDF >Reporter: David Wong >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > This involved jar file containing Lucene components for TMUDF. When run the > TMUDF, get... > *** ERROR[11224] The Java virtual machine raised an exception. Details: > java.lang.IllegalArgumentException: An SPI class of type > org.apache.lucene.codecs.Codec with name 'Lucene60' does not exist. You need > to add the corresponding JAR file supporting this SPI to your classpath. The > current classpath supports the following names: [] [2016-08-23 16:56:47] > Add jar to CLASSPATH in $MY_SQROOT/etc/ms.env is a workaround (didn't need to > bounce anything). Exception does not occur. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2349) Solve issue with mapping of value ids for statistics for common subexpr
[ https://issues.apache.org/jira/browse/TRAFODION-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2349: --- Fix Version/s: 2.2-incubating > Solve issue with mapping of value ids for statistics for common subexpr > --- > > Key: TRAFODION-2349 > URL: https://issues.apache.org/jira/browse/TRAFODION-2349 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmp >Affects Versions: 2.1-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When we introduce a temp table, we want to "transplant" the statistics from > the original query tree representing the common subexpression to the new scan > and its ancestors. This requires two mappings, one from the original tree to > the temp table columns, the other from the temp table cols to the > characteristic outputs of the MapValueIds above the temp scan. > While most of these mappings are done in terms of VEGRefs, the statistics use > BaseColumns instead. Right now, we map the BaseColumns of single column stats > to VEGRefs in the new tree. We might be able to do the same for MC stats. > Or, alternatively, we may need to decided on specific BaseColumns to map to. > See methods Scan::synthEstLogProp() and MapValueIds::synthEstLogProp(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2349) Solve issue with mapping of value ids for statistics for common subexpr
[ https://issues.apache.org/jira/browse/TRAFODION-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866712#comment-15866712 ] Hans Zeller commented on TRAFODION-2349: Part of this was addressed by https://github.com/apache/incubator-trafodion/pull/928. This fixed many issues with single column stats, but there is still some work missing for multi-column stats. > Solve issue with mapping of value ids for statistics for common subexpr > --- > > Key: TRAFODION-2349 > URL: https://issues.apache.org/jira/browse/TRAFODION-2349 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmp >Affects Versions: 2.1-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When we introduce a temp table, we want to "transplant" the statistics from > the original query tree representing the common subexpression to the new scan > and its ancestors. This requires two mappings, one from the original tree to > the temp table columns, the other from the temp table cols to the > characteristic outputs of the MapValueIds above the temp scan. > While most of these mappings are done in terms of VEGRefs, the statistics use > BaseColumns instead. Right now, we map the BaseColumns of single column stats > to VEGRefs in the new tree. We might be able to do the same for MC stats. > Or, alternatively, we may need to decided on specific BaseColumns to map to. > See methods Scan::synthEstLogProp() and MapValueIds::synthEstLogProp(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
[ https://issues.apache.org/jira/browse/TRAFODION-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2477: --- Fix Version/s: 2.2-incubating > Invalid characters in UCS2 to UTF8 translation are not handled correctly > > > Key: TRAFODION-2477 > URL: https://issues.apache.org/jira/browse/TRAFODION-2477 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... > UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. > However, if we encounter invalid code points or invalid UTF-16 surrogate > pairs, those could raise errors. Right now we just suppress those errors. > Instead we should either translate them to the Unicode "replacement > character" U+FFFD or we should raise an error. Ideally, we should have a CQD > that decides which of these two actions to take. > Test case: > create table tbaducs2(a char(10) character set ucs2); > -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid > insert into tbaducs2 values(_ucs2 X'DC41'); > select translate(a using ucs2toutf8) from tbaducs2; > -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TRAFODION-2127) enhance Trafodion implementation of WITH clause
[ https://issues.apache.org/jira/browse/TRAFODION-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller updated TRAFODION-2127: --- Fix Version/s: 2.2-incubating > enhance Trafodion implementation of WITH clause > --- > > Key: TRAFODION-2127 > URL: https://issues.apache.org/jira/browse/TRAFODION-2127 > Project: Apache Trafodion > Issue Type: Improvement >Reporter: liu ming >Assignee: Hans Zeller > Fix For: 2.2-incubating > > > TRAFODION-1673 described some details about how to support WITH clause in > Trafodion. > As initial implementation, we use a simple pure-parser method. > That way, Trafodion can support WITH clause functionally, but not good from > performance point of view, > also need to enhance the parser to be more strict in syntax. > This JIRA is a follow up JIRA, to track following effort to support Trafodion > WITH clause. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2477) Invalid characters in UCS2 to UTF8 translation are not handled correctly
Hans Zeller created TRAFODION-2477: -- Summary: Invalid characters in UCS2 to UTF8 translation are not handled correctly Key: TRAFODION-2477 URL: https://issues.apache.org/jira/browse/TRAFODION-2477 Project: Apache Trafodion Issue Type: Bug Components: sql-cmp Affects Versions: 2.0-incubating Reporter: Hans Zeller Assignee: Hans Zeller When translating from UCS-2 to UTF-8, using CAST or TRANSLATE(... UCS2TOUTF8), all valid characters will map easily to a UTF-8 character. However, if we encounter invalid code points or invalid UTF-16 surrogate pairs, those could raise errors. Right now we just suppress those errors. Instead we should either translate them to the Unicode "replacement character" U+FFFD or we should raise an error. Ideally, we should have a CQD that decides which of these two actions to take. Test case: create table tbaducs2(a char(10) character set ucs2); -- DC00 is a low-order UTF-16 surrogate, on its own this is invalid insert into tbaducs2 values(_ucs2 X'DC41'); select translate(a using ucs2toutf8) from tbaducs2; -- this returns an empty string - no error, no replacement character -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-1826) Error 1389 when dropping external Hive table and Hive table has changed
[
https://issues.apache.org/jira/browse/TRAFODION-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller closed TRAFODION-1826.
--
> Error 1389 when dropping external Hive table and Hive table has changed
> ---
>
> Key: TRAFODION-1826
> URL: https://issues.apache.org/jira/browse/TRAFODION-1826
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 1.1 (pre-incubation)
> Environment: Any
>Reporter: Hans Zeller
>Assignee: Roberta Marton
>
> When Trafodion looks up the corresponding Trafodion external table for a Hive
> table, it compares the two column layouts and raises error 1389 when they
> don't match. It also does that in the DROP EXTERNAL TABLE command, and that
> shouldn't happen. We should be able to drop an external table even (and
> especially!) if it doesn't match the underlying table.
> Hive:
> {quote}create table t(a int);{quote}
> Trafodion:
> {quote}create external table t for hive.hive.t;{quote}
> Hive:
> {quote}drop table t;
> create table t(b int);{quote}
> Trafodion:
> {quote}drop external table t for hive.hive.t;
> ERROR 1389: Object TRAFODION."_HV_HIVE_".T does not exist in
> Trafodion.{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2467) HBase region start key straddling a datetime key value gives error or crashes
[ https://issues.apache.org/jira/browse/TRAFODION-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2467. Resolution: Fixed > HBase region start key straddling a datetime key value gives error or crashes > - > > Key: TRAFODION-2467 > URL: https://issues.apache.org/jira/browse/TRAFODION-2467 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > > There is no simple test case for this, but Wei-Shiun has observed a problem > that fits the theory, and I was able to force a problem in a debugger. > The issue is this: HBase creates region start keys, and the length of those > keys may provide only a part (a prefix) of a column value. So, HBase may have > a 2 byte region start key of \x07\xCE for a table with a DATE key. This > provides the year, 1998 in this case, but not month or day. Normally, we > extend this partial key with zeroes for most data types, including > date/time/timestamp. However, for these datetime types we need to add logic > that set the year, month and day fields to 1 if extending the key with zeroes > would make it 0 otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TRAFODION-2467) HBase region start key straddling a datetime key value gives error or crashes
[ https://issues.apache.org/jira/browse/TRAFODION-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2467. -- > HBase region start key straddling a datetime key value gives error or crashes > - > > Key: TRAFODION-2467 > URL: https://issues.apache.org/jira/browse/TRAFODION-2467 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > > There is no simple test case for this, but Wei-Shiun has observed a problem > that fits the theory, and I was able to force a problem in a debugger. > The issue is this: HBase creates region start keys, and the length of those > keys may provide only a part (a prefix) of a column value. So, HBase may have > a 2 byte region start key of \x07\xCE for a table with a DATE key. This > provides the year, 1998 in this case, but not month or day. Normally, we > extend this partial key with zeroes for most data types, including > date/time/timestamp. However, for these datetime types we need to add logic > that set the year, month and day fields to 1 if extending the key with zeroes > would make it 0 otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TRAFODION-2467) HBase region start key straddling a datetime key value gives error or crashes
Hans Zeller created TRAFODION-2467: -- Summary: HBase region start key straddling a datetime key value gives error or crashes Key: TRAFODION-2467 URL: https://issues.apache.org/jira/browse/TRAFODION-2467 Project: Apache Trafodion Issue Type: Bug Components: sql-cmp Affects Versions: 2.0-incubating Environment: Any Reporter: Hans Zeller Assignee: Hans Zeller There is no simple test case for this, but Wei-Shiun has observed a problem that fits the theory, and I was able to force a problem in a debugger. The issue is this: HBase creates region start keys, and the length of those keys may provide only a part (a prefix) of a column value. So, HBase may have a 2 byte region start key of \x07\xCE for a table with a DATE key. This provides the year, 1998 in this case, but not month or day. Normally, we extend this partial key with zeroes for most data types, including date/time/timestamp. However, for these datetime types we need to add logic that set the year, month and day fields to 1 if extending the key with zeroes would make it 0 otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TRAFODION-2457) regress/compGeneral/TEST042 shows non-deterministic results
[ https://issues.apache.org/jira/browse/TRAFODION-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2457. Resolution: Fixed Fix Version/s: 2.1-incubating > regress/compGeneral/TEST042 shows non-deterministic results > --- > > Key: TRAFODION-2457 > URL: https://issues.apache.org/jira/browse/TRAFODION-2457 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Anoop Sharma >Assignee: Hans Zeller > Fix For: 2.1-incubating > > > compGeneral/TEST042 runs tests for query caching feature and > validates it by looking at the number of cache hits and other parameters. > These values change based on where this test is being run, > what the environment is and other changes. > It fails off and on and folks have been updating expected files > or known diff files to handle it. > Because of the non-deterministic behavior, the test as it is currently > written is unreliable and developers/jenkins failures are ignored. > The test suite need to be modified and written so it doesn't make > the cache hit/miss values as part of expected output. Or some other > way to tolerate a range or the difference in output. > For this checkin, this test has been disabled. A proper way to write > this test will be discussed and once that is done, it will be reenabled. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (TRAFODION-2456) TMUDF doesn't work in this scenario with its parallism
[
https://issues.apache.org/jira/browse/TRAFODION-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2456.
Resolution: Resolved
> TMUDF doesn't work in this scenario with its parallism
> --
>
> Key: TRAFODION-2456
> URL: https://issues.apache.org/jira/browse/TRAFODION-2456
> Project: Apache Trafodion
> Issue Type: Improvement
>Reporter: Kevin Xu
>Assignee: Hans Zeller
>Priority: Critical
> Attachments: ParTest2.java
>
>
> Attached ParTest2.java.
> Expected to print 6 lines, but got only 1 line.
> Workaround is to add the following lines:
> @Override
> public void describeStatistics(UDRInvocationInfo info) throws
> UDRException {
> info.out().setEstimatedNumRows(100);
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (TRAFODION-2385) Add support in clients for boolean, tinyint, largeint unsigned datatypes
Title: Message Title Hans Zeller resolved as Fixed Apache Trafodion / TRAFODION-2385 Add support in clients for boolean, tinyint, largeint unsigned datatypes Change By: Hans Zeller Resolution: Fixed Fix Version/s: 2.1-incubating Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2214) add support of OLAP function LAG and LEAD
Title: Message Title Hans Zeller updated an issue Apache Trafodion / TRAFODION-2214 add support of OLAP function LAG and LEAD Change By: Hans Zeller Issue Type: Sub-task New Feature Parent: TRAFODION-2060 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2214) add support of OLAP function LAG and LEAD
Title: Message Title Hans Zeller updated an issue Apache Trafodion / TRAFODION-2214 add support of OLAP function LAG and LEAD Change By: Hans Zeller Issue Type: New Feature Sub-task Parent: TRAFODION-2060 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2214) add support of OLAP function LAG and LEAD
Title: Message Title Hans Zeller closed an issue as Fixed Apache Trafodion / TRAFODION-2214 add support of OLAP function LAG and LEAD Change By: Hans Zeller Status: Resolved Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2214) add support of OLAP function LAG and LEAD
Title: Message Title Hans Zeller resolved as Fixed Apache Trafodion / TRAFODION-2214 add support of OLAP function LAG and LEAD Change By: Hans Zeller Resolution: Fixed Fix Version/s: 2.1-incubating Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2315) Heuristic decision on when to use a temp table for WITH clause
Title: Message Title Hans Zeller closed an issue as Fixed Apache Trafodion / TRAFODION-2315 Heuristic decision on when to use a temp table for WITH clause Change By: Hans Zeller Status: Resolved Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (TRAFODION-2315) Heuristic decision on when to use a temp table for WITH clause
Title: Message Title Hans Zeller resolved as Fixed Apache Trafodion / TRAFODION-2315 Heuristic decision on when to use a temp table for WITH clause Change By: Hans Zeller Resolution: Fixed Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Created] (TRAFODION-2438) Missing index rows with UPSERT, executor/test015
Hans Zeller created TRAFODION-2438:
--
Summary: Missing index rows with UPSERT, executor/test015
Key: TRAFODION-2438
URL: https://issues.apache.org/jira/browse/TRAFODION-2438
Project: Apache Trafodion
Issue Type: Bug
Components: sql-cmp
Affects Versions: 2.0-incubating
Reporter: Hans Zeller
Assignee: Hans Zeller
Fix For: 2.1-incubating
Selva found the cause of this problem and suggested the fix.
The problem was discovered as a failure of regression test executor/test015,
with some missing index rows.
Selva found that this is because we insert a "tombstone" into each alternate
index for every row, even if it is a new row. Those tombstones for the
non-existing old rows contain junk or NULL values, and they can potentially
wipe out existing index rows.
Selva's suggested fix is to use the "pre-condition" for the index delete to
check whether an old row actually existed, and only do a delete if it did.
Here is a simple test case, also from Selva:
{noformat}
drop table t015t8 cascade ;
create table t015t8 (i int not null, j int, k int, primary key(i));
create unique index t015t8i1 on t015t8(j);
insert into t015t8 values (4,null,4);
select * from t015t8;
-- enable this CQD to try the new, efficient tree
-- cqd TRAF_UPSERT_TO_EFF_TREE 'on';
prepare s1 from upsert into t015t8 values (1,2,3),(11,12,13) ;
execute s1 ;
set parserflags 1;
select * from t015t8;
select * from table (index_table t015t8i1);
-- wrong answer, 2 rows instead of 3
{noformat}
If this does not reproduce the error or if you want to see the extra
tombstones, do an HBase raw scan to see them:
{noformat}
hbase shell
scan 'TRAFODION.SCH.T015T8I1' , {RAW=>true}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TRAFODION-2437) Implement a monitor for communications between ESPs of a fragment
Hans Zeller created TRAFODION-2437: -- Summary: Implement a monitor for communications between ESPs of a fragment Key: TRAFODION-2437 URL: https://issues.apache.org/jira/browse/TRAFODION-2437 Project: Apache Trafodion Issue Type: New Feature Components: sql-exe Reporter: Hans Zeller Assignee: Hans Zeller Priority: Minor Today, the ESPs that are executing parallel fragment instances can't talk to each other. They all do their work independently and may finish at different times. This works fine for normal parallel algorithms, but it somewhat restricts what we can do in parallel. Sometimes, a communication or maybe just a simple synchronization would be desirable. Examples: - Make all ESPs wait until the last ESP has finished before proceeding. This could create a global blocking operator, for example when we need a blocking operator to prevent the Halloween problem. - Dynamic range-repartitioning: After doing local sorts, negotiate common split boundaries to perform a range-repartitioning, so that we can produce a globally sorted result in parallel. - Communication between parallel instances of TMUDFs. Some TMUDFs may need similar communication, which could allow them to do more things in parallel. When implementing this, we may also need to pay some attention to potential deadlocks, although with blocking operators like sorts there should not be any deadlock issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2165) Select min() returns wrong result
[ https://issues.apache.org/jira/browse/TRAFODION-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2165. Resolution: Resolved Fix Version/s: 2.1-incubating This problem does not happen anymore, but unfortunately I can't pinpoint where and how it got fixed. > Select min() returns wrong result > - > > Key: TRAFODION-2165 > URL: https://issues.apache.org/jira/browse/TRAFODION-2165 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-exe >Affects Versions: 2.1-incubating >Reporter: Weishiun Tsai >Assignee: Hans Zeller > Fix For: 2.1-incubating > > > As shown below, the following sequence of statements ends with a select min() > query. It should have returned 10, as the query returns 13 rows with the same > u1 values of 10 without min(). But it returns 0 right now. > >>create schema mytest2; > --- SQL operation complete. > >>set schema mytest2; > --- SQL operation complete. > >> > >>create table OPTABLE > +>( p1 largeint not null > +>, u1 smallint unsigned > +>, zi1 smallint not null > +>, f1 double precision > +>, n1 numeric (4,2) unsigned > +>, d1 decimal (4,2) > +>, t1 date > +>, c1 char > +>, p2 integer not null > +>, u2 integer unsigned > +>, zi2 integer not null > +>, f2 real > +>, n2 numeric (6,3) unsigned > +>, d2 decimal (6,3) > +>, t2 time > +>, c2 char(2) > +>, p3 smallint not null > +>, u3 largeint > +>, zi3 largeint not null > +>, f3 float > +>, n3 numeric (12,4) > +>, d3 decimal (12,4) > +>, t3 interval hour to second > +>, c3 char(3) > +>, z char (10) > +>, primary key (p1, p2, p3) ) > +>; > --- SQL operation complete. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1959-12-31', 'a' , > +>9, 9, 9, 9, 9, 9, time '23:59:59', 'aa', > +>9,null, -1,null,null,null, null, null, null > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>10, 10, 10, 10, 10, 10, time '00:00:00', 'aa' , > +>10, 10, 10, 10, 10, 10, interval '00:00:00' hour to second, 'aaa', 'Row01' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>10, 10, 10, 10, 10, 10, time '00:00:15', 'aa' , > +>20, 20, 20, 20, 20, 20, interval '00:00:15' hour to second, 'aab', 'Row02' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>10, 10, 10, 10, 10, 10, time '00:00:30', 'aa' , > +>30, 30, 30, 30, 30, 30, interval '00:00:30' hour to second, 'aac', 'Row03' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>20, 20, 20, 20, 20, 20, time '00:00:45', 'ab' , > +>10, 10, 10, 10, 10, 10, interval '00:00:45' hour to second, 'aba', 'Row04' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>20, 20, 20, 20, 20, 20, time '00:01:00', 'ab' , > +>20, 20, 20, 20, 20, 20, interval '00:01:00' hour to second, 'abb', 'Row05' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>20, 20, 20, 20, 20, 20, time '00:01:15', 'ab' , > +>30, 30, 30, 30, 30, 30, interval '00:01:15' hour to second, 'abc', 'Row06' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>30, 30, 30, 30, 30, 30, time '00:01:30', 'ac' , > +>10, 10, 10, 10, 10, 10, interval '00:01:30' hour to second, 'aca', 'Row07' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>30, 30, 30, 30, 30, 30, time '00:01:45', 'ac' , > +>20, 20, 20, 20, 20, 20, interval '00:01:45' hour to second, 'acb', 'Row08' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>10, 10, 10, 10, 10, 10, date '1960-01-01', 'a' , > +>30, 30, 30, 30, 30, 30, time '00:02:00', 'ac' , > +>30, 30, 30, 30, 30, 30, null, 'acc', 'Row09' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>20, 20, 20, 20, 20, 20, date '1960-01-01', 'b' , > +>10, 10, 10, 10, 10, 10, null, 'ba' , > +>10, 10, 10, 10, 10, 10, interval '00:02:00' hour to second, 'baa', 'Row10' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>20, 20, 20, 20, 20, 20, date '1960-01-01', 'b' , > +>10, 10, 10, 10, 10, 10, time '00:59:00', 'ba' , > +>20, 20, 20, 20, 20, 20, interval '00:59:00' hour to second, 'bab', 'Row11' > +>); > --- 1 row(s) inserted. > >> > >>insert into OPTABLE values ( > +>20, 20, 20, 20, 20, 20, date '1960-01-01', 'b' , >
[jira] [Resolved] (TRAFODION-2383) Support TINYINT for TMUDFs
[
https://issues.apache.org/jira/browse/TRAFODION-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2383.
Resolution: Fixed
Fix Version/s: 2.1-incubating
Merged into Trafodion master on 12/23/2016 with
https://github.com/apache/incubator-trafodion/pull/882.
> Support TINYINT for TMUDFs
> --
>
> Key: TRAFODION-2383
> URL: https://issues.apache.org/jira/browse/TRAFODION-2383
> Project: Apache Trafodion
> Issue Type: Sub-task
> Components: sql-cmp
>Affects Versions: 2.1-incubating
>Reporter: Hans Zeller
>Assignee: Hans Zeller
>Priority: Minor
> Fix For: 2.1-incubating
>
>
> We need to update the TMUDF interfaces for C++ and Java to support TINYINT as
> well.
> Here is an example where we run into problems:
> Create the UDF shown in TRAFODION-2382.
> Run this query:
> {code:sql}
> select *
> from udf(join_udf(16, 'A', 'B'));
> {code}
> This will the following error, since the integer parameter is considered a
> NUMERIC with a length of 1 byte:
> {code}
> *** ERROR[11252] TypeInfo::getLong() and getDouble() not supported for SQL
> type 4 (SQLSTATE 38902)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2320) Make subquery unnesting work with common subexpressions
[ https://issues.apache.org/jira/browse/TRAFODION-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2320. Resolution: Not A Problem The subquery unnesting transformations happen in Join::semanticQueryOptimizeNode(). This method first calls semanticQueryOptimizeNode() on the children before attempting any unnesting transformations. Therefore the CommonSubExprRef nodes will be eliminated by the time we reach it, and therefore those nodes don't cause a problem. Testing TPC-DS queries with CSEs enabled and also warnings for missing unnesting operations enabled confirmed that this is not an issue. > Make subquery unnesting work with common subexpressions > --- > > Key: TRAFODION-2320 > URL: https://issues.apache.org/jira/browse/TRAFODION-2320 > Project: Apache Trafodion > Issue Type: Sub-task > Components: sql-cmp >Affects Versions: 2.1-incubating >Reporter: Hans Zeller >Assignee: Hans Zeller > > The subquery unnesting code relies on a certain structure of the query tree, > with join and groupby nodes arranged in a certain way. If we have > CommonSubExprRef nodes present in the tree, the unnesting logic doesn't > recognize some of the patterns and subquery unnesting doesn't take place. > Hopefully we can somehow ignore these extra nodes in subquery unnesting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2418) Allow group by push-down to a fact table
[
https://issues.apache.org/jira/browse/TRAFODION-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2418.
Resolution: Fixed
> Allow group by push-down to a fact table
>
>
> Key: TRAFODION-2418
> URL: https://issues.apache.org/jira/browse/TRAFODION-2418
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> Trafodion has a rule that pushes a groupby down over a join when possible,
> but it will not push the groupby to the left child of a join - it relies on
> join commutativity. This might have been ok 20 years ago when this was coded
> (by me...), but it isn't good for some situations today. Example:
> {noformat}
> select d.y, count(f.a), sum(f.b)
> from big_fact f join small_dim d on f.x=d.y
> where d.a = 1
> group by d.y
> {noformat}
> The plan we would like is a hash join with the group by on big_fact as the
> left child. To do this, we need to remove the heuristic that prevents this
> form of push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (TRAFODION-2418) Allow group by push-down to a fact table
[
https://issues.apache.org/jira/browse/TRAFODION-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller reopened TRAFODION-2418:
Sorry, jumped the gun here. This fix is not yet checked in.
> Allow group by push-down to a fact table
>
>
> Key: TRAFODION-2418
> URL: https://issues.apache.org/jira/browse/TRAFODION-2418
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> Trafodion has a rule that pushes a groupby down over a join when possible,
> but it will not push the groupby to the left child of a join - it relies on
> join commutativity. This might have been ok 20 years ago when this was coded
> (by me...), but it isn't good for some situations today. Example:
> {noformat}
> select d.y, count(f.a), sum(f.b)
> from big_fact f join small_dim d on f.x=d.y
> where d.a = 1
> group by d.y
> {noformat}
> The plan we would like is a hash join with the group by on big_fact as the
> left child. To do this, we need to remove the heuristic that prevents this
> form of push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2392) Avoid a costly sort for highly reducing TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller resolved TRAFODION-2392. Resolution: Fixed Fix Version/s: 2.1-incubating Fix checked in on 12/23/2016 with https://github.com/apache/incubator-trafodion/pull/882 > Avoid a costly sort for highly reducing TMUDFs > -- > > Key: TRAFODION-2392 > URL: https://issues.apache.org/jira/browse/TRAFODION-2392 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.1-incubating > > > When an input table with a PARTITION BY is specified in a TMUDF, the > Trafodion optimizer ensures that the input rows are sorted on (a permutation > of) the PARTITION BY columns, so that each parallel TMUDF instance sees the > input rows of such a logical partition in contiguous rows. This way the TMUDF > can process each group separately. > This is usually a good way to process the data, except when we are dealing > with a large input table and a TMUDF that highly reduces the input data. In > that case it may be better to maintain a hash table of groups in the TMUDF > and to avoid the costly sort of the input table. > My proposal is to add a new function type to UDRInvocationInfo.FunctionType, > called REDUCER_NC (for Non-Contiguous). Setting the function type to this new > type would indicate to the optimizer not to request a sort order on the > partitioning columns. > The table below shows how the function type and PARTITION BY and ORDER BY > clauses would determine the effective sort order produced by the optimizer: > ||Function type||PARTITION BY||ORDER BY||Data is sorted by|| > |REDUCER (existing)|a,b|c,d|a,b,c,d| > |REDUCER (existing)|a,b||a,b| > |REDUCER_NC (proposed)|a,b|c,d|c,d| > |REDUCER_NC (proposed)|a,b||| > In all other aspects, REDUCER and REDUCER_NC function types would behave the > same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (TRAFODION-2392) Avoid a costly sort for highly reducing TMUDFs
[ https://issues.apache.org/jira/browse/TRAFODION-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Zeller closed TRAFODION-2392. -- > Avoid a costly sort for highly reducing TMUDFs > -- > > Key: TRAFODION-2392 > URL: https://issues.apache.org/jira/browse/TRAFODION-2392 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: Any >Reporter: Hans Zeller >Assignee: Hans Zeller > Fix For: 2.1-incubating > > > When an input table with a PARTITION BY is specified in a TMUDF, the > Trafodion optimizer ensures that the input rows are sorted on (a permutation > of) the PARTITION BY columns, so that each parallel TMUDF instance sees the > input rows of such a logical partition in contiguous rows. This way the TMUDF > can process each group separately. > This is usually a good way to process the data, except when we are dealing > with a large input table and a TMUDF that highly reduces the input data. In > that case it may be better to maintain a hash table of groups in the TMUDF > and to avoid the costly sort of the input table. > My proposal is to add a new function type to UDRInvocationInfo.FunctionType, > called REDUCER_NC (for Non-Contiguous). Setting the function type to this new > type would indicate to the optimizer not to request a sort order on the > partitioning columns. > The table below shows how the function type and PARTITION BY and ORDER BY > clauses would determine the effective sort order produced by the optimizer: > ||Function type||PARTITION BY||ORDER BY||Data is sorted by|| > |REDUCER (existing)|a,b|c,d|a,b,c,d| > |REDUCER (existing)|a,b||a,b| > |REDUCER_NC (proposed)|a,b|c,d|c,d| > |REDUCER_NC (proposed)|a,b||| > In all other aspects, REDUCER and REDUCER_NC function types would behave the > same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (TRAFODION-2418) Allow group by push-down to a fact table
[
https://issues.apache.org/jira/browse/TRAFODION-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller closed TRAFODION-2418.
--
> Allow group by push-down to a fact table
>
>
> Key: TRAFODION-2418
> URL: https://issues.apache.org/jira/browse/TRAFODION-2418
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> Trafodion has a rule that pushes a groupby down over a join when possible,
> but it will not push the groupby to the left child of a join - it relies on
> join commutativity. This might have been ok 20 years ago when this was coded
> (by me...), but it isn't good for some situations today. Example:
> {noformat}
> select d.y, count(f.a), sum(f.b)
> from big_fact f join small_dim d on f.x=d.y
> where d.a = 1
> group by d.y
> {noformat}
> The plan we would like is a hash join with the group by on big_fact as the
> left child. To do this, we need to remove the heuristic that prevents this
> form of push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2418) Allow group by push-down to a fact table
[
https://issues.apache.org/jira/browse/TRAFODION-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2418.
Resolution: Fixed
Fix Version/s: 2.1-incubating
Fix checked in on 12/23/2016 with
https://github.com/apache/incubator-trafodion/pull/882
> Allow group by push-down to a fact table
>
>
> Key: TRAFODION-2418
> URL: https://issues.apache.org/jira/browse/TRAFODION-2418
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> Trafodion has a rule that pushes a groupby down over a join when possible,
> but it will not push the groupby to the left child of a join - it relies on
> join commutativity. This might have been ok 20 years ago when this was coded
> (by me...), but it isn't good for some situations today. Example:
> {noformat}
> select d.y, count(f.a), sum(f.b)
> from big_fact f join small_dim d on f.x=d.y
> where d.a = 1
> group by d.y
> {noformat}
> The plan we would like is a hash join with the group by on big_fact as the
> left child. To do this, we need to remove the heuristic that prevents this
> form of push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (TRAFODION-2399) Syntax error on a string with double charset qualifier on load from salted table
[
https://issues.apache.org/jira/browse/TRAFODION-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller closed TRAFODION-2399.
--
> Syntax error on a string with double charset qualifier on load from salted
> table
>
>
> Key: TRAFODION-2399
> URL: https://issues.apache.org/jira/browse/TRAFODION-2399
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating
> Environment: any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> This problem was reported by Eric Owhadi.
> Eric saw this error message when doing a load that had a salted table as a
> source and that was using a selection predicate in the source select:
> {noformat}
> *** ERROR[3127] An invalid character was found in identifier
> _ISO88591_ISO88591.
> *** ERROR[15001] A syntax error occurred at or before:
> CAST ( _ISO88591_ISO88591'' AS CHAR(1) CHARACTER SET ISO88591 COLLATE DEFAULT)
> ^ (11 characters from start of SQL statement)
> *** ERROR[2235] Compiler Internal Error: An unknown error, originated from
> file ../optimizer/EncodedKeyValue.cpp at line 223.
> {noformat}
> Here is a simple test case to reproduce the issue:
> {noformat}
> drop table summarized_items;
> CREATE TABLE summarized_items
> (has_item_list CHAR(1) NOT NULL,
> item_type CHAR(1) NOT NULL,
> PRIMARY KEY (has_item_list,
> item_type))
> SALT USING 2 PARTITIONS ON
> (item_type)
> ;
> drop table summarized_items2;
> CREATE TABLE summarized_items2
> (
> item_type CHAR(1) NOT NULL)
> ;
> cqd COMP_BOOL_226 'on';
> prepare s from
> LOAD TRANSFORM INTO summarized_items2 select item_type from summarized_items
> where has_item_list = 'F';
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TRAFODION-2399) Syntax error on a string with double charset qualifier on load from salted table
[
https://issues.apache.org/jira/browse/TRAFODION-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller resolved TRAFODION-2399.
Resolution: Fixed
Fix checked in on 12/23/2016 with
https://github.com/apache/incubator-trafodion/pull/882
> Syntax error on a string with double charset qualifier on load from salted
> table
>
>
> Key: TRAFODION-2399
> URL: https://issues.apache.org/jira/browse/TRAFODION-2399
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating
> Environment: any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
>
> This problem was reported by Eric Owhadi.
> Eric saw this error message when doing a load that had a salted table as a
> source and that was using a selection predicate in the source select:
> {noformat}
> *** ERROR[3127] An invalid character was found in identifier
> _ISO88591_ISO88591.
> *** ERROR[15001] A syntax error occurred at or before:
> CAST ( _ISO88591_ISO88591'' AS CHAR(1) CHARACTER SET ISO88591 COLLATE DEFAULT)
> ^ (11 characters from start of SQL statement)
> *** ERROR[2235] Compiler Internal Error: An unknown error, originated from
> file ../optimizer/EncodedKeyValue.cpp at line 223.
> {noformat}
> Here is a simple test case to reproduce the issue:
> {noformat}
> drop table summarized_items;
> CREATE TABLE summarized_items
> (has_item_list CHAR(1) NOT NULL,
> item_type CHAR(1) NOT NULL,
> PRIMARY KEY (has_item_list,
> item_type))
> SALT USING 2 PARTITIONS ON
> (item_type)
> ;
> drop table summarized_items2;
> CREATE TABLE summarized_items2
> (
> item_type CHAR(1) NOT NULL)
> ;
> cqd COMP_BOOL_226 'on';
> prepare s from
> LOAD TRANSFORM INTO summarized_items2 select item_type from summarized_items
> where has_item_list = 'F';
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TRAFODION-2399) Syntax error on a string with double charset qualifier on load from salted table
[
https://issues.apache.org/jira/browse/TRAFODION-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hans Zeller updated TRAFODION-2399:
---
Fix Version/s: 2.1-incubating
> Syntax error on a string with double charset qualifier on load from salted
> table
>
>
> Key: TRAFODION-2399
> URL: https://issues.apache.org/jira/browse/TRAFODION-2399
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating
> Environment: any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
> Fix For: 2.1-incubating
>
>
> This problem was reported by Eric Owhadi.
> Eric saw this error message when doing a load that had a salted table as a
> source and that was using a selection predicate in the source select:
> {noformat}
> *** ERROR[3127] An invalid character was found in identifier
> _ISO88591_ISO88591.
> *** ERROR[15001] A syntax error occurred at or before:
> CAST ( _ISO88591_ISO88591'' AS CHAR(1) CHARACTER SET ISO88591 COLLATE DEFAULT)
> ^ (11 characters from start of SQL statement)
> *** ERROR[2235] Compiler Internal Error: An unknown error, originated from
> file ../optimizer/EncodedKeyValue.cpp at line 223.
> {noformat}
> Here is a simple test case to reproduce the issue:
> {noformat}
> drop table summarized_items;
> CREATE TABLE summarized_items
> (has_item_list CHAR(1) NOT NULL,
> item_type CHAR(1) NOT NULL,
> PRIMARY KEY (has_item_list,
> item_type))
> SALT USING 2 PARTITIONS ON
> (item_type)
> ;
> drop table summarized_items2;
> CREATE TABLE summarized_items2
> (
> item_type CHAR(1) NOT NULL)
> ;
> cqd COMP_BOOL_226 'on';
> prepare s from
> LOAD TRANSFORM INTO summarized_items2 select item_type from summarized_items
> where has_item_list = 'F';
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
