[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947548#comment-13947548 ] Alan Gates commented on HIVE-6499: -- Ran the tests locally, all tests pass. Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.2.patch, HIVE-6499.patch, HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[
https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936054#comment-13936054
]
Hive QA commented on HIVE-6499:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634599/HIVE-6499.2.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1795/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1795/console
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/hwi/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hwi ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hwi ---
[INFO] Compiling 2 source files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hwi ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hwi ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hwi ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hwi ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive ODBC 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-odbc ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/odbc (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-odbc ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-odbc ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-odbc ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-odbc ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-odbc ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/odbc/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-odbc/0.14.0-SNAPSHOT/hive-odbc-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Shims Aggregator 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-aggregator
---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-shims-aggregator ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[
https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934209#comment-13934209
]
Hive QA commented on HIVE-6499:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634158/HIVE-6499.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1767/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1767/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1767/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'contrib/src/test/results/clientnegative/udtf_explode2.q.out'
Reverted 'contrib/src/test/results/clientnegative/case_with_row_sequence.q.out'
Reverted 'contrib/src/test/results/clientnegative/invalid_row_sequence.q.out'
Reverted 'contrib/src/test/results/clientpositive/udaf_example_min_n.q.out'
Reverted 'contrib/src/test/results/clientpositive/udaf_example_max_n.q.out'
Reverted 'contrib/src/test/results/clientpositive/lateral_view_explode2.q.out'
Reverted 'contrib/src/test/results/clientpositive/udaf_example_avg.q.out'
Reverted
'contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out'
Reverted
'contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out'
Reverted 'contrib/src/test/results/clientpositive/udaf_example_min.q.out'
Reverted 'contrib/src/test/results/clientpositive/udf_row_sequence.q.out'
Reverted 'contrib/src/test/results/clientpositive/udaf_example_max.q.out'
Reverted 'contrib/src/test/results/clientpositive/udf_example_add.q.out'
Reverted 'contrib/src/test/results/clientpositive/udf_example_format.q.out'
Reverted 'contrib/src/test/results/clientpositive/udtf_output_on_close.q.out'
Reverted 'contrib/src/test/results/clientpositive/udtf_explode2.q.out'
Reverted 'contrib/src/test/results/clientpositive/dboutput.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_local_resource.q.out'
Reverted
'ql/src/test/results/clientnegative/udf_function_does_not_implement_udf.q.out'
Reverted 'ql/src/test/results/clientnegative/create_unknown_udf_udaf.q.out'
Reverted 'ql/src/test/results/clientnegative/create_function_nonudf_class.q.out'
Reverted
'ql/src/test/results/clientnegative/create_function_nonexistent_db.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_test_error_reduce.q.out'
Reverted 'ql/src/test/results/clientnegative/drop_native_udf.q.out'
Reverted 'ql/src/test/results/clientnegative/create_udaf_failure.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out'
Reverted 'ql/src/test/results/clientnegative/create_unknown_genericudf.q.out'
Reverted 'ql/src/test/results/clientnegative/cluster_tasklog_retrieval.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_test_error.q.out'
Reverted
'ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out'
Reverted 'ql/src/test/results/clientpositive/create_func1.q.out'
Reverted 'ql/src/test/results/clientpositive/macro.q.out'
Reverted 'ql/src/test/results/clientpositive/create_view.q.out'
Reverted 'ql/src/test/results/clientpositive/udaf_sum_list.q.out'
Reverted 'ql/src/test/results/clientpositive/create_genericudaf.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_logic_java_boolean.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_compare_java_string.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_testlength.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_testlength2.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_udf.q.out'
Reverted 'ql/src/test/results/clientpositive/autogen_colalias.q.out'
Reverted 'ql/src/test/results/clientpositive/create_udaf.q.out'
Reverted 'ql/src/test/results/clientpositive/create_genericudf.q.out'
Reverted 'ql/src/test/results/clientpositive/ptf_register_tblfn.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_context_aware.q.out'
Reverted 'ql/src/test/results/clientpositive/windowing_udaf2.q.out'
Reverted 'ql/src/test/results/clientpositive/compile_processor.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_using.q.out'
Reverted
[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934332#comment-13934332 ] Sushanth Sowmyan commented on HIVE-6499: Looks like the patch needed to be updated after the changes in HIVE-3938. Regenerating. Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.patch, HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912174#comment-13912174 ] Thejas M Nair commented on HIVE-6499: - +1 Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13911327#comment-13911327 ] Sushanth Sowmyan commented on HIVE-6499: Created review board link : https://reviews.apache.org/r/18456/ [~thejas], could you please review this? [~owen.omalley], this is a bug that you pinged me about, if you could test that it works for your usecase as well, that'd be great. Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
