[jira] [Commented] (IMPALA-8379) TestMtDopParquet.test_parquet_filtering flaky - exhaustive run
[
https://issues.apache.org/jira/browse/IMPALA-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819749#comment-16819749
]
Tim Armstrong commented on IMPALA-8379:
---
I think we thought that this was IMPALA-341, but it appears not. Let me see if
I can repro
> TestMtDopParquet.test_parquet_filtering flaky - exhaustive run
> --
>
> Key: IMPALA-8379
> URL: https://issues.apache.org/jira/browse/IMPALA-8379
> Project: IMPALA
> Issue Type: Bug
>Reporter: Gabor Kaszab
>Assignee: Tim Armstrong
>Priority: Critical
>
> Jenkins job output
> {code:java}
> 06:33:37 query_test/test_mt_dop.py:103: in test_parquet_filtering
> 06:33:37 self.run_test_case('QueryTest/parquet-filtering', vector)
> 06:33:37 common/impala_test_suite.py:446: in run_test_case
> 06:33:37 verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> 06:33:37 common/test_result_verifier.py:569: in verify_runtime_profile
> 06:33:37 "\n\nPROFILE:\n%s\n" % (function, field, expected_value,
> actual_value, actual))
> 06:33:37 E AssertionError: Aggregation of SUM over NumRowGroups did not
> match expected results.
> 06:33:37 E EXPECTED VALUE:
> 06:33:37 E 24
> 06:33:37 E
> 06:33:37 E ACTUAL VALUE:
> 06:33:37 E 22
> 06:33:37 E
> 06:33:37 E PROFILE:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8336) TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
[
https://issues.apache.org/jira/browse/IMPALA-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8336.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
> -
>
> Key: IMPALA-8336
> URL: https://issues.apache.org/jira/browse/IMPALA-8336
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> The test expects the memory limit to be exceeded, but it doesn't happen:
> {noformat}
> query_test.test_nested_types.TestNestedTypes.test_tpch_mem_limit_single_node[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> orc/def/block]
> query_test/test_nested_types.py:123: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:487: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex: .*Memory limit exceeded:
> Failed to allocate [0-9]+ bytes for collection
> 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.*{noformat}
> Seen once on Centos 6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8336) TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
[
https://issues.apache.org/jira/browse/IMPALA-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8336.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
> -
>
> Key: IMPALA-8336
> URL: https://issues.apache.org/jira/browse/IMPALA-8336
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> The test expects the memory limit to be exceeded, but it doesn't happen:
> {noformat}
> query_test.test_nested_types.TestNestedTypes.test_tpch_mem_limit_single_node[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> orc/def/block]
> query_test/test_nested_types.py:123: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:487: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex: .*Memory limit exceeded:
> Failed to allocate [0-9]+ bytes for collection
> 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.*{noformat}
> Seen once on Centos 6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-8329) Update the Ranger minicluster with various fixes
[ https://issues.apache.org/jira/browse/IMPALA-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819742#comment-16819742 ] ASF subversion and git services commented on IMPALA-8329: - Commit 5fa076e95cfbfcc044dc14cbb20af825936af82a in impala's branch refs/heads/master from Fredy Wijaya [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5fa076e ] IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. The patch also fixes some TODOs to replace the rangerPlugin.init() hack with rangerPlugin.refreshPoliciesAndTags() API available in this Ranger build. Testing: - Ran core tests - Manually verified that no regression when starting Hive 3 with USE_CDP_HIVE=true Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Reviewed-on: http://gerrit.cloudera.org:8080/13002 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Update the Ranger minicluster with various fixes > > > Key: IMPALA-8329 > URL: https://issues.apache.org/jira/browse/IMPALA-8329 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > > Update the Ranger minicluster once the various fixes are in. The TODOs in > Impala that rely on those fixes will need to be resolved as well. > https://issues.apache.org/jira/browse/RANGER-2374 > https://issues.apache.org/jira/browse/RANGER-2349 > https://issues.apache.org/jira/browse/RANGER-2367 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8392) make -j $IMPALA_BUILD_THREADS docker_images fails with OSError: [Errno 17] File exists: '/home/ubuntu/Impala/docker/build_context'
[
https://issues.apache.org/jira/browse/IMPALA-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819717#comment-16819717
]
Tim Armstrong commented on IMPALA-8392:
---
https://gitlab.kitware.com/cmake/cmake/issues/17585
> make -j $IMPALA_BUILD_THREADS docker_images fails with OSError: [Errno 17]
> File exists: '/home/ubuntu/Impala/docker/build_context'
> --
>
> Key: IMPALA-8392
> URL: https://issues.apache.org/jira/browse/IMPALA-8392
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Minor
>
> Looks like there is a bug in the CMake file. I didn't see this with serial
> make or with ninja at all. I use ninja locally which is why I didn't notice
> this while developing it.
> {noformat}
> 02:45:52 Scanning dependencies of target catalogd_image
> 02:45:52 Scanning dependencies of target coord_exec_image
> 02:45:52 Scanning dependencies of target statestored_image
> 02:45:52 Scanning dependencies of target coordinator_image
> 02:45:52 Scanning dependencies of target executor_image
> 02:45:52 [ 97%] Creating impala base build context.
> 02:45:52 [ 97%] Creating impala base build context.
> 02:45:52 [ 97%] Creating impala base build context.
> 02:45:52 [ 98%] Creating impala base build context.
> 02:45:52 [ 98%] Creating impala base build context.
> 02:45:52 Traceback (most recent call last):
> 02:45:52 Traceback (most recent call last):
> 02:45:52 Traceback (most recent call last):
> 02:45:52 File "./setup_build_context.py", line 38, in
> 02:45:52 File "./setup_build_context.py", line 38, in
> 02:45:52 File "./setup_build_context.py", line 38, in
> 02:45:52 os.makedirs(OUTPUT_DIR)
> 02:45:52 os.makedirs(OUTPUT_DIR)
> 02:45:52 os.makedirs(OUTPUT_DIR)
> 02:45:52 File
> "/home/ubuntu/Impala/bin/../infra/python/env/lib/python2.7/os.py", line 157,
> in makedirs
> 02:45:52 File
> "/home/ubuntu/Impala/bin/../infra/python/env/lib/python2.7/os.py", line 157,
> in makedirs
> 02:45:52 File
> "/home/ubuntu/Impala/bin/../infra/python/env/lib/python2.7/os.py", line 157,
> in makedirs
> 02:45:52 mkdir(name, mode)
> 02:45:52 mkdir(name, mode)
> 02:45:52 mkdir(name, mode)
> 02:45:52 OSErrorOSErrorOSError: [Errno 17] File exists:
> '/home/ubuntu/Impala/docker/build_context'
> 02:45:52 : [Errno 17] File exists:
> '/home/ubuntu/Impala/docker/build_context':
> 02:45:52 [Errno 17] File exists: '/home/ubuntu/Impala/docker/build_context'
> 02:45:52 Traceback (most recent call last):
> 02:45:52 File "./setup_build_context.py", line 60, in
> 02:45:52 symlink_file_into_dir(so, LIB_DIR)
> 02:45:52 File "./setup_build_context.py", line 51, in symlink_file_into_dir
> 02:45:52 os.symlink(src_file, os.path.join(dst_dir,
> os.path.basename(src_file)))
> 02:45:52 OSError: [Errno 17] File exists
> 02:45:52 docker/CMakeFiles/executor_image.dir/build.make:74: recipe for
> target 'docker/build_context/.timestamp' failed
> 02:45:52 make[3]: *** [docker/build_context/.timestamp] Error 1
> 02:45:52 docker/CMakeFiles/catalogd_image.dir/build.make:74: recipe for
> target 'docker/build_context/.timestamp' failed
> 02:45:52 make[3]: *** [docker/build_context/.timestamp] Error 1
> 02:45:52 CMakeFiles/Makefile2:13731: recipe for target
> 'docker/CMakeFiles/executor_image.dir/all' failed
> 02:45:52 make[2]: *** [docker/CMakeFiles/executor_image.dir/all] Error 2
> 02:45:52 make[2]: *** Waiting for unfinished jobs
> 02:45:52 CMakeFiles/Makefile2:13665: recipe for target
> 'docker/CMakeFiles/catalogd_image.dir/all' failed
> 02:45:52 make[2]: *** [docker/CMakeFiles/catalogd_image.dir/all] Error 2
> 02:45:52 docker/CMakeFiles/coordinator_image.dir/build.make:74: recipe for
> target 'docker/build_context/.timestamp' failed
> 02:45:52 make[3]: *** [docker/build_context/.timestamp] Error 1
> 02:45:52 docker/CMakeFiles/coord_exec_image.dir/build.make:74: recipe for
> target 'docker/build_context/.timestamp' failed
> 02:45:52 make[3]: *** [docker/build_context/.timestamp] Error 1
> 02:45:52 CMakeFiles/Makefile2:13797: recipe for target
> 'docker/CMakeFiles/coord_exec_image.dir/all' failed
> 02:45:52 make[2]: *** [docker/CMakeFiles/coord_exec_image.dir/all] Error 2
> 02:45:52 CMakeFiles/Makefile2:13764: recipe for target
> 'docker/CMakeFiles/coordinator_image.dir/all' failed
> 02:45:52 make[2]: *** [docker/CMakeFiles/coordinator_image.dir/all] Error 2
> 02:45:52 [ 98%] Building Impala base docker image.
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/33/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (IMPALA-3825) Distribute runtime filter aggregation across cluster
[ https://issues.apache.org/jira/browse/IMPALA-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-3825: - Assignee: Abhishek Rawat (was: Rahul Shivu Mahadev) > Distribute runtime filter aggregation across cluster > > > Key: IMPALA-3825 > URL: https://issues.apache.org/jira/browse/IMPALA-3825 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 2.6.0 >Reporter: Henry Robinson >Assignee: Abhishek Rawat >Priority: Major > Labels: runtime-filters > > Runtime filters can be tens of MB or more, and incasting all filters from all > shuffle joins to the coordinator can put a lot of memory pressure on that > node. To alleviate this we should consider spreading out the aggregation > operation across the cluster, so that a different node aggregates each > runtime filter. > This still restricts aggregation to #runtime-filters nodes, which will > usually be less than the cluster size. If we want to smooth that out further > we could use tree-based aggregation, but let's measure the benefits of simply > distributing the aggregation work first. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8375) Add metrics for spill disk usage
[ https://issues.apache.org/jira/browse/IMPALA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819702#comment-16819702 ] ASF subversion and git services commented on IMPALA-8375: - Commit a80ebb7a729c1ffd25db262527d4261e01763646 in impala's branch refs/heads/master from Abhishek Rawat [ https://gitbox.apache.org/repos/asf?p=impala.git;h=a80ebb7 ] IMPALA-8375: Add metrics for spill disk usage Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark & tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current value for spilled bytes, respectively. A new class AtomicHighWaterMarkGauge was added to keep track of the HWM value. The new class also encapsulates a metric object which keeps track of the current value for the spilled bytes. The current value is incremented every time a new range is allocated from a temporary file. The current value for spilled bytes is decremented when a temporary file is closed. The new metrics are not updated when ranges are recycled from a file. We can add a new metric in future for keeping track of actual spilled bytes. The HWM value is updated whenever the current value is greater than the HWM value. Testing: - Added new unit tests to the metrics-test test case. - E2E testing for both the metrics by running concurrent spilling queries and ensuring that both the current value metric and the HWM metric were behaving as expected. Ran concurrent queries and monitored the metrics on the impala daemon's metric page. Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Reviewed-on: http://gerrit.cloudera.org:8080/12956 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins > Add metrics for spill disk usage > > > Key: IMPALA-8375 > URL: https://issues.apache.org/jira/browse/IMPALA-8375 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Tim Armstrong >Assignee: Abhishek Rawat >Priority: Major > > * High water mark of bytes allocated > * Maybe - some metric of utilisation (what % of bytes allocated are actually > being used) > * Maybe - total bytes allocated over the lifetime of the process > * Maybe - statistics about the sizes of blocks allocated -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8428) Add support for caching file handles on s3
Joe McDonnell created IMPALA-8428: - Summary: Add support for caching file handles on s3 Key: IMPALA-8428 URL: https://issues.apache.org/jira/browse/IMPALA-8428 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 3.3.0 Reporter: Joe McDonnell The file handle cache is currently disabled for S3, as the S3 connector needed to implement proper unbuffer support. Now that https://issues.apache.org/jira/browse/HADOOP-14747 is fixed, Impala should provide an option to cache S3 file handles. This is particularly important for data caching, as accessing the data cache happens after obtaining a file handle. If getting a file handle is slow, the caching will be less effective. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8428) Add support for caching file handles on s3
Joe McDonnell created IMPALA-8428: - Summary: Add support for caching file handles on s3 Key: IMPALA-8428 URL: https://issues.apache.org/jira/browse/IMPALA-8428 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 3.3.0 Reporter: Joe McDonnell The file handle cache is currently disabled for S3, as the S3 connector needed to implement proper unbuffer support. Now that https://issues.apache.org/jira/browse/HADOOP-14747 is fixed, Impala should provide an option to cache S3 file handles. This is particularly important for data caching, as accessing the data cache happens after obtaining a file handle. If getting a file handle is slow, the caching will be less effective. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8415) test_corrupt_stats exhaustive test failed
[
https://issues.apache.org/jira/browse/IMPALA-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-8415.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> test_corrupt_stats exhaustive test failed
> -
>
> Key: IMPALA-8415
> URL: https://issues.apache.org/jira/browse/IMPALA-8415
> Project: IMPALA
> Issue Type: Test
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Attila Jeges
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> IMPALA-6050 broke test_corrupt_stats exhaustive test:
> {code:java}
> metadata/test_compute_stats.py:238: in test_corrupt_stats
> self.run_test_case('QueryTest/corrupt-stats', vector, unique_database)
> common/impala_test_suite.py:517: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:370: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:449: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:239: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' partitions=1/2 files=1 size=24B'
> E Items in actual results:
> E '02:EXCHANGE [UNPARTITIONED]'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=3'
> E '| output: count(*)'
> E '| row-size=8B cardinality=0'
> E 'test_corrupt_stats_bb50e9c7.corrupted'
> E 'WARNING: The following tables have potentially corrupt table
> statistics.'
> E 'Per-Host Resource Estimates: Memory=52MB'
> E '03:AGGREGATE [FINALIZE]'
> E ' partition predicates: org = 1'
> E 'PLAN-ROOT SINK'
> E ''
> E '| output: count:merge(*)'
> E ' HDFS partitions=1/2 files=1 size=24B'
> E ' row-size=0B cardinality=0'
> E '|'
> E '00:SCAN HDFS [test_corrupt_stats_bb50e9c7.corrupted]'
> E '01:AGGREGATE'
> E 'Drop and re-compute statistics to resolve this problem.'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8415) test_corrupt_stats exhaustive test failed
[
https://issues.apache.org/jira/browse/IMPALA-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-8415.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> test_corrupt_stats exhaustive test failed
> -
>
> Key: IMPALA-8415
> URL: https://issues.apache.org/jira/browse/IMPALA-8415
> Project: IMPALA
> Issue Type: Test
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Attila Jeges
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> IMPALA-6050 broke test_corrupt_stats exhaustive test:
> {code:java}
> metadata/test_compute_stats.py:238: in test_corrupt_stats
> self.run_test_case('QueryTest/corrupt-stats', vector, unique_database)
> common/impala_test_suite.py:517: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:370: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:449: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:239: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' partitions=1/2 files=1 size=24B'
> E Items in actual results:
> E '02:EXCHANGE [UNPARTITIONED]'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=3'
> E '| output: count(*)'
> E '| row-size=8B cardinality=0'
> E 'test_corrupt_stats_bb50e9c7.corrupted'
> E 'WARNING: The following tables have potentially corrupt table
> statistics.'
> E 'Per-Host Resource Estimates: Memory=52MB'
> E '03:AGGREGATE [FINALIZE]'
> E ' partition predicates: org = 1'
> E 'PLAN-ROOT SINK'
> E ''
> E '| output: count:merge(*)'
> E ' HDFS partitions=1/2 files=1 size=24B'
> E ' row-size=0B cardinality=0'
> E '|'
> E '00:SCAN HDFS [test_corrupt_stats_bb50e9c7.corrupted]'
> E '01:AGGREGATE'
> E 'Drop and re-compute statistics to resolve this problem.'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-6050) Query profiles should clearly indicate storage layer(s) used
[ https://issues.apache.org/jira/browse/IMPALA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819562#comment-16819562 ] ASF subversion and git services commented on IMPALA-6050: - Commit c0a6aad28dd1efaf388f0cfeaa056995213d56e5 in impala's branch refs/heads/master from Joe McDonnell [ https://gitbox.apache.org/repos/asf?p=impala.git;h=c0a6aad ] IMPALA-8415: Fix tests broken by storage layer information Storage layer information was added to the query profile by IMPALA-6050. This broke some tests on exhaustive and s3 runs due to changes in formatting. This fixes the issues: 1. Replace HDFS SCAN with $FILESYSTEM_NAME SCAN in some test files 2. Add $FILESYSTEM_NAME to partition information string Testing: - Ran exhaustive HDFS tests - Ran s3 tests Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Reviewed-on: http://gerrit.cloudera.org:8080/13025 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Query profiles should clearly indicate storage layer(s) used > > > Key: IMPALA-6050 > URL: https://issues.apache.org/jira/browse/IMPALA-6050 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Sailesh Mukil >Assignee: Sahil Takiar >Priority: Major > Labels: adls, profile, s3, supportability > Fix For: Impala 3.3.0 > > > Currently, the query profile doesn't have the location of tables and > partitions, which makes it hard to figure out what storage layer a > table/partition that was queried was on. > As we're seeing more users run Impala workloads against cloud based storage > like S3 and ADLS, we should have the query profiles show this information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8415) test_corrupt_stats exhaustive test failed
[
https://issues.apache.org/jira/browse/IMPALA-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819561#comment-16819561
]
ASF subversion and git services commented on IMPALA-8415:
-
Commit c0a6aad28dd1efaf388f0cfeaa056995213d56e5 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c0a6aad ]
IMPALA-8415: Fix tests broken by storage layer information
Storage layer information was added to the query profile by
IMPALA-6050. This broke some tests on exhaustive and s3 runs
due to changes in formatting.
This fixes the issues:
1. Replace HDFS SCAN with $FILESYSTEM_NAME SCAN in some test files
2. Add $FILESYSTEM_NAME to partition information string
Testing:
- Ran exhaustive HDFS tests
- Ran s3 tests
Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872
Reviewed-on: http://gerrit.cloudera.org:8080/13025
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> test_corrupt_stats exhaustive test failed
> -
>
> Key: IMPALA-8415
> URL: https://issues.apache.org/jira/browse/IMPALA-8415
> Project: IMPALA
> Issue Type: Test
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Attila Jeges
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> IMPALA-6050 broke test_corrupt_stats exhaustive test:
> {code:java}
> metadata/test_compute_stats.py:238: in test_corrupt_stats
> self.run_test_case('QueryTest/corrupt-stats', vector, unique_database)
> common/impala_test_suite.py:517: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:370: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:449: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:239: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' partitions=1/2 files=1 size=24B'
> E Items in actual results:
> E '02:EXCHANGE [UNPARTITIONED]'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=3'
> E '| output: count(*)'
> E '| row-size=8B cardinality=0'
> E 'test_corrupt_stats_bb50e9c7.corrupted'
> E 'WARNING: The following tables have potentially corrupt table
> statistics.'
> E 'Per-Host Resource Estimates: Memory=52MB'
> E '03:AGGREGATE [FINALIZE]'
> E ' partition predicates: org = 1'
> E 'PLAN-ROOT SINK'
> E ''
> E '| output: count:merge(*)'
> E ' HDFS partitions=1/2 files=1 size=24B'
> E ' row-size=0B cardinality=0'
> E '|'
> E '00:SCAN HDFS [test_corrupt_stats_bb50e9c7.corrupted]'
> E '01:AGGREGATE'
> E 'Drop and re-compute statistics to resolve this problem.'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8427) Document the behavior change in IMPALA-7800
Michael Ho created IMPALA-8427:
--
Summary: Document the behavior change in IMPALA-7800
Key: IMPALA-8427
URL: https://issues.apache.org/jira/browse/IMPALA-8427
Project: IMPALA
Issue Type: Documentation
Components: Clients
Affects Versions: Impala 3.3.0
Reporter: Michael Ho
Assignee: Alex Rodoni
IMPALA-7800 changes the default behavior of client connection timeout with HS2
and Beeswax Thrift servers. Quote from the commit message:
{noformat}
The current implementation of the FE thrift server waits
indefinitely to open the new session, if the maximum number of
FE service threads specified by --fe_service_threads has been
allocated.
This patch introduces a startup flag to control how the server
should treat new connection requests if we have run out of the
configured number of server threads.
If --accepted_client_cnxn_timeout > 0, new connection requests are
rejected by the server if we can't get a server thread within
the specified timeout.
We set the default timeout to be 5 minutes. The old behavior
can be restored by setting --accepted_client_cnxn_timeout=0,
i.e., no timeout. The timeout applies only to client facing thrift
servers, i.e., HS2 and Beeswax servers.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8427) Document the behavior change in IMPALA-7800
Michael Ho created IMPALA-8427:
--
Summary: Document the behavior change in IMPALA-7800
Key: IMPALA-8427
URL: https://issues.apache.org/jira/browse/IMPALA-8427
Project: IMPALA
Issue Type: Documentation
Components: Clients
Affects Versions: Impala 3.3.0
Reporter: Michael Ho
Assignee: Alex Rodoni
IMPALA-7800 changes the default behavior of client connection timeout with HS2
and Beeswax Thrift servers. Quote from the commit message:
{noformat}
The current implementation of the FE thrift server waits
indefinitely to open the new session, if the maximum number of
FE service threads specified by --fe_service_threads has been
allocated.
This patch introduces a startup flag to control how the server
should treat new connection requests if we have run out of the
configured number of server threads.
If --accepted_client_cnxn_timeout > 0, new connection requests are
rejected by the server if we can't get a server thread within
the specified timeout.
We set the default timeout to be 5 minutes. The old behavior
can be restored by setting --accepted_client_cnxn_timeout=0,
i.e., no timeout. The timeout applies only to client facing thrift
servers, i.e., HS2 and Beeswax servers.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819545#comment-16819545
]
Michael Ho commented on IMPALA-7802:
bq. On the coordinator nodes, we could see sockets open and had to resort to
way to kill the socket as root. I would expect the remote OS to clean up the
sockets when the process died/killed but somehow that wasn't working.
I am pretty sure the connections will be closed on the coordinator side and the
session will be closed if the remote client is terminated.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8426) Logging error in DROP_TABLE event processing.
Bharathkrishna Guruvayoor Murali created IMPALA-8426:
Summary: Logging error in DROP_TABLE event processing.
Key: IMPALA-8426
URL: https://issues.apache.org/jira/browse/IMPALA-8426
Project: IMPALA
Issue Type: Bug
Reporter: Bharathkrishna Guruvayoor Murali
Assignee: Bharathkrishna Guruvayoor Murali
We have the following logging code in DROP_TABLE process() method, here always
the first condition is triggered and the second will never be matched. So we
should switch the order of the conditions.
{code:java}
if (!tblMatched.getRef()) {
LOG.warn(debugString("Table %s was not removed from "
+ "catalog since the creation time of the table did not match", tblName_));
} else if (!tblWasFound.getRef()) {
debugLog("Table {} was not removed since it did not exist in catalog.",
tblName_);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8426) Logging error in DROP_TABLE event processing.
[
https://issues.apache.org/jira/browse/IMPALA-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated IMPALA-8426:
-
Priority: Minor (was: Major)
> Logging error in DROP_TABLE event processing.
> -
>
> Key: IMPALA-8426
> URL: https://issues.apache.org/jira/browse/IMPALA-8426
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
>
> We have the following logging code in DROP_TABLE process() method, here
> always the first condition is triggered and the second will never be matched.
> So we should switch the order of the conditions.
> {code:java}
> if (!tblMatched.getRef()) {
> LOG.warn(debugString("Table %s was not removed from "
> + "catalog since the creation time of the table did not match", tblName_));
> } else if (!tblWasFound.getRef()) {
> debugLog("Table {} was not removed since it did not exist in catalog.",
> tblName_);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8426) Logging error in DROP_TABLE event processing.
Bharathkrishna Guruvayoor Murali created IMPALA-8426:
Summary: Logging error in DROP_TABLE event processing.
Key: IMPALA-8426
URL: https://issues.apache.org/jira/browse/IMPALA-8426
Project: IMPALA
Issue Type: Bug
Reporter: Bharathkrishna Guruvayoor Murali
Assignee: Bharathkrishna Guruvayoor Murali
We have the following logging code in DROP_TABLE process() method, here always
the first condition is triggered and the second will never be matched. So we
should switch the order of the conditions.
{code:java}
if (!tblMatched.getRef()) {
LOG.warn(debugString("Table %s was not removed from "
+ "catalog since the creation time of the table did not match", tblName_));
} else if (!tblWasFound.getRef()) {
debugLog("Table {} was not removed since it did not exist in catalog.",
tblName_);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8425) Evaluate size of Impala docker containers
Tim Armstrong created IMPALA-8425: - Summary: Evaluate size of Impala docker containers Key: IMPALA-8425 URL: https://issues.apache.org/jira/browse/IMPALA-8425 Project: IMPALA Issue Type: Sub-task Components: Infrastructure Reporter: Tim Armstrong Assignee: Tim Armstrong We should take a look at the size of the containers produced by the build and see if it is worth optimising. I think the main contributor to size right now is likely the impala binary, which we could shrink by stripping debug symbols. We could also look at using a different base image from Ubuntu. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8425) Evaluate size of Impala docker containers
Tim Armstrong created IMPALA-8425: - Summary: Evaluate size of Impala docker containers Key: IMPALA-8425 URL: https://issues.apache.org/jira/browse/IMPALA-8425 Project: IMPALA Issue Type: Sub-task Components: Infrastructure Reporter: Tim Armstrong Assignee: Tim Armstrong We should take a look at the size of the containers produced by the build and see if it is worth optimising. I think the main contributor to size right now is likely the impala binary, which we could shrink by stripping debug symbols. We could also look at using a different base image from Ubuntu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8424) Impala Doc: Doc the column level permission on views
Alex Rodoni created IMPALA-8424: --- Summary: Impala Doc: Doc the column level permission on views Key: IMPALA-8424 URL: https://issues.apache.org/jira/browse/IMPALA-8424 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8424) Impala Doc: Doc the column level permission on views
Alex Rodoni created IMPALA-8424: --- Summary: Impala Doc: Doc the column level permission on views Key: IMPALA-8424 URL: https://issues.apache.org/jira/browse/IMPALA-8424 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8338) Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates
[ https://issues.apache.org/jira/browse/IMPALA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali resolved IMPALA-8338. -- Resolution: Fixed > Check CREATION_TIME of databases in event processor to avoid > incorrect/redundant invalidates > > > Key: IMPALA-8338 > URL: https://issues.apache.org/jira/browse/IMPALA-8338 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > HIVE-21077 adds CREATION_TIME to Database, and it will be available in the > metastore notifications. Hence, > # For CREATE_DATABASE events, we should compare the creationTime of the > Database in catalog with the Database in the event to make sure we are > ignoring the event only if catalog has the latest Database object. > # For DROP_DATABASE events, we should check that creation time of database > in the notification and the catalog match to make sure that we are removing > the correct database object. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8338) Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates
[ https://issues.apache.org/jira/browse/IMPALA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali resolved IMPALA-8338. -- Resolution: Fixed > Check CREATION_TIME of databases in event processor to avoid > incorrect/redundant invalidates > > > Key: IMPALA-8338 > URL: https://issues.apache.org/jira/browse/IMPALA-8338 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > HIVE-21077 adds CREATION_TIME to Database, and it will be available in the > metastore notifications. Hence, > # For CREATE_DATABASE events, we should compare the creationTime of the > Database in catalog with the Database in the event to make sure we are > ignoring the event only if catalog has the latest Database object. > # For DROP_DATABASE events, we should check that creation time of database > in the notification and the catalog match to make sure that we are removing > the correct database object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-8413) SystemStateInfoTest.ReadProcNetDev fails inside a Docker container
[
https://issues.apache.org/jira/browse/IMPALA-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819459#comment-16819459
]
ASF subversion and git services commented on IMPALA-8413:
-
Commit a223353c6d2daa6184566914c08aeab5eb3d3a08 in impala's branch
refs/heads/master from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a223353 ]
IMPALA-8413: Don't assume non-zero network counters in tests
In virtualized test environments (e.g. docker) we have seen cases where
some of the network counters are still zero when the
system-state-info-test runs. This change adjusts the test to allow
counters to be zero, since other tests make sure that non-zero values
are parsed correctly.
Change-Id: I39e18510b1c20749e218748eaaffb74abeae4ebe
Reviewed-on: http://gerrit.cloudera.org:8080/13017
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> SystemStateInfoTest.ReadProcNetDev fails inside a Docker container
> --
>
> Key: IMPALA-8413
> URL: https://issues.apache.org/jira/browse/IMPALA-8413
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Laszlo Gaal
>Priority: Major
>
> SystemStateInfoTest.ReadProcNetDev fails consistently when executed inside a
> Docker container using test-with-docker.py (this is the container version
> that hosts the complete minicluster inside the container).
> The test seems to assume that by the time it runs, the number of transmitted
> and received bytes and packets will both be nonzero (excluding the loopback
> interface). However, this assumption fails inside the Docker container used
> by test-with-docker.py.
> I started a Docker-based run with test-with-docker.py, running only BE_TEST.
> As the tests were running, I jumped into the container when about 75% of the
> BE tests were already executed, and captured the contents of /proc/net/dev :
> {code:java}
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED
> STATUS PORTS NAMES
> 0e967d480e8d efd25829a079 "/mnt/base/entrypoin…" 17 minutes
> ago Up 17 minutes i-20190414-111948-be-test
> $ docker exec -it i-20190414-111948-be-test /bin/bash
> impdev@i-20190414-111948:~/Impala$ cat /proc/net/dev
> Inter-| Receive | Transmit
> face |bytes packets errs drop fifo frame compressed multicast|bytes
> packets errs drop fifo colls carrier compressed
> lo: 232573986 353806 0 0 0 0 0 0 232573986
> 353806 0 0 0 0 0 0
> eth0: 828 10 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> {code}
> The layout is ugly, but it clearly shows that the number of transmitted
> anything for {{eth0}} is still zero.
> Additionally, it would be useful if the test logged the contents of
> /proc/net/dev for failure cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8338) Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates
[ https://issues.apache.org/jira/browse/IMPALA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819461#comment-16819461 ] ASF subversion and git services commented on IMPALA-8338: - Commit 55881609f1c21eb7a59c383de60171ffa8943db8 in impala's branch refs/heads/master from Bharath Krishna [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5588160 ] IMPALA-8338 : Check CREATION_TIME while processing DROP_DATABASE events. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Reviewed-on: http://gerrit.cloudera.org:8080/12938 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Check CREATION_TIME of databases in event processor to avoid > incorrect/redundant invalidates > > > Key: IMPALA-8338 > URL: https://issues.apache.org/jira/browse/IMPALA-8338 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > HIVE-21077 adds CREATION_TIME to Database, and it will be available in the > metastore notifications. Hence, > # For CREATE_DATABASE events, we should compare the creationTime of the > Database in catalog with the Database in the event to make sure we are > ignoring the event only if catalog has the latest Database object. > # For DROP_DATABASE events, we should check that creation time of database > in the notification and the catalog match to make sure that we are removing > the correct database object. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6718) Add support column-level permissions on views
[ https://issues.apache.org/jira/browse/IMPALA-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819460#comment-16819460 ] ASF subversion and git services commented on IMPALA-6718: - Commit 53798b0454332df44efce0032a88ef49e770a095 in impala's branch refs/heads/master from Fredy Wijaya [ https://gitbox.apache.org/repos/asf?p=impala.git;h=53798b0 ] IMPALA-6718: Add support for column-level permissions on views This patch adds support for column-level permissions on views. This behavior is compatible with Hive column-level permissons on views. The following statements are now supported. GRANT select(col) ON db.my_view TO ROLE my_role -- Sentry only REVOKE select(col) ON db.my_view FROM ROLE my_role -- Sentry only GRANT select(col) ON db.my_view TO USER my_user -- Ranger only REVOKE select(col) ON db.my_view FROM USER my_user -- Ranger only GRANT select(col) ON db.my_view TO GROUP my_group -- Ranger only REVOKE select(col) ON db.my_view FROM GROUP my_group -- Ranger only Testing: - Updated AuthorizationStmtTest to with new test cases - Ran all FE tests - Ran all E2E authorization tests Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 Reviewed-on: http://gerrit.cloudera.org:8080/12959 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add support column-level permissions on views > - > > Key: IMPALA-6718 > URL: https://issues.apache.org/jira/browse/IMPALA-6718 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Affects Versions: Impala 2.11.0 >Reporter: Adam Holley >Assignee: Fredy Wijaya >Priority: Critical > Labels: security, usability > Fix For: Impala 3.3.0 > > > Column-level permissions on views are not supported: > https://github.com/apache/impala/blob/da153104f26ef94f1b4db323f5629d678c55f9ee/fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java#L248-L251 > This means "GRANT SELECT(COLUMN) on " will throw an exception. Impala > needs to be updated to support column-level permissions on views. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8363) Disable column masking and row filtering
[ https://issues.apache.org/jira/browse/IMPALA-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819462#comment-16819462 ] ASF subversion and git services commented on IMPALA-8363: - Commit 986dbbc5145a4cfd796453d94e6c74eb786ea0d7 in impala's branch refs/heads/master from Fredy Wijaya [ https://gitbox.apache.org/repos/asf?p=impala.git;h=986dbbc ] IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger This patch updates the Ranger authorization checker code to deny access when column masking and row filtering is enabled in Ranger for queries that that have columns/tables specified in column mask and row filter policies. This is to prevent data leak, such that the data that is masked/filtered in Hive should not be visible at all in Impala until Impala has full support for column masking and row filtering. Testing: - Added tests in AuthorizationStmtTest to test queries with column masking and row filtering enabled. - Ran all FE tests - Ran all E2E tests Change-Id: If46b4bf24d916e4a4ea8a36ff4acfd95d5f45c8e Reviewed-on: http://gerrit.cloudera.org:8080/12927 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins > Disable column masking and row filtering > > > Key: IMPALA-8363 > URL: https://issues.apache.org/jira/browse/IMPALA-8363 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > Because of policy sharing between Hive and Ranger, Impala needs throw an > authorization exception when column masking and row filtering is enabled in > Hive. This is only temporary until Impala has proper support for column > masking and row filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8363) Disable column masking and row filtering
[ https://issues.apache.org/jira/browse/IMPALA-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-8363. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Disable column masking and row filtering > > > Key: IMPALA-8363 > URL: https://issues.apache.org/jira/browse/IMPALA-8363 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > Because of policy sharing between Hive and Ranger, Impala needs throw an > authorization exception when column masking and row filtering is enabled in > Hive. This is only temporary until Impala has proper support for column > masking and row filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8363) Disable column masking and row filtering
[ https://issues.apache.org/jira/browse/IMPALA-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-8363. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Disable column masking and row filtering > > > Key: IMPALA-8363 > URL: https://issues.apache.org/jira/browse/IMPALA-8363 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > Because of policy sharing between Hive and Ranger, Impala needs throw an > authorization exception when column masking and row filtering is enabled in > Hive. This is only temporary until Impala has proper support for column > masking and row filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8421) get problem when start catalogd with the
[ https://issues.apache.org/jira/browse/IMPALA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8421. --- Resolution: Invalid If you have questions about using or developing impala, please send a question to one of the mailing lists: https://impala.apache.org/community.html > get problem when start catalogd with the > > > Key: IMPALA-8421 > URL: https://issues.apache.org/jira/browse/IMPALA-8421 > Project: IMPALA > Issue Type: Bug >Reporter: Lycan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8423) Add rule to remove useless SELECT node
Quanlong Huang created IMPALA-8423:
--
Summary: Add rule to remove useless SELECT node
Key: IMPALA-8423
URL: https://issues.apache.org/jira/browse/IMPALA-8423
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
We can add some rules to optimize the plan after we chose a cheapest plan based
on cost. For example, one useful rule can be "removing useless SELECT nodes".
Impala will generated a useless SELECT for the following query:
{code:sql}
SELECT t.id, t.int_col
FROM functional.alltypestiny t
LEFT JOIN
(SELECT id, int_col
FROM functional.alltypestiny) t2
ON (t.id = t2.id)
WHERE t.int_col = t.id
UNION ALL
VALUES (NULL, NULL){code}
Its single node plan is
{code:java}
PLAN-ROOT SINK
|
00:UNION
| constant-operands=1
| row-size=8B cardinality=1
|
04:SELECT
| predicates: t.id = t.int_col
| row-size=12B cardinality=0
|
03:HASH JOIN [RIGHT OUTER JOIN]
| hash predicates: id = t.id
| runtime filters: RF000 <- t.id
| row-size=12B cardinality=1
|
|--01:SCAN HDFS [functional.alltypestiny t]
| HDFS partitions=4/4 files=4 size=460B
| predicates: t.int_col = t.id
| row-size=8B cardinality=1
|
02:SCAN HDFS [functional.alltypestiny]
HDFS partitions=4/4 files=4 size=460B
runtime filters: RF000 -> id
row-size=4B cardinality=8{code}
The SELECT node (id=04) is useless since its only predicate "t.id = t.int_col"
has been enforced in the SCAN node (id=01) which is the right hand side of the
RIGHT OUTER JOIN. The SELECT node won't filter out any more rows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8422) get problem when start catalogd with the tarballs builded by myself
Lycan created IMPALA-8422: - Summary: get problem when start catalogd with the tarballs builded by myself Key: IMPALA-8422 URL: https://issues.apache.org/jira/browse/IMPALA-8422 Project: IMPALA Issue Type: Bug Environment: centos6.5 Reporter: Lycan I compiled impala under Ubuntu16.04，and run catalogd under centos6.5.Then I get the following problem: E0416 19:17:35.646098 153074 logging.cc:147] stderr will be logged to this file. java.lang.ClassFormatError: org.apache.impala.common.JniUtil (unrecognized class file version) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java.security.SecureClassLoader.defineClass(libgcj.so.10) at java.net.URLClassLoader.findClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) F0416 19:17:36.217576 153074 init.cc:313] Failed to find JniUtil class. . Impalad exiting. *** Check failure stack trace: *** @ 0x7fd12ac07dac @ 0x7fd12ac09651 @ 0x7fd12ac07786 @ 0x7fd12ac0ad4d @ 0x7fd12d8efef3 @ 0x7fd12f634d29 @ 0x697906 @ 0x7fd129c0a92f @ 0x6d6038 loadFileSystems error: ClassFormatError: org.apache.hadoop.fs.FileSystem (unrecognized class file version)java.lang.ClassFormatError: org.apache.hadoop.fs.FileSystem (unrecognized class file ver sion) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java.security.SecureClassLoader.defineClass(libgcj.so.10) at java.net.URLClassLoader.findClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) this is my java version: [root@impala3.2]# java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8421) get problem when start catalogd with the
Lycan created IMPALA-8421: - Summary: get problem when start catalogd with the Key: IMPALA-8421 URL: https://issues.apache.org/jira/browse/IMPALA-8421 Project: IMPALA Issue Type: Bug Reporter: Lycan -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8421) get problem when start catalogd with the
Lycan created IMPALA-8421: - Summary: get problem when start catalogd with the Key: IMPALA-8421 URL: https://issues.apache.org/jira/browse/IMPALA-8421 Project: IMPALA Issue Type: Bug Reporter: Lycan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (IMPALA-8404) failed when i build Impala in centos7
[
https://issues.apache.org/jira/browse/IMPALA-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lycan closed IMPALA-8404.
-
Resolution: Fixed
> failed when i build Impala in centos7
> --
>
> Key: IMPALA-8404
> URL: https://issues.apache.org/jira/browse/IMPALA-8404
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
> Environment: centos7 Impala3.2
>Reporter: Lycan
>Priority: Minor
>
> when I builded impala under centos7 through the
> documents：https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala，I
> got the flowing problem：
> {code:java}
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/row_batch.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_row_batch.proto.dir/all] Error
> 2
> make[2]: *** Waiting for unfinished jobs
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/common.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_common.proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/histogram.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/histogram_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> [ 16%] Built target thrift-deps
> make[3]: *** [be/src/kudu/util/pb_util.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/pb_util_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/maintenance_manager.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/maintenance_manager_proto.dir/all]
> Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/version_info.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/version_info_proto.dir/all] Error 2
> make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> make: *** [impalad] Error 2
> {code}
> And then I check the CMakeErrorlog, I got the flowwing problem:
> {code:java}
> Run Build Command:"/usr/bin/gmake" "cmTC_2a4fc/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_2a4fc.dir/build.make
> CMakeFiles/cmTC_2a4fc.dir/build
> gmake[1]: Entering directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -o
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -c
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c
> Linking C executable cmTC_2a4fc
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/cmake-3.8.2-p1/bin/cmake -E
> cmake_link_script CMakeFiles/cmTC_2a4fc.dir/link.txt --verbose=1
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -rdynamic
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -o cmTC_2a4fc
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o: In function `main':
> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_2a4fc] Error 1
> gmake[1]: Leaving directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_2a4fc/fast] Error 2
> File
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
> /* */
> #include
> int main(int argc, char** argv)
> {
> (void)argv;
> #ifndef pthread_create
> return ((int*)(_create))[argc];
> #else
> (void)argc;
> return 0;
> #endif
> }
> {code}
> It seems the pthread lib is missed because uninstalled or unwirded.But,when I
> check the /usr/lib/, the pthread.so/pthread.a is available.
> {code:java}
> -rwxr-xr-x 1 root root 129956 Nov 20 2015 libpthread-2.17.so
> -rw-r--r-- 1 root root 153162 Apr 10 14:28 libpthread.a
> lrwxrwxrwx 1 root root 24 Apr 10 11:22 libpthread.so ->
> /usr/lib64/libpthread.so
> lrwxrwxrwx 1 root root 18 Apr 23 2018 libpthread.so.0 ->
> libpthread-2.17.so{code}
> This problem is really puzzled.Any sloution is appreciating
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8404) failed when i build Impala in centos7
[
https://issues.apache.org/jira/browse/IMPALA-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lycan closed IMPALA-8404.
-
Resolution: Fixed
> failed when i build Impala in centos7
> --
>
> Key: IMPALA-8404
> URL: https://issues.apache.org/jira/browse/IMPALA-8404
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
> Environment: centos7 Impala3.2
>Reporter: Lycan
>Priority: Minor
>
> when I builded impala under centos7 through the
> documents：https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala，I
> got the flowing problem：
> {code:java}
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/row_batch.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_row_batch.proto.dir/all] Error
> 2
> make[2]: *** Waiting for unfinished jobs
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/common.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_common.proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/histogram.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/histogram_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> [ 16%] Built target thrift-deps
> make[3]: *** [be/src/kudu/util/pb_util.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/pb_util_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/maintenance_manager.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/maintenance_manager_proto.dir/all]
> Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/version_info.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/version_info_proto.dir/all] Error 2
> make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> make: *** [impalad] Error 2
> {code}
> And then I check the CMakeErrorlog, I got the flowwing problem:
> {code:java}
> Run Build Command:"/usr/bin/gmake" "cmTC_2a4fc/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_2a4fc.dir/build.make
> CMakeFiles/cmTC_2a4fc.dir/build
> gmake[1]: Entering directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -o
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -c
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c
> Linking C executable cmTC_2a4fc
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/cmake-3.8.2-p1/bin/cmake -E
> cmake_link_script CMakeFiles/cmTC_2a4fc.dir/link.txt --verbose=1
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -rdynamic
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -o cmTC_2a4fc
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o: In function `main':
> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_2a4fc] Error 1
> gmake[1]: Leaving directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_2a4fc/fast] Error 2
> File
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
> /* */
> #include
> int main(int argc, char** argv)
> {
> (void)argv;
> #ifndef pthread_create
> return ((int*)(_create))[argc];
> #else
> (void)argc;
> return 0;
> #endif
> }
> {code}
> It seems the pthread lib is missed because uninstalled or unwirded.But,when I
> check the /usr/lib/, the pthread.so/pthread.a is available.
> {code:java}
> -rwxr-xr-x 1 root root 129956 Nov 20 2015 libpthread-2.17.so
> -rw-r--r-- 1 root root 153162 Apr 10 14:28 libpthread.a
> lrwxrwxrwx 1 root root 24 Apr 10 11:22 libpthread.so ->
> /usr/lib64/libpthread.so
> lrwxrwxrwx 1 root root 18 Apr 23 2018 libpthread.so.0 ->
> libpthread-2.17.so{code}
> This problem is really puzzled.Any sloution is appreciating
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
