Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- (Updated May 27, 2020, 5:02 a.m.) Review request for hive and Sankar Hariappan. Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs (updated) - itests/src/test/resources/testconfiguration.properties 92ae8c28e8 ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java 7821f40a82 ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 92d10cd0e1 Diff: https://reviews.apache.org/r/72462/diff/6/ Changes: https://reviews.apache.org/r/72462/diff/5-6/ Testing --- Thanks, Adesh Rao
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/#review220868 --- ql/src/test/results/clientnegative/msck_repair_6.q.out Lines 30 (patched) <https://reviews.apache.org/r/72462/#comment309587> Most of the other ddl commands throw the same error code when finished. Do we need to create a new MetastoreException subclass, and then use it to catch and throw new error code? - Adesh Rao On May 18, 2020, 10:43 a.m., Adesh Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72462/ > --- > > (Updated May 18, 2020, 10:43 a.m.) > > > Review request for hive and Sankar Hariappan. > > > Repository: hive-git > > > Description > --- > > The fix converts partition keys to lowercase present in hdfs directory, but > store the hdfs directory as is for partition path. > > > Diffs > - > > > ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java > 7821f40a82 > ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION > ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION > ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION > ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION > ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION > ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION > ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java > 5287f47e21 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java > 6f4400a8ef > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java > f4e109d1b0 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > 92d10cd0e1 > > > Diff: https://reviews.apache.org/r/72462/diff/5/ > > > Testing > --- > > > Thanks, > > Adesh Rao > >
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- (Updated May 18, 2020, 10:43 a.m.) Review request for hive and Sankar Hariappan. Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs (updated) - ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java 7821f40a82 ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 92d10cd0e1 Diff: https://reviews.apache.org/r/72462/diff/5/ Changes: https://reviews.apache.org/r/72462/diff/4-5/ Testing --- Thanks, Adesh Rao
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- (Updated May 18, 2020, 8:40 a.m.) Review request for hive and Sankar Hariappan. Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs (updated) - ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java 7821f40a82 ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 92d10cd0e1 Diff: https://reviews.apache.org/r/72462/diff/4/ Changes: https://reviews.apache.org/r/72462/diff/3-4/ Testing --- Thanks, Adesh Rao
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- (Updated May 15, 2020, 12:28 p.m.) Review request for hive and Sankar Hariappan. Changes --- Validating that each partition should correspond to single path in fs Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs (updated) - ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 92d10cd0e1 Diff: https://reviews.apache.org/r/72462/diff/3/ Changes: https://reviews.apache.org/r/72462/diff/2-3/ Testing --- Thanks, Adesh Rao
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
> On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote: > > ql/src/test/queries/clientpositive/msck_repair_4.q > > Lines 8 (patched) > > <https://reviews.apache.org/r/72462/diff/1/?file=2229577#file2229577line8> > > > > Add a testcase with table path repairtable_n4 having upper case. You > > can achieve it by setting location in create table. Done. > On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote: > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java > > Lines 116 (patched) > > <https://reviews.apache.org/r/72462/diff/1/?file=2229579#file2229579line116> > > > > Why do we need pathSet? Can we check if path == null instead? Removed. > On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > > Line 1441 (original), 1441 (patched) > > <https://reviews.apache.org/r/72462/diff/1/?file=2229582#file2229582line1441> > > > > Even table name in directory can be any case. Should we use > > equalsIgnoresCase here? Msck list all the directories under the actual table path, and then checks if the new child directories are valid partitions. Since all the partition directories are child of the actual table path, it won't matter if we are using equals/equalsIgnoreCase. The second question which comes is, should we check for partitions under all paths that matches the actual table path ignoring the case. I don't think we should do that, because, then msck will have to check in too many combinations of base directory in hdfs (which will be ~2^length of table name). > On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > > Line 1454 (original), 1456 (patched) > > <https://reviews.apache.org/r/72462/diff/1/?file=2229582#file2229582line1456> > > > > Need to check how Hive treats ptn='A' and ptn='a' as ptn keys are > > lowercase but values can be any case. Added a test case. These two values should be treated as different as they both are different strings. - Adesh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/#review220771 --- On May 15, 2020, 5:42 a.m., Adesh Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72462/ > --- > > (Updated May 15, 2020, 5:42 a.m.) > > > Review request for hive and Sankar Hariappan. > > > Repository: hive-git > > > Description > --- > > The fix converts partition keys to lowercase present in hdfs directory, but > store the hdfs directory as is for partition path. > > > Diffs > - > > ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION > ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION > ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java > 5287f47e21 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java > 6f4400a8ef > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java > f4e109d1b0 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > 92d10cd0e1 > > > Diff: https://reviews.apache.org/r/72462/diff/2/ > > > Testing > --- > > > Thanks, > > Adesh Rao > >
Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- (Updated May 15, 2020, 5:42 a.m.) Review request for hive and Sankar Hariappan. Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs (updated) - ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 92d10cd0e1 Diff: https://reviews.apache.org/r/72462/diff/2/ Changes: https://reviews.apache.org/r/72462/diff/1-2/ Testing --- Thanks, Adesh Rao
Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72462/ --- Review request for hive and Sankar Hariappan. Repository: hive-git Description --- The fix converts partition keys to lowercase present in hdfs directory, but store the hdfs directory as is for partition path. Diffs - ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java 5287f47e21 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java 6f4400a8ef standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java f4e109d1b0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java 7c4e129738 Diff: https://reviews.apache.org/r/72462/diff/1/ Testing --- Thanks, Adesh Rao
Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/ --- (Updated April 26, 2020, 3:32 a.m.) Review request for hive and Sankar Hariappan. Repository: hive-git Description --- Since limit constraint was ignored while creating splits. If multiple llap daemons execute splits, output contains more rows than specified by limit constraint. Diffs (updated) - itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java 8cbca69737 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java defbe78802 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java 330174513c itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java bc2480a422 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapRow.java d954d0e2fe itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapVectorArrow.java 35eda6cb0a itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapVectorArrowBatch.java bba599b883 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNewGetSplitsFormat.java ab1798d571 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNewGetSplitsFormatReturnPath.java a437998490 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 00a6c89b1e Diff: https://reviews.apache.org/r/72379/diff/4/ Changes: https://reviews.apache.org/r/72379/diff/3-4/ Testing --- Waiting for Hive QA run. Thanks, Adesh Rao
Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits
> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java > > Lines 181 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223577#file2223577line181> > > > > Add test for limit with predicates, groupby queries as well. Also, make > > sure, we have partitioned table too. > > Adesh Rao wrote: > Added queries with predicates and groupby. Since the existing test suite > does not have partitioned table, need to go through other tests to figure out > the standard way to generate a partitioned table. done. - Adesh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/#review220363 ------- On April 21, 2020, 11:17 a.m., Adesh Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72379/ > --- > > (Updated April 21, 2020, 11:17 a.m.) > > > Review request for hive and Sankar Hariappan. > > > Repository: hive-git > > > Description > --- > > Since limit constraint was ignored while creating splits. If multiple llap > daemons execute splits, output contains more rows than specified by limit > constraint. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java > 8cbca69737 > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java > defbe78802 > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java > 330174513c > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > 00a6c89b1e > > > Diff: https://reviews.apache.org/r/72379/diff/3/ > > > Testing > --- > > Waiting for Hive QA run. > > > Thanks, > > Adesh Rao > >
Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/ --- (Updated April 21, 2020, 11:17 a.m.) Review request for hive and Sankar Hariappan. Changes --- Added test for partitioned table Repository: hive-git Description --- Since limit constraint was ignored while creating splits. If multiple llap daemons execute splits, output contains more rows than specified by limit constraint. Diffs (updated) - itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java 8cbca69737 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java defbe78802 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java 330174513c ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 00a6c89b1e Diff: https://reviews.apache.org/r/72379/diff/3/ Changes: https://reviews.apache.org/r/72379/diff/2-3/ Testing --- Waiting for Hive QA run. Thanks, Adesh Rao
Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits
> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java > > Lines 181 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223577#file2223577line181> > > > > Add test for limit with predicates, groupby queries as well. Also, make > > sure, we have partitioned table too. Added queries with predicates and groupby. Since the existing test suite does not have partitioned table, need to go through other tests to figure out the standard way to generate a partitioned table. > On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > > Line 240 (original), 241 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223578#file2223578line241> > > > > Do we need this computation? Can't we just use forceSingleSplit instead > > of generateSingleSplit? Yes, we don't need this for limit query. Removed. > On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > > Line 259 (original), 260 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223578#file2223578line260> > > > > Error message to be updated for limit query too. Not required, as not generating single split for limit queries. > On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > > Lines 344 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223578#file2223578line344> > > > > This issue should be there even for "order by" queries. > > How it work there? By table, I meant the original table. The materialized table will have a single file only. > On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > > Lines 347 (patched) > > <https://reviews.apache.org/r/72379/diff/1/?file=2223578#file2223578line347> > > > > Since, we always materialize the results for limit query, I think, > > forceSingleSplit logic is not needed. Even, if we generate multiple splits > > on this temp table, the eventual results will have only expected number of > > rows. Ack. modified the approach to not generate single split. - Adesh ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/#review220363 --- On April 20, 2020, 6:10 p.m., Adesh Rao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72379/ > --- > > (Updated April 20, 2020, 6:10 p.m.) > > > Review request for hive and Sankar Hariappan. > > > Repository: hive-git > > > Description > --- > > Since limit constraint was ignored while creating splits. If multiple llap > daemons execute splits, output contains more rows than specified by limit > constraint. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java > 8cbca69737 > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java > defbe78802 > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java > 330174513c > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > 00a6c89b1e > > > Diff: https://reviews.apache.org/r/72379/diff/2/ > > > Testing > --- > > Waiting for Hive QA run. > > > Thanks, > > Adesh Rao > >
Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/ --- (Updated April 20, 2020, 6:10 p.m.) Review request for hive and Sankar Hariappan. Changes --- Not forcing single split generation for limit queries Summary (updated) - "get_splits" udf ignores limit constraint when creating splits Repository: hive-git Description (updated) --- Since limit constraint was ignored while creating splits. If multiple llap daemons execute splits, output contains more rows than specified by limit constraint. Diffs (updated) - itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java 8cbca69737 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java defbe78802 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java 330174513c ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 00a6c89b1e Diff: https://reviews.apache.org/r/72379/diff/2/ Changes: https://reviews.apache.org/r/72379/diff/1-2/ Testing --- Waiting for Hive QA run. Thanks, Adesh Rao
Review Request 72379: HIVE-23230: HiveWarehouseConnector executeQuery api for query having "limit" clause returns more rows
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72379/ --- Review request for hive and Sankar Hariappan. Repository: hive-git Description --- This issue occurs when there are multiple llap daemons running Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511 itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java 8cbca69737 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 00a6c89b1e Diff: https://reviews.apache.org/r/72379/diff/1/ Testing --- Waiting for Hive QA run. Thanks, Adesh Rao
Review Request 59139: Add done check for reducer operator to avoid reading unnecessary data in case of Tez
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59139/ --- Review request for hive. Repository: hive-git Description --- In case of tez, there is no check for reducer operators whether it is done or not. Hence, reducer ends up reading complete data even if the operators are done. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java b1d2f523be Diff: https://reviews.apache.org/r/59139/diff/1/ Testing --- Thanks, Adesh Rao
