This is an automated email from the ASF dual-hosted git repository.
yumwang pushed a change to branch slow
in repository https://gitbox.apache.org/repos/asf/spark.git
was 6650667c407 fix
This change permanently discards the following revisions:
discard 6650667c407 fix
discard e0df57fdbbd Use ExpressionSet and add UT
discard b042b755f93 [CARMEL-6796] Pull out complex aggregate expressions
discard 29bd591cd2f [carmel-6785] Add configuration
spark.sql.materializedView.name.prefix to support definition of available mv
(#1307)
discard b3cce6eb3a0 [MINOR] Bug fix for HiveUDF codegen (#1303)
discard 905523e4cd2 [CARMEL-6734] Fix reorder filter condition issue (#1293)
discard b3036d65479 [CARMEL-6741] Tag Queries with Join Expansion in Runtime
(#1299)
discard a6d9e7c163d [CARMEL-5941] Add how to request HDM data access to error
message (#1301)
discard 285aff3d2a4 [CARMEL-6739][SPARK-43050][SQL] Fix construct aggregate
expressions by replacing grouping functions (#1300)
discard d803e943edb [CARMEL-6751] More reasonable error message when heavily
skewed partition (#1297)
discard 2e9d18bd5dd [CARMEL-6760] Correct metadata ‘mv_updated_time’ which is
used for data validation (#1298)
discard 2973a317042 [CARMEL-6633] Reduce Skew Join Split Size Considering
Expand Node (#1295)
discard 46be35accc9 [CARMEL-6664] Fix the InterruptedException error and
incorrect state for cancelled download (#1291)
discard ee430edaf39 [CARMEL-6703] Take too much time to build bloom filter
(#1292)
discard f76533f3acc [Carmel 6640] Support create materialized view as
datasource table (#1280)
discard 7efe9b78cff [CARMEL-6647] Enhance DemoteBucketJoin to support Alias
Aware Output Partitioning (#1290)
discard e7d9e88ddcb [CARMEL-6583][SPARK-42500][SQL] ConstantPropagation
support more cases (#1287)
discard d8fa6e4e368 [CARMEL-6621][SPARK-42789][SQL] Rewrite multiple
GetJsonObjects to a JsonTuple if their json expressions are the same (#1286)
discard 5b0fffed633 [CARMEL-6705] Bug fix for query output row count (#1289)
discard 96c7333cb22 [CARMEL-6615] Backport [SPARK-42052][SQL] Codegen Support
for HiveSimpleUDF (#1288)
discard a53b95c0c4d [CARMEL-6675] Support to enable decommission nodes when
hive service discovery is disabled (#1282)
discard c08771ff897 [CARMEL-6683][SPARK-31008][SQL] Support json_array_length
function (#1284)
discard b243ff8b59e [CARMEL-6674] Project fail to be collapsed (#1283)
discard 93f89f941dc [CARMEL-6587] Support Generic Skew Join Patterns (#1281)
discard 81d76f6b116 [CARMEL-6652][SPARK-40501][SQL] Add
PushProjectionThroughLimit for Optimizer (#1278)
discard 3e8e629a2d0 [CARMEL-6655] Do not trim whitespaces by default when
downloading data as CSV file (#1279)
discard b3e8faafbda [CARMEL-6651] Remove repartition if it is the child of
LocalLimit (#1277)
discard 2ddd418edf4 [CARMEL-6632] Fix the running time of download statement
in query log (#1275)
discard 9e899c166ce [CARMEL-6586] Ignore SinglePartition when determining
expectedChildrenNumPartitions (#1252)
discard 25a3b904b61 [CARMEL-6327] Support Broadcast Join with Stream Side Skew
(#1272)
discard c3f17a52289 [CARMEL-6608] Increase bucket table scan partitions (#1269)
discard 434b16e0cc8 [CARMEL-6439] Define new query execution event and log
column lineage asynchronously (#1268)
discard e77e54fd7da [CARMEL-6609] Casts types according to bucket info support
view (#1270)
discard 872399e7a5e [CARMEL-6593] Upgrade parquet to 1.12.3.0.1.0 (#1271)
discard 40900d24073 [CARMEL-6604] Stop posting duplicate execution event
(#1267)
discard e2036ae1eda [CARMEL-6591][SPARK-42597] Support unwrap date type to
timestamp type (#1263)
discard f7c77ff2bbd [CARMEL-6582][SPARK-42513][SQL] Push down topK through
join (#1257)
discard a214b2b67fd [CARMEL-6541] Support Query Level SQL Conf leveraging Hint
(#1256)
discard 10a27944ccf [CARMEL-6568] Analyze join operator and support data
expansion check (#1265)
discard 56c9bad9395 [CARMEL-6511] Disable rename temp table (#1254)
discard 71489e91204 [CARMEL-6371] Partial aggregation push through left/right
outer join (#1233)
discard 7905f88b1f3 [CARMEL-6581] TakeOrderedAndProject should not replace
project if project expression is not deterministic (#1249)
discard 142b10b1580 [CARMEL-6339] Implementation of materialized view (#1244)
discard 1cd2cbcaa2f [CARMEL-6556] Avoid coalesce partitions from different
UNION sides (#1246)
discard 413224b3270 [CARMEL-6495] Do not quit am even when all nodes are in
blacklist (#1248)
discard 9f306b4ddf4 [CARMEL-6439] Add configuration to enable log column
lineage (#1238)
discard 5f91f872777 [CARMEL-6553] Backport [SPARK-35673][SQL] Fix user-defined
hint and unrecognized hint in subquery (#1236)
discard d8b439948dd [CARMEL-6525][MINOR] Support tag different drivers in the
queue (#1237)
discard c4930985ae1 [CARMEL-6537] [Followup] Support Iceberg with maven
dependency in Carmel - use correct jar (#1234)
discard adbb50db673 [CARMEL-6498] handle multi-identifier in
HiveAuthorizer/PrivilegesBuilder for V2 command (#1219)
discard 62da19c8ae7 [CARMEL-6531] Fail to instantiate the custom v2 session
catalog: DelegatingCatalog (#1221)
discard 91bc3d5d8ea [CARMEL-6537] Support Iceberg with maven dependency in
Carmel (#1225)
discard 2101f0f784b [CARMEL-6525] Support tag different drivers in the queue
(#1229)
discard 6fa10a9d035 [CARMEL-6542] Change ADVISORY_PARTITION_SIZE_IN_BYTES to
SKEW_JOIN_SKEWED_PARTITION_THRESHOLD in HandleOuterJoinBuildSideSkew (#1224)
discard cc254f23bcf [CARMEL-6383] Allow introduce new shuffle in skew handling
(#1161)
discard e49b25b544b [CARMEL-6532] Fix NoSuchElementException in
HandleOuterJoinBuildSideSkew (#1220)
discard 7709f60dbf0 [CARMEL-6491] Collapse percentile functions if possible
(#1216)
discard 11334447e4e [CARMEL-6497] Left outer join and left has lots of nulls
(#1202)
discard deb3e8254da [CARMEL-6518] Record temp view usage in workload analysis
(#1208)
discard 8c7e20f9d0a [CARMEL-6513] Bug fix for reorder predicate (#1207)
discard 21b3c47488b [CARMEL-6511] Add log if rename temporary table (#1215)
discard 3970b4641be [SPARK-41805][SQL] Reuse expressions in
WindowSpecDefinition (#1212)
discard 2d2f2b094ef
[CARMEL-6489][SPARK-33205][SPARK-33725][SPARK=35277][BUILD] Bump snappy-java
version to 1.1.8.4 (#1198)
discard 2c84685793d [CARMEL-6496] Implement bloom filter join hint #1201
discard cb107711df0 [CARMEL-6490] Lots of unfinished state sql in the query
log (#1199)
discard 7d73036b0e5 [CARMEL-6484] Infer IsNotNull constraints for
GetJsonObject (#1196)
discard 24ddd14b143 [CARMEL-6482] HiveThriftServer2Listener memory leak
because of event lost (#1194)
discard 5bdfd6e1f16 [CARMEL-6483] Can't zip RDDs with unequal numbers of
partitions in skew join handling (#1195)
discard 1329dcac486 CARMEL-6449: Enhance runtime filter (#1180)
discard e03c9211bc6 CARMEL-6447: Do not push down sum if it is decimal type
(#1188)
discard 06e0907b6de [CARMEL-6461] Too much join expansion cause many zombie
tasks (#1183)
discard 33506f3581f [CARMEL-6471] Upload api sometimes throw Delegation Token
error (#1191)
discard 48d4f4381d5 [CARMEL-6388] Backport [SPARK-33092][SQL] Support
subexpression elimination in ProjectExec (#1166)
discard 5d686257756 [CARMEL-6200][FollowUp4] Improve table lineage extraction
(#1181)
discard 7d90243c56e HADP-47632. Update ZSTD Version to 1.4.8-1 (#1175)
discard b178cb79931 [CARMEL-6426] Adjust scan parallelism dynamically
improvement (#1178)
discard 6c15686f9a7 [CARMEL-6400][FollowUp] Log detail sql execution metrics -
improvement (#1182)
discard 8b865eb753a [CARMEL-6076][FollowUp2] Disable read lock for empty paths
in Apollo when table location parameter is empty (#1170)
discard 405009e9688 [CARMEL-6444] Get correct query objects from AQE plan
(#1176)
discard ce6d038760d [CARMEL-6200][FollowUp3] Improve table lineage extraction
(#1169)
discard c35e50e3157 [CARMEL-6434] Expose active sql related metrics from
driver (#1171)
discard a99b6e326fc [CARMEL-6400] Log detail sql execution metrics (#1162)
discard e3898e836a9 [CARMEL-6392] Bug fix for hive operation event (#1151)
discard d6b2dacab51 [CARMEL-6396] Reduce the instances of OPTIMIZE TAG (#1165)
discard 8b119607e4b [CARMEL-6427] TaskResultGetter eventloop exit unexpectedly
(#1167)
discard bdbb6dcf7b7 [CARMEL-6200][Follow-Up2] Improve table lineage extraction
(#1164)
discard 817c553f6cd [CARMEL-6394] Implement column lineage tools (#1159)
discard da6af85bb05 [CARMEL-6401] Bug fix for SortMergeJoin outer join codegen
(#1158)
discard aae616d7ce2 [CARMEL-6398] Workaround: `UnresolvedException: Invalid
call to dataType on unresolved object` (#1154)
discard 33803654937 [CARMEL-6367][FOLLOW-UP] Insert bloom filter if it is skew
bucket join (#1157)
discard f91eb96f082 [CARMEL-6399] Download hive-1.2.1 jar files to local to
improve load properties performance (#1160)
discard 6b533c191eb [CARMEL-6367] Insert bloom filter if it is skew bucket
join (#1152)
discard 09e83b7b6a4 [CARMEL-6387] Backport [SPARK-35447][SQL] Optimize skew
join before coalescing shuffle partitions (#1153)
discard 4d6e245a1fa [CARMEL-6381] Remove unnecessary sql metrics for UnionExec
(#1146)
discard c25fdd6d460 [CARMEL-6375] Support Potential Skewed Operator Tagging
(#1147)
discard 3f2a8a9ae70 [CARMEL-6380] Merge task metrics in task result getter
threads instead of DAGScheduler (#1145)
discard e210385dd48 [CARMEL-6365] Don't store data in quota free folder when
create temp table (#1148)
discard 36176069f6a [CARMEL-6355][FOLLOWUP] Auto update table statistics based
on write metrics (#1142)
discard d1c2c3a0bf8 Revert "[CARMEL-6365] Don't store data in quota free
folder when create temp … (#1126)" (#1143)
discard 0bc12a616ba [CARMEL-6354] Increase parallelism for get_json_object
(#1139)
discard 3eddc31eaeb [CARMEL-6372] Adjust skewed join split size (#1141)
discard 177c4b0b506 [CARMEL-6355] Auto update table statistics based on write
metrics (#1123)
discard c6c737d2f2c [CARMEL-6365] Don't store data in quota free folder when
create temp … (#1126)
discard 813d18323ad [CARMEL-6345] Support backup table command (#1124)
discard 9a4f8c3830d [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join
condition (#1099)
discard 41e552438c8 [CARMEL-6324][MINOR 2] Support bucket skew detection
(#1140)
discard 8a942231b35 [CARMEL-6358] Change to file commit algorithm V2 in CTAS
command for Hive table (#1125)
discard 9c158d84810 [CARMEL-6324][MINOR] Support bucket skew detection (#1134)
discard faae0248513 [CARMEL-6360] Backport date_from_unix_date, unix_date,
timestamp_seconds, timestamp_millis, timestamp_micros, unix_seconds,
unix_millis, unix_micros functions (#1121)
discard 19e4c637e8d [CARMEL-6352] Adjust scan partition size dynamically
considering potential cost (#1110)
discard d0a000ee91c [CARMEL-6324] Support bucket skew detection (#1130)
discard 9f3b826db3d [CARMEL-6254][FOLLOW-UP] Collapse the project after
pulling out complex join condition (#1115)
discard 6a3834d0552 [CARMEL-6243][FOLLOW-UP] The right side max size should
larger than `spark.sql.adaptive.advisoryPartitionSizeInBytes` (#1120)
discard ce67e3f8b3b [CARMEL-6361] Too many casewhen expression cause driver
full gc (#1122)
discard 70b48fae0a5 [CARMEL-6300][FOLLOW-UP] Casts types according to bucket
info only accept decimal type (#1111)
discard cbcd848144e [CARMEL-6300] Casts types according to bucket info for
Equality expressions (#1107)
discard 708a1d887da [CARMEL-6306] backport [SPARK-33399][SPARK-31078]
Normalize output partitioning and sortorder with respect to aliases to avoid
unneeded exchange/sort nodes (#1092)
discard 5ba77832724 [CARMEL-6330] Support Range Partition Skew Handling (#1109)
discard fe3ece3359d [HADP-46618][HADP-46404] add DelegatingCatalog to delegate
to various… (#1076)
discard 479e64ed9ad [CARMEL-6314] Make sure registerExecutor event must be
handled in HeartBeatReceiver (#1097)
discard a726a477f40 [CARMEL-6243][FOLLOW-UP] Insert partial aggregate for
right side of left anti join (#1100)
discard e42088d73e0 [CARMEL-6305] Support to show partitions for Delta table
(#1094)
discard 14b521dd8dd [CARMEL-6315] Add physicalPlanDescription column for
adaptive query execution event (#1098)
discard ed9aa9491bd [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join
condition (#1078)
discard 42640cd9902 [CARMEL-6254] Pull out complex join condition (#1091)
discard af0d2d09875 [CARMEL-6231] Cleanup shuffle merge data after query
finished (#1075)
discard b78322f5329 [CARMEL-6299] Expose stage/task retry count to Carmel
Overview (#1090)
discard 1be4a01343f [CARMEL-6200][Follow-Up2] Fix task not serializable
exception for table lineage (#1086)
discard 9147a5cbd0a [CARMEL-6292] Fix spilled query result leak (#1089)
discard ee779238d9e [CARMEL-6185] Expose row count for RepeatableIterator
(#1074)
discard e0ff5303e89 [CARMEL-6280] Data download query stuck in job commit
phase (#1084)
discard aa62055bef3 [CARMEL-6243] Handle outer join build side skew (#1088)
discard ea35516ae1c [CARMEL-6209] Make defaultJoin in
BroadcastNestedLoopJoinExec running in parallel (#1080)
discard 5ee3529e6ea [CARMEL-6152] [Followup3] Handle FileNotFoundException
when listing event log files (#1083)
discard 5eafda27cd4 [CARMEL-6278] Avoid generating constraints if isnull and
isnotnull refer to multiple expressions (#1082)
discard 280b61976db [CARMEL-6283] DAGScheduler may stuck when there are too
many result task and result is spilled (#1085)
discard d8092193bb4 [CARMEL-6265] Only push down low cost expression (#1081)
discard c67115ac4b5 [CARMEL-6222]Improve congestion in shuffle service (#1068)
discard 6b6221e43f2 [CARMEL-6218] Throw exceptions to ensure the completeness
of event log files (#1073)
discard 86b6e50a65f [CARMEL-6208] Extract important information from
physicalPlanDescription in EventLog (#1069)
discard fd3df9d4d0f [CARMEL-6200][Follow-Up] Improve table lineage extraction
(#1070)
discard 950b4174ca8 [CARMEL-6148] Adjust Partition Size considering ExpandExec
project size in AQE (#1072)
discard eb2fd9a70cb [CARMEL-6117] Demote Bucket Join if Shuffle Cost
Acceptable (#1066)
discard 8cf4ec759c6 [CARMEL-6200] Add unified command to extract table lineage
information for Hadoop platform (#1067)
discard 5e6eb523ad3 [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join
condition (#1064)
discard ac84911529c [CARMEL-6190] Upgrade Scala to 2.12.16 (#1063)
discard c95d3418b52 [CARMEL-6178] Too many duplicate values of the join key
(#1062)
discard 9387627ee51 [CARMEL-6184] Make outputtofile meaningful (#1057)
discard f02d2ee27b3 [CARMEL-6166] Push local limit to both sides if join
condition is empty
discard 62e82b577f5 [CARMEL-6186] Skip constraint validation when CTAS for
Hive tables (#1058)
discard 063400d900c [CARMEL-6152] [Followup2] Calculate more accurate
information in the view- fix StackOverflowError (#1061)
discard e9e5f39780a [CARMEL-4174] Backport Hash join PR / Merge pull request
#1054 from carmel/hash_join
discard 97b0ece899a [CARMEL-6152] [Follow-up] Calculate more accurate
information in the view (#1055)
discard 156d8d94515 [CARMEL-6151] Exchange Push Down through Aggregate (#1053)
discard 590dc6bace4 [CARMEL-4174] Backport [CARMEL-5757] Avoid transform SMJ
to BHJ in AQE if rows > 50000000 detected
discard fc6a448e57b [SPARK-37753][FOLLOWUP][SQL] Add comments to unit test
discard f34f4eb38d1 [SPARK-37753][FOLLOWUP][SQL] Fix unit tests sometimes
failing
discard 2b43ba9bfd0 [SPARK-37753][SQL] Fine tune logic to demote Broadcast
hash join in DynamicJoinSelection
discard c67900a379e [SPARK-36898][SQL] Make the shuffle hash join factor
configurable
discard bc1dd6d5b6a [SPARK-35984][SQL][TEST] Config to force applying shuffled
hash join
discard 961174e4bdf [SPARK-35282][SQL] Support AQE side shuffled hash join
formula using rule
discard 53f3368d6bf [CARMEL-6159] Generate download records in
carmel_ql_object_v (#1052)
discard f5a43a2bb68 [CARMEL-6156] Fix PredicateReorder issue (#1047)
discard 4cab0abb286 [CARMEL-6121][Followup] Skip subsequent checks after the
first authorization is passed (#1051)
discard f1e565aba9e [CARMEL-6144] Add external shuffle service to blacklist if
connection creation fails (#1039)
discard 5067bae9b68 [CARMEL-6152] Support analyze view command to detect view
information (#1045)
discard 1a7fb37896a [CARMEL-6155] Fix limit only query contain aggregate
functions (#1044)
discard 6dd2e0b1955 [CARMEL-5851] Push partial aggregate through join (#1043)
discard e12e8d3d533 [CARMEL-6113] Support BHJ fallback to SMJ in AQE (#1042)
discard 20fb4393a0a [Carmel-6124] backport [SPARK-37618][CORE] Remove shuffle
blocks using the shuffle service for released executors (#1041)
discard 58c55ead3a3 [CARMEL-6143] Refactor bucket information in workload
analysis (#1040)
discard 9f6405ea976 [CARMEL-6133][FOLLOWUP] Fix scala.MatchError (#1038)
discard 21caf477856 [CARMEL-6134] Backport [SPARK-34628][SQL] Remove
GlobalLimit operator if its child max rows not larger than limit number (#1033)
discard 734e1fc64ff [CARMEL-6133] Improve adaptively adjust
spark.sql.adaptive.advisoryPartitionSizeInBytes (#1032)
discard d933af5ec44 [CARMEL-6139] Task thread stuck when coalese and
multi-thread scan is enable (#1037)
discard a52df6a12ac [CARMEL-6136] Use bucket scan size limit instead of normal
limit for disabled bucketing (#1036)
discard 8cd02349b6d [CARMEL-6135] Report Selected Buckets Number and Tasks
Count when Bucket Scan Disabled by Planner (#1034)
discard 038befb5d6e [CARMEL-6120] [Follow Up] Support track the optimized plan
using tag (#1035)
discard f5378c8f647 [CARMEL-6129] Backport [SPARK-39989][SQL] Support estimate
column statistics if it is foldable expression (#1031)
discard bcdbf824c89 [CARMEL-5936] Backport [SPARK-30602]SPIP: Support
push-based shuffle to improve shuffle efficiency
discard f50697160c4 [CARMEL-6121] Check subquery privilege objects in UPDATE
and DELETE command (#1025)
discard 24da01a40d7 [CARMEL-6100] Reduce Hadoop RPC call when compacting table
(#1029)
discard 4a56f858459 [CARMEL-6122] Add queue and application info into workload
analysis data (#1027)
discard 04b98c499d3 [CARMEL-6120] Support track the optimized plan using tag
(#1026)
discard 336cd980e71 [CARMEL-6106] Improve join stats estimation if one side
can keep uniqueness (#1024)
discard 0040dda69ce [CARMEL-6024]Bug fix for fetching large shuffle block
discard be522499491 [CARMEL-5936][BACKPORT][CARMEL-3545] Shuffle memory usage
improvement
discard dbe3670b5b2 [CARMEL-5936] Limit merge writer tasks number
discard df9c75bebfb [CARMEL-5936] Bug fix for bind exception from netty server
discard a1cda81bb3c [CARMEL-5936] Optimize merge mapStatuses and mergeStatuses
code
discard 663a32fdbf9 [CARMEL-5936] Limit the number of merge locations by
spark.shuffle.push.maxMergerLocations
discard 8d87c4199cd [CARMEL-5936] Optimize RemoteBlockPushResolver
discard bcd665b94ac [SPARK-39072][SHUFFLE]Fast fail the remaining push blocks
if shuffle stage finalized
discard c7c5febe5f6 [BACKPORT][SPARK-37675][SPARK-37793] Prevent overwriting
of push shuffle merged files once the shuffle is finalized
discard 169bdfa21bc [BACKPORT][SPARK-35548][CORE][SHUFFLE] Handling new
attempt has started error message in BlockPushErrorHandler in client
discard e52f762cc70 [SPARK-39080][SHUFFLE]Optimize shuffle error handler
discard 63b327ea2c5 [CARMEL-5936] Optimize RemoteBlockPushResolver
discard 13ee589ae88 [CARMEL-5936] Change external shuffle service port to 7338
discard 8a988351085 [SPARK-36772] FinalizeShuffleMerge fails with an exception
due to attempt id not matching
discard ede862c7298 [SPARK-33206][CORE] Fix shuffle index cache weight
calculation for small index files
discard ee0aa819a4e [SPARK-36378][SHUFFLE] Switch to using RPCResponse to
communicate common block push failures to the client
discard aa7cb12b29f [SPARK-33037][SHUFFLE] Remove knownManagers to support
user's custom shuffle manager plugin
discard 46fcdae487f [SPARK-32077][CORE] Support host-local shuffle data
reading when external shuffle service is disabled
discard 820b44411c2 [SPARK-37481][CORE][WEBUI] Fix disappearance of skipped
stages after they retry
discard ad360b72d95 [MINOR]Bug fix for testCases
discard 91462cf14d6 [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers
for push based shuffle
discard de3e938f071 [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization
for push-based shuffle
discard a6a831f4a6e [SPARK-37023][CORE] Avoid fetching merge status when
shuffleMergeEnabled is false for a shuffleDependency during retry
discard 14df3d5297b [SPARK-36892][CORE] Disable batch fetch for a shuffle when
push based shuffle is enabled
discard ea53c4fe17a [SPARK-36705][FOLLOW-UP] Support the case when user's
classes need to register for Kryo serialization
discard d539145fc16 [SPARK-36705][FOLLOW-UP] Fix unnecessary logWarning when
PUSH_BASED_SHUFFLE_ENABLED is set to false
discard 8bc42a8d961 [SPARK-36705][SHUFFLE] Disable push based shuffle when IO
encryption is enabled or serializer is not relocatable
discard 8ab806a20c5 [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId
shuffle files when finalize request for higher shuffleMergeId is received
discard 8ba68f24fac [SPARK-36389][CORE][SHUFFLE] Revert the change that
accepts negative mapId in ShuffleBlockId
discard 728d09a933d [SPARK-32919][FOLLOW-UP] Filter out driver in the merger
locations and fix the return type of RemoveShufflePushMergerLocations
discard d0164c9d4a6 [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage
retries for push-based shuffle
discard 1f3188d4476 [SPARK-36362][CORE][SQL][FOLLOWUP] Fix java linter errors
discard 00e0bc78511 [SPARK-36362][CORE][SQL][TESTS] Omnibus Java code static
analyzer warning fixes
discard 2cefa8e890e [SPARK-36374][FOLLOW-UP] Change config key
spark.shuffle.server.mergedShuffleFileManagerImpl to
spark.shuffle.push.server.mergedShuffleFileManagerImpl
discard f9b2e51451b [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level
user documentation
discard dacdc0ef9df [SPARK-32920][CORE][FOLLOW-UP] Fix string interpolator in
the log
discard 228aef5ba73 [SPARK-36266][SHUFFLE] Rename classes in shuffle RPC used
for block push operations
discard d79213d57b5 [SPARK-32920][FOLLOW-UP] Fix shuffleMergeFinalized
directly calling rdd.getNumPartitions as RDD is not serialized to executor
discard a2e4e844eca [SPARK-32920][FOLLOW-UP][CORE] Shutdown
shuffleMergeFinalizeScheduler when DAGScheduler stop
discard 70f9b4a0689 [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on
FileNotFound exceptions
discard 09db3a0eb1e [SPARK-36273][SHUFFLE] Fix identical values comparison
discard 0427756cc43 [SPARK-35546][SHUFFLE] Enable push-based shuffle when
multiple app attempts are enabled and manage concurrent access to the state in
a better way
discard 0940c41ea2a [SPARK-32922][SHUFFLE][CORE][FOLLOWUP] Fixes few issues
when the executor tries to fetch push-merged blocks
discard 52ba7f22c0b [SPARK-32920][CORE][SHUFFLE][FOLLOW-UP] Fix to run
push-based shuffle tests in DAGSchedulerSuite in ad-hoc manner
discard 1de132197e9 [SPARK-32922][SHUFFLE][CORE] Adds support for executors to
fetch local and remote merged shuffle data
discard 562fb49eb56 [SPARK-35836][SHUFFLE][CORE] Removed the reference to
spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite
discard 0cd82f5082f [SPARK-35671][SHUFFLE][CORE] Add support in the ESS to
serve merged shuffle block meta and data to executors
discard 49db4ca817c [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to
create merge directory and to get the local shuffle merged data
discard 6fb49cac6db [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge
with Push based shuffle and preparation step for the reduce stage
discard 0fbcb39439b [SPARK-32003][CORE] When external shuffle service is used,
unregister outputs for executor on fetch failure after executor is lost
discard cb902fef436 [SPARK-32210][CORE] Fix NegativeArraySizeException in
MapOutputTracker with large spark.default.parallelism
discard 4593e4ae265 [SPARK-32921][SHUFFLE] MapOutputTracker extensions to
support push-based shuffle
discard 573345a2f57 [SPARK-32055][CORE][SQL] Unify getReader and
getReaderForRange in ShuffleManager
discard b6cace76d37 [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged
shuffle …
discard c0e93b34152 [SPARK-32917][SHUFFLE][CORE] Adds support for executors to
push shuffle blocks after successful map task completion
discard 8ea13672912 [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Ensure
the number of chunks in meta file and index file are equal
discard 3fef6a6d3d5 [SPARK-32918][SHUFFLE] RPC implementation to support
control plane coordination for push-based shuffle
discard 4b4e1b30159 [SPARK-32919][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Driver
side changes for coordinating push based shuffle by selecting external shuffle
services for merging partitions
discard 427bf6817cc [SPARK-32103][CORE] Support IPv6 host/port in core module
discard ef3c3436b9e [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove
the newly added YarnShuffleServiceSuite.java
discard b9cb3c86669 [SPARK-32916][SHUFFLE] Implementation of shuffle service
that leverages push-based shuffle in YARN deployment mode
discard fd7f265360b [SPARK-32915][CORE] Network-layer and shuffle RPC layer
changes to support push shuffle blocks
discard b11685d693d [CARMEL-6055] Backport code-gen code for SortMergeJoin
(#987)
discard a1ec6fb223f [CARMEL-6057] Fix limit push down through window (#1016)
discard dbb8d88fa9f [CARMEL-6103] Log final spark plan and query id in Event
Analysis (#1023)
discard 8d8ab5780bd [CARMEL-6050] Support Remove Redundant Partial Aggregate
(#1022)
discard 06c431b382a [CARMEL-6109] Improve adaptive bloom filter join makes it
only works after both side shuffle (#1021)
discard a16abafc3c2 [CARMEL-6061] Skip the storage detection for access-denied
paths (#1013)
discard 70beb72d98c [CARMEL-6076][Followup] Fix the incorrect qualified table
path to determine the read lock (#1020)
discard 34780adeb1c [CARMEL-6077] Use `TakeOrderedAndProject` if maxRows below
the topKSortMaxRowsThreshold (#1015)
discard de7f557b696 [CARMEL-6087] User is continually blocked by circuit
breaker (#1009)
discard dabbefe1579 [CARMEL-6086] DAGScheduler exit because of jobId not exist
(#1008)
discard 396aa517b7e [CARMEL-6085] Show metrics after running analyze event
command (#1012)
discard 1e147656e91 [CARMEL-6055] Backport
[SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery
reuse (#993)
discard 25d0233595e [CARMEL-6055][SPARK-35874][SQL] AQE Shuffle should wait
for its subqueries to finish before materializing (#998)
discard c31f7276aa9 [CARMEL-5845] Add a logical plan visitor to propagate the
distinct attributes (#884)
discard 8adae4c167a [CARMEL-6083] Support batch account for non-columnar
format data read (#1007)
discard e5a79edcaa9 [CARMEL-6076][Followup] Fix UT failure caused by
compacting empty directory (#1010)
discard 01fb33611df [CARMEL-6076] Only obtain read lock in Hermes cluster when
file listing for empty directory (#1005)
discard c03dd28ac2d [CARMEL-6074] Should not insert adaptive bloom filter join
if it is partial aggregate #1001
discard c3ccc2d8c21 [CARMEL-6072] Improve error message for
SchemaColumnConvertNotSupportedException (#1004)
discard 2f8a1e6a294 [SPARK-34807][SQL] Transpose Window nodes with Project
between them (#995)
discard 48ecfcee899 [CARMEL-6068][SPARK-36137][SQL] HiveShim should fallback
to getAllPartitionsOf even if directSQL is enabled in remote HMS
discard ceb72aae9ac [CARMEL-6073] Make PartitionRecombinationExec effective
when bucketing disabled by planner (#1000)
discard 3e89b9963d0 [CARMEL-5818] Support analyze event log command (#994)
discard c1cd2be8afd [CARMEL-6069] Improve error message when cannot broadcast
the table with more than 50000000 rows
discard 2b932c08756 [CARMEL-6057] Limit push down through window (#990)
discard 27841f2be96 [CARMEL-6044] Push down limit 1 for right side of left
semi/anti join if join condition is empty (#984)
discard d9f56b90e83 [CARMEL-6030][FollowUp] Fix OutOfMemoryError-original fix
may cause task thread stuck (#986)
discard b06e3a1150b [CARMEL-6043] Avoid file number exceeds
spark.sql.dynamic.partition.maxCreatedFiles (#983)
discard b0cf5e225f4 [CARMEL-5997][FOLLOW UP] Support more sql patterns for
deciding bucketed scan dynamically (#980)
discard 20b9c9bc9ea [CARMEL-6030] Fix OutOfMemoryError: unable to create new
native thread (#979)
discard 70a1bfe3896 [CARMEL-6033] NullPointException in uploading when NOT
NULL constraint enforced (#976)
discard 2a16fb31aec [CARMEL-6012] Enhance rewrite table location to absolute
location based on database location (#970)
discard 8b1c5a39be7 [CARMEL-6010] Distinguish duplicate execution event log
for command query (#963)
discard 5550fdb67ee [CARMEL-5997] Support more sql patterns for deciding
bucketed scan dynamically (#975)
discard 0fe346f24dc [CARMEL-5970] Avoid introducing lots of partitions when
bucketing disabled by SQL Planner (#973)
discard 5819623bb51 [CARMEL-5986] Make Coalesce/Rebucketing effective when
bucketing disabled by planner (#961)
discard d6fd502829b [CARMEL-6009] Add realUser to caller context to support NN
throttling (#968)
discard a1b61bfa364 [CARMEL-6006] Log stack trace to track the source of query
cancelation (#962)
discard 6fc4feabca4 [CARMEL-6021] Improve error message for user specified
column list (#971)
discard b836220cfac [CARMEL-5987][SPARK-39217] Makes DPP support the pruning
side has Union (#967)
discard 347f4bfc963 [CARMEL-5999] Fix incorrect sql details in event log for
jobs that detect small files (#956)
discard 2a2758989f1 [CARMEL-5979] Add view usage information in event log for
workload analysis (#945)
discard 1e1391c32c6 [CARMEL-5995] Backport [SPARK-39216][SQL] Do not collapse
projects in CombineUnions if it hasCorrelatedSubquery (#966)
discard f1b868e3422 [CARMEL-6008]Bug fix for extracting db name from relation
multipartIdentifier (#959)
discard 0d2a6e762e7 [CARMEL-5962][backport][SPARK-33480][SQL] Support
char/varchar type (#936)
discard 72445922d94 [CARMEL-5994] Lock conflict between drop table and
metadata scan (#954)
discard 9c19847ebc4 [CARMEL-5963] CartesianProduct may introduce many
concurrent connections to JDBC Relation (#949)
discard 720ad125978 [CARMEL-5983] Backport [SPARK-34581][SQL] Don't optimize
out grouping expressions from aggregate expressions without aggregate function
(#941)
discard a4b54a83a71 [CARMEL-5869] Delete kafka.xml from source code(#885)
discard f101f70b421 [CARMEL-5984] Backport [SPARK-35264][SQL] Support AQE side
broadcastJoin threshold (#942)
discard fa58ac39d7f [CARMEL-5981][SPARK-34037][SQL] Remove unnecessary
upcasting for Avg & Sum which handle by themself internally (#940)
discard 1522b2ee22c [CARMEL-4914][FOLLOWUP] Fix Number of partitions (0) must
be positive
discard 8379a2a3a7b [CARMEL-5918]backport upstream performace changes (#913)
discard dcf55a1fcfc [CARMEL-5948] Backport SPARK-32859 and related follow-up:
Introduce p… (#937)
discard 9804e2141a3 [CARMEL-5914] Adaptive Bloom filter Join (#905)
discard 0d7b7c54224 [CARMEL-5978]Read table with a lot of empty patitions
might fail
discard 828c80ebda1 [CARMEL-5915][FollowUp] Filter view when listing tables to
avoid AnalysisException
discard b57689a9496 [CARMEL-5955] Drop NOT NULL constraint when alter column
type (#930)
discard 72c944a9ccf [CARMEL-5873] Upgrade Parquet to 1.12.2 (#901)
discard 5035f66e5d0 [CARMEL-5912] Support PARALLEL Hint against Delta Table
(#935)
discard 0dd2021f507 [CARMEL-5939]Delete the temp local block file after
reading it (#920)
discard 45da4c1b72a [CARMEL-5779] Compact Auto Scheduling (#922)
discard b1508a177e0 [CARMEL-5908] Support Table PARALLEL Hint in Number (#918)
discard 2d3f2737a8f [CARMEL-5941] Add request permission through
https://bdp.vip.ebay.com/aap/request-entry to error message
discard 43597eb391d [CARMEL-5933] Runtime Filter supports pruning side has
window (#909)
discard 8c2a7dde3d7 [CARMEL-5915] Add command to detect HDM storage leak (#904)
discard f27f1d56081 [CARMEL-5854] Data Read DML concurrent conflict (#906)
discard 669368b1717 [CARMEL-5931] Avoid file number exceeds
spark.sql.dynamic.partition.maxCreatedFiles(#908)
discard d647e582032 [CARMEL-5878] Backport upstream changes #888 from
hongyzhang/CARMEL-5878
discard 0ef658501a0 [CARMEL-5906] Expose constraint information in DESC
command for Delta table (#902)
discard ab97079c958 [CARMEL-5892] Support DPP if there is no selective
predicate on the filtering side (#903)
discard 0825e220177 [CARMEL-5902] Take dynamicPruningMaxInsetNum of rows to
avoid Driver OOM (#898)
discard 7221f3fd19c [SPARK-36874][SQL] DeduplicateRelations should copy
dataset_id tag to avoid ambiguous self join
discard a3427999f76 [CARMEL-5878]backp upstream test cases
discard d4efe4dbb51 [CARMEL-5855]Compact Transparency - Other DDL concurrent
conflict (#878)
discard d6ae21e4799 [SPARK-37524][SQL] We should drop all tables after testing
dynamic partition pruning
discard f566f43ceff [SPARK-37196][SQL] HiveDecimal enforcePrecisionScale
failed return null
discard ec878aa9723 [SPARK-37076][SQL] Implement StructType.toString
explicitly for Scala 2.13
discard 84d9ade193a [SPARK-37046][SQL] Alter view does not preserve column case
discard 2b8ca4e7850 [SPARK-36798][CORE] Wait for listeners to finish before
flushing metrics
discard 80c025038ba [SPARK-36717][CORE] Incorrect order of variable
initialization may lead incorrect behavior
discard 544f12e8c08 [SPARK-36919][SQL] Make BadRecordException fields transient
discard ccc92aa51e7 [SPARK-37556][SQL] Deser void class fail with Java
serialization
discard 0d2eee5a285 [CARMEL-5852] CHECK constraint release (#882)
discard 1578ee28720 [CARMEL-5867] ShowCreateTableCommand and
DescribeTableCommand miss nullable information (#880)
discard 4c840e36ee5 [CARMEL-5868] Backport SPARK-37199: Add deterministic
field to QueryPlan (#879)
discard c9524eece44 [Minor]Bug fix for driver env config (#881)
discard e443776d9e7 [CARMEL-5861]Reduce references to LiveExecutionData in
HiveThriftServer2Listener (#876)
discard f907c133d79 [CARMEL-5804] Enable NOT NULL definition for normal
parquet table - Fix (#875)
discard 7a0764736ee [CARMEL-5861] Limit the max size of statement and
executePlan (#874)
discard 5e7ce1be507 Log error message instead of stopping spark context (#872)
discard 651ea9972b7 [MINOR] Add close useless sessions through Hermes console
to error message (#869)
discard 24a9955d937 [CARMEL-5846] Disable Bloom filter pruning if filtering
side has scalar subquery (#868)
discard c83a4619679 [CARMEL-5467][FOLLOWUP]Support contains join (#867)
discard c0065f56b0c [CARMEL-5804] Enable NOT NULL definition for normal
parquet table (#857)
discard b069b9df893 [CARMEL-5841] Limit the row reference count in Range Index
(#865)
discard a86d25efb1e [CARMEL-4737][follow up] Do not add lock in inmemory file
index, since we may ready a empty table that we can not write
discard 21fb8f5c268 [CARMEL-4737] add lock for compact transparency (#849)
discard a2099bba252 [CARMEL-5467]Support contains join (#858)
discard 8694675e695 [FOLLOWUP][CARMEL-5736]Add SQL command to register HDM
table readiness (#859)
discard bca89e22305 [SPARK-36183][SQL][FOLLOWUP] Fix push down limit 1 through
Aggregate (#861)
discard cdf792c9a33 [CARMEL-5736]Add SQL command to register HDM table
readiness (#839)
discard f7c546e805d [CARMEL-5782][SPARK-37915][SQL] Combine unions if there is
a project between them (#855)
discard 2b9f41e428c [CARMEL-5783] Backport [SPARK-37922][SQL] Combine to one
cast if we can safely up-cast two casts
discard 49585ca882a [CARMEL-5750] Refactor basic table compaction functions
for Delta reuse (#841)
discard 5c6ea274543 [SPARK-35185][SQL] Improve Distinct statistics estimation
(#846)
discard 31efe91c2ed [CARMEL-5625] Support Table PARALLEL Hint in Size (#819)
discard 4f09165a492 [CARMEL-5757] Avoid transform SMJ to BHJ in AQE if rows >
50000000 detected (#844)
discard 0363b2c013a [CARMEL-5733] repartition number can be very small for
small table with huge number of small files (#842)
discard fffcdce6e13 [CARMEL-5758] Download support keep origin data type (#843)
discard d30861970d4 [CARMEL-5708] Runtime filter result may not be correct
because of weakReference Broadcast (#835)
discard e0ebba62e27 [CARMEL-5672] Use hive-service-rpc dep instead of inline
the generated code (#828)
discard 40a1c9627d9 [CARMEL-5668] Only show optimized logical plan with stats,
physical plan and rules time (#833)
discard 0f5c22224e5 [CARMEL-5677][FollowUp]Set the class log level on demand
(#834)
discard 0d793a3f0bc [CARMEL-5493]Quit spark thrift server when the
SparkContext is stopped (#821)
discard 9afe4b96bc2 [CARMEL-5673] Track the Memory Usage of Large Objects in
Driver (#829)
discard e40187eba6f [CARMEL-5677]Dynamic set class log level (#830)
discard 31d4a0ddc76 [CARMEL-5669] Use weakReference to store broadcast
relation in driver (#827)
discard 5fd4104304c [CARMEL-5661] Upgrade mysql-connector-java to 8.0.27 (#831)
discard 20deccde05f [CARMEL-5643]Add rpc consumed records metrcis (#815)
discard b4f7b7ad0e1 [CARMEL-5556] Invalid HDFS ACL set for empty table created
by CTAS (#813)
discard 82b7c866459 [CARMEL-5626] Use File Commit V2 for CTAS (#812)
discard 26ae84e55da [CARMEL-5658] NullPointerException in
BlockManagerMasterEndpoint.removeBlockManager (#826)
discard 0e04c1e3c71 [SPARK-20977][CORE] Use a non-final field for the state of
CollectionAccumulator (#824)
discard f785b02f417 Revert "Revert "[CARMEL-5007] Make
BlockManagerMasterEndpoint to run in multiple threads (#567)" (#822)" (#825)
discard 436560b09c2 Revert "[CARMEL-5007] Make BlockManagerMasterEndpoint to
run in multiple threads (#567)" (#822)
discard 16a077c126b [CARMEL-5619] Port [CARMEL-3436] Mechanism to protect
Carmel away from uncertified UDFs (#810)
discard a0ae2cb3662 [SPARK-34622][SQL] Push down limit through Project with
Join (#818)
discard bc870099a4b [CARMEL-5007] Make BlockManagerMasterEndpoint to run in
multiple threads (#567)
discard a480eaa9516 [CARMEL-5529] ignore thread interrupted signal and do not
remove the … (#795)
discard 9c0d8feafa7 [CARMEL-5606] Use File Commit V2 instead of V1 for temp
table creation (#808)
discard d9cb9542656 [CARMEL-5566] Large result getter thread slow (#807)
discard 473a503525c [CARMEL-5564] Backport [SPARK-37270][SQL] Fix push
foldable into CaseWhen branches if elseValue is empty (#806)
discard cb48c83cdbb [CARMEL-5554] Failed to rename FileStatus when inserting
dynamic partition by quota-free committing with V1 algorithm (#797)
discard 51e485a7881 [CARMEL-5527] NPE throws in HiveThriftServer2Listener
(#802)
discard 009e2fdc08e [MINOT] Fix String format exception (#801)
discard c575b509bd0 [CARMEL-5426] Upgrade Delta version to
0.9.0.carmel0.1-SNAPSHOHT (#803)
discard 8026567ce63 [CARMEL-5541] Clean blockmgr temp files when no reference
(#796)
discard d157adcd2b6 [CARMEL-5515] Filter push down through window if
partitionSpec isEmpty (#798)
discard f190f66785a [CARMEL-5526] Skip cancelled stages when calculate active
tasks for queries(#792)
discard a291c18b0b1 [CARMEL-5426] Upgrade Delta version to
0.8.0.carmel0.4-SNAPSHOH (#794)
discard d7126e86949 [CARMEL-5504] Improve error message when creating table in
non exists database
discard cecda22144d [CARMEL-5514] Push down limit through JDBC relation (#788)
discard 9ee791e8609 [CARMEL-5506] False positive probability must be within
range (0.0, 1.0) (#787)
discard 2d44e1d3b31 [CARMEL-4798] switch to admin when insert into whitelist
(#781)
discard 9a4ac945e53 SPARK-31809: Infer IsNotNull for join condition (#782)
discard 58c8d8e6ed6 [CARMEL-5426] Upgrade Delta version in Carmel (#784)
discard 9fa4561ecd3 [CARMEL-5063][FollowUp] Fix dynamic allocation issue and
make stats type configurable (#783)
discard 9d8e60d827f [CARMEL-5501] Update Update/DELETE is only supported with
v2 tables messages (#780)
discard db42727596e [CARMEL-5494] Clean physical plan information before stage
execution (#779)
discard 5ab2d826351 [KUDU-88] Implement SupportsReportStatistics in
V1ScanWrapper for reading Kudu tables (#775)
discard 81ec8a48ee3 [CARMEL-5063][FollowUp] Fix task number not accurate issue
when doing dynamic allocation (#777)
discard 08c28c2aa33 [CARMEL-5442] Fall back to true if InSet size exceed
DYNAMIC_PRUNING_MAX_INSET_NUM (#762)
discard 460dd4d1fbf [KUDU-90] DDL for carmel table should be case insensitive
(#766)
discard 412134fce01 [CARMEL-5426] Upgrade Delta version in Carmel
discard bac5e112bd3 [CARMEL-4798][FOLLOWUP] Improve errormessage (#771)
discard 8ab7c99dc36 [CARMEL-5458][FOLLOWUP]Add maxAccurateSkewedBlockNumber if
shuffle blocks size skewed (#773)
discard 90d8a2d05e3 [CARMEL-5400][FollowUp] Add more log for shuffle fetch
request congestion control (#774)
discard c7326684744 [CARMEL-5063][FollowUp] Fix dynamic allocation mechanism
issue (#765)
discard 589127a9c7f [CARMEL-5431] Limit the cached FileStatus size in driver
memory (#767)
discard d56ca0e08f5 [CARMEL-5458]Add spark.shuffle.accurateBlockSkewedFactor
parameter to determine whether to report a shuffle block size(#770)
discard cc1fa67c4c0 [CARMEL-5454] Tasks from some users cannot be scheduled in
pymt queue (#768)
discard 45b04560839 [CARMEL-5440] Correct running tasks by considering the
status update delay (#758)
discard 2ebf9c3ba1d [CARMEL-4798] block small file table insert and unblock
using compact command (#522)
discard 6617923dd04 [CARMEL-5400] Too many chunkFetchRequest make shuffle
service very slow (#764)
discard 5dbff328a4c [CARMEL-5437] Log sessionId in hadoop rpc caller context
(#756)
discard 8a8643f9b17 [CARMEL-5307][FOLLOWUP] Skip authorization check for
ExplainOptimizeCommand (#760)
discard 5cf793ee2ca [CARMEL-5448] Fix scala.MatchError in
ExplainOptimizeCommand (#763)
discard 7c2a20bb551 Revert "[CARMEL-5426] Upgrade Delta version in Carmel
(#753)" (#761)
discard 632ae75904c [CARMEL-5426] Upgrade Delta version in Carmel (#753)
discard 93a030a3cad [CARMEL-5435] Implement bloom filter join to improve join
performance (#743)
discard 420362a29ad [CARMEL-4636] Add partition predicate if scan partitioned
tables (#752)
discard f581aa565b3 [CARMEL-5415] Use CPU time instead of system time to
calculate analysis time (#754)
discard 8823e715753 [CARMEL-5394] Query result not sorted as expected when
spill happens (#749)
discard a9acf92a76b [CARMEL-5396] Enable Range Join with Interval Overlap
Pattern (#747)
discard cc8f402e07d [CARMEL-5185][FollowUp] Fix disk leak issue when spill
large broadcast result (#748)
discard 08779be0149 [CARMEL-5333]fix NegativeArraySizeException for rownumber
optimization (#744)
discard 38eecd5f383 [CARMEL-5393] Don't store broadcast relation in driver's
memory (#746)
discard cff7f62414d [CARMEL-5347] Fix NPE issue for user fair scheduling (#745)
discard 4e74261ac60 [KUDU-86] Support create a Kudu table with clause "USING
carmel" (#740)
discard 5b78893f94e [KUDU-37][FOLLOW-UP] Fix authorization mistake (#741)
discard f0ceb923b98 [KUDU-83] Log task wait/block time (#742)
discard cbba4b80bf9 [CARMEL-5185] Spill result to disk for large broadcast
(#649)
discard 6751b0b774b [CARMEL-5243] Support Range Join by Identifying Range
Pattern from Co… (#739)
discard e3446de365c [CARMEL-5370] Local HDFS cache may not be applied after
upgrade to Sp… (#735)
discard 6527961c290 [CARMEL-5063][FollowUp] Improve spark dynamic allocation
mechanism (#736)
discard 82c51caeff9 [CARMEL-5308][FollowUp] Fix minus stats issue for running
sql (#737)
discard db8977f6fbf [CARMEL-5356][FollowUp] Make even one partition data don't
stored the whole deserialized result rows in memory (#738)
discard 83c1a7c7d48 [CARMEL-5347] Fair task scheduler consider user resource
usage (#734)
discard 91d39457f43 [CARMEL-5063] Improve spark dynamic allocation mechanism
(#699)
discard 76f3f75a1ef [CARMEL-5356] Avoid too many deserialized result rows
stored in driver memory (#733)
discard baf20002292 [CARMEL-5319] Make BroadcastNestloopJoin interruptible
(#732)
discard c7e488e47ce [CARMEL-5337] Suppress logs related to session event
logger (#730)
discard a6b7da31ecb [CARMEL-5335] Fix shuffle service slow issue (#731)
discard feec4c95714 [CARMEL-5308][FollowUp] Fix stage stats endtime and other
issue (#728)
discard 91a03f06cae [CARMEL-5327] Use IP as thrift server address instead of
hostname (#729)
discard 3c3eac47328 [KUDU-18] Refine KUDU table DDL (#726)
discard 1a58c662695 [CARMEL-5308][FollowUp] Expose running sqls and stats to
hermes console for monitoring (#727)
discard 3a64526731d [CARMEL-4546][FOLLOWUP] Avoid unnecessary shuffle when
enable bucketing (#698)
discard 1609c5e9085 [followup][CARMEL-5264&CARMEL-4546]enable window function
optimize & bucket compact by default (#705)
discard c3e8727301c [CARMEL-5308] Expose running sqls and stats to hermes
console for monitoring (#723)
discard b891aa32348 [SPARK-34812][SQL] RowNumberLike and RankLike should not
be nullable (#725)
discard 50cc69ab99d [SPARK-36444][SQL] Remove OptimizeSubqueries from batch of
PartitionPruning (#724)
discard 625769f30bc [CARMEL-5307] Skip authorization check for
ExplainOptimizeCommand (#710)
discard 92f45b602c4 [CARMEL-5272] Improve coalesce shuffle partitions (#688)
discard e8b17190e00 [CARMEL-5119] Don't block chunkfetchrequest in the
chunkrequesthandler thread (#692)
discard d984d93088a [CARMEL-3532] Input metrics miss for file scan (#701)
discard 7dab2085624 [CARMEL-5264] improve window function (#695)
discard eb39aa0ac89 [KUDU-65] Failed to push down project to a DSv2
relation(KuduRelation) in a subquery (#648)
discard 037ebd18df7 [CARMEL-5268] Authorization fail throw improper error
message (#690)
discard 3c6ca685604 [CARMEL-5202] Backport SPARK-36086, SPARK-33225 and
SPARK-36093 to fix NoSuchElementException when creating DELTA table (#664)
discard 565c670babf [CARMEL-4193][FOLLOWUP]skip privilege check if
authorization is disabled (#689)
discard 3388aaf818c [CARMEL-5255] Update-Join and Delete-Join command parse
failed (#685)
discard 085e5aed99a [CARMEL-4546]support compact command for bucket table
(#686)
discard f9e8217470a [CARMEL-5234][FOLLOW-UP] Support Reuse
BroadcastRangeExchangeExec (#684)
discard 33e2c8dfe69 [CARMEL-5229]Resolve table schema with upload csv header
in not case insens… (#679)
discard 719d7c0f759 [CARMEL-5234] Split BroadcastRangeExchangeExec from
BroadcastRangeJoinExec (#681)
discard 9eb6597635d [CARMEL-5203][Follow-Up] A lot of tables missing metadata
when running DESC EXTENDED (#678)
discard fdca37616ed [CARMEL-5223] Backport [SPARK-36183][SQL] Push down limit
1 through Aggregate if it is group only (#680)
discard afd3fd06bf0 [CARMEL-5218] Fully Support Partial Range Join (#676)
discard f393e8f2d43 [SPARK-34534] Fix blockIds order when use
FetchShuffleBlocks to fetch blocks (#665)
discard d2223e7ceba [HOT-FIX] Disable Parquet not in filter push down if its
values exceeds pushDownInFilterThreshold (#675)
discard 9ebc4750b6e [CARMEL-5210][SPARK-28863][SQL] Make sure optimized plan
will not be re-analyzed (#673)
discard 6ee9c7bc809 [CARMEL-5205] Follow-up of [CARMEL-5124] Fix missing
shuffle in a com… (#671)
discard f81d589e8b4 [CARMEL-5171]Add scan size limitation for bucketed RDD
(#642)
discard 7df88cca918 [CARMEL-5173]Show correct error message if table not
exists in CTAS (#666)
discard 3eb33849b70 [MINOT] Fix flaky test in BucketedReadSuite (#672)
discard 7576d44efb5 [CARMEL-5200]ODBC Call for Decimal Data Precision returned
as zero (#667)
discard e79968d48cd [CARMEL-5199][FOLLOWUP] Broadcast outer join stream side
support AQE (#668)
discard 206a4610125 [CARMEL-5208] Support Partial RangeJoin (#670)
discard d2e5bd4b6d2 [CARMEL-5207] Carmel UT failed with newly built Delta jar
(#669)
discard 12eec2e7dfd [CARMEL-2514] keep the output partition name align with
partition schema (#96)
discard f694ae2b342 [CARMEL-5150][Follow-Up] Failed to rename FileStatus (#658)
discard 1a6c71cd551 [CARMEL-5199] Broadcast the stream side of outer join
(#660)
discard 8c1e1a8f9ca [CARMEL-5201] Support Cross Join with Range Pattern (#661)
discard f3d01b549bc [CARMEL-5198] Backport [SPARK-32792][SQL] Improve Parquet
In filter pushdown (#663)
discard 1eb6bbd48b0 Fix bug of single stage skewed join (#656)
discard 072ffee04d9 [CARMEL-5194] Support disabling timeline in Spark UI to
fix slowness of page loading (#654)
discard fa7689de47a [CARMEL-5190] Enhance makeOffer lock and dump when wait
too long (#653)
discard 1f558536a44 [followup][CARMEL-5114]fix serialization failure for
ExecutionEnd event (#645)
discard 56cf4c41b62 [CARMEL-5143] Add configuration to enable final check
(#646)
discard de8e80ee394 [CARMEL-5155] Expose executors number to track cluster
resource usage (#638)
discard 2b3e5c9f5c3 [CARMEL-5150] Failed to rename FileStatus (#639)
discard f3864318b4d [CARMEL-5151][SPARK-35886][SQL][3.0] PromotePrecision
should not overwrite genCode
discard 25fe015fea1 [Followup][CARMEL-5114]fix kryo dependency conflict (#640)
discard 70d07eb3813 [CARMEL-5139]Resolve aggregate column reference in window
expression (#618)
discard 3cc457374e1 [CARMEL-5154] Correct the data size of range index (#631)
discard d10bfa10bb7 [CARMEL-5143] Add a final check to ensure no Shuffle/Sort
is missing before execution (#622)
discard ca961150306 [CARMEL-5157] KafkaBatchEventProducer fail to stop (#636)
discard 2ef95c8b799 [CARMEL-5114] support kryo serializer in kafka event queue
(#626)
discard 8fbc892b6ab [CARMEL-5153] Add more UT for Range Join and Disable Range
Overlap (#630)
discard b8ffd429f77 [CARMEL-5142]Bug fix for ArrayIndexOutOfBoundsException
during messag… (#620)
discard c9fb44ed620 [CARMEL-3990][FOLLOW-UP] Add query submitted time (#619)
discard 2fb61344994 [CARMEL-5083] Support Range Join in Spark 3.0 (#611)
discard 0f0edb48082 [SPARK-32038][SQL] Regression in handling NaN values in
COUNT(DISTINCT) (#616)
discard 822eec3b6d3 [CARMEL-5094]Backport [SPARK-33741][CORE] Add min
threshold time spec… (#586)
discard ed91d5076e5 [CARMEL-5138] Fix the makeBatchOffers
ArrayIndexOutOfBoundsException during decomm (#617)
discard 56e9db51b9d [CARMEL-1077][FOLLOW-UP] Improve bucket table scan
performance, if the matched bucket number is only a little, treat the bucket
table as normal table (#604)
discard 5c4dcd66e9c [CARMEL-4232] Support list all tables AND their size in a
specific HDM (#553)
discard 10b2945ee50 [SPARK-33726][SQL] Fix for Duplicate field names during
Aggregation (#607)
discard 665886db27b [SPARK-32559][SQL][3.0] Fix the trim logic in
UTF8String.toInt/toLong did't handle non-ASCII characters correctly (#613)
discard 036d078d7b5 [SPARK-32753][SQL][3.0] Only copy tags to node with no
tags (#612)
discard b9173c3bbcd [CARMEL-5126] Backport [SPARK-32908][SQL] Fix target error
calculation in percentile_approx() (#605)
discard c3b8761c188 [SPARK-33260][SQL] Fix incorrect results from SortExec
when sortOrder is Stream (#609)
discard de32f22f598 [SPARK-33404][SQL][3.0] Fix incorrect results in
`date_trunc` expression (#608)
discard b0d1c6f3cfb [SPARK-32136][SQL] NormalizeFloatingNumbers should work on
null struct (#615)
discard d68bd8a9ab4 [SPARK-32764][SQL] -0.0 should be equal to 0.0 (#610)
discard d6007d865ab update codestyle (#606)
discard 734e7fda666 Revert "[CARMEL-5061] Backport [SPARK-35568][SQL] Add the
BroadcastExchange after re-optimizing the physical plan to fix the
UnsupportedOperationException when enabling both AQE and DPP"
discard b25f4211c18 [CARMEL-5129]Bug fix for thread leak in
JobCancellationSuite (#603)
discard 62885330ac6 [CARMEL-5124] Fix missing shuffle in a complex query (#602)
discard 8aa4b9a78da [CARMEL-5124] add more debug log to EnsureRequirements
(#601)
discard 5a8c4b3a0c2 [CARMEL-5110] Clean up quota free directory when session
closed (#598)
discard a42f5636f08 [SPARK-31179] Fast fail the connection while last
connection failed in fast fail time window (#597)
discard 3f2eca01b99 [CARMEL-5106] Fix ReusedSubqueryExec cannot be cast to
LogicalPlan (#593)
discard d84d589be0b [CARMEL-5102] Support Sync Decomm Nodes to Yarn (#590)
discard 8c1c43fbff3 [CARMEL-5095]Add limitation of user sql size (#587)
discard de2e66f75d6 [CARMEL-5101]Do not cleanup ResultShareGroup in
SparkExecuteStatementOperation (#588)
discard 54c3acb2b1b [CARMEL-5061] Backport [SPARK-35568][SQL] Add the
BroadcastExchange after re-optimizing the physical plan to fix the
UnsupportedOperationException when enabling both AQE and DPP
discard 7bc7132a92a [CARMEL-5088] "java.util.NoSuchElementException: key not
found" in DAGScheduler (#589)
discard 2f338fc836b [CARMEL-1077] Improve bucket table scan performance, if
the matched bucket number is only a little, treat the bucket table as normal
table (#591)
discard 2b7e9768323 [CARMEL-5080] Optimize Kafka event serializer dispatcher
(#585)
discard 71d45b50f17 [KUDU-37] Support authorization of DSv2 update/delete
operation & Support drop Kudu table (#584)
discard 6cfe0fd36b4 [CARMEL-5078] AssertionError throws for some sql (#583)
discard 68a72c508cb [CARMEL-5070] Support more accurate CPU usage calculation
(#580)
discard be3930160b4 [CARMEL-4037] Unnecessary sort is introduced when skew
handling is applied on join with only one query stage (#581)
discard 761580971de [CARMEL-5087][SPARK-33979][SQL] Order filters based on
selectivity and cost (#356)
discard 51fcf14b0c1 [CARMEL-5084] Backport [SPARK-34808][SQL] Removes outer
join if it only has DISTINCT on streamed side (#579)
discard c7afce9f429 [CARMEL-5061][FOLLOWUP] Move
DynamicPartitionPruningSuiteAEOn to ExtendedSQLTest (#582)
discard 596554ffe86 [CARMEL-4974][FOLLOWUP] Improve lock cententions of the
TaskResult handling (#572)
discard 8bd8e4a58ae [CARMEL-4959][follow up]kafka event queue improve and add
config for capacity (#569)
discard b2eff041e99 [CARMEL-5048] Support Decommission Nodes (#577)
discard 840196dcae3 [KUDU-48] add missing configuration to publish jar (#578)
discard ea2029b6ddc [CARMEL-5061] Avoid UnsupportedOperationException when
enabling both AQE and DPP
discard e30fa0e2554 [CARMEL-5062] Fix the AM un-expected restart due to RM
failover (#575)
discard ff9fc5fd810 [CARMEL-5054] Backport [SPARK-20628][CORE][K8S] Start to
improve Spark decommissioning & preemption support (#574)
discard e85150fb22e [CARMEL-5019] fix case mismatch for hive table (#563)
discard 1bc0c8c1918 [KUDU-48] Publish Carmel-Spark to eBay repo for Kudu
reference (#568)
discard d27a74a95ee [CARMEL-5030] Query failed with exception 'zip RDDs with
unequal numbers of partitions' (#566)
discard 257fd794bcd [CARMEL-5029] Fix query plan time in ql_metrics table
incorrect (#564)
discard 72fe85d7b4e [CARMEL-5021] Backport [SPARK-33822] Use the
CastSupport.cast method in HashJoin (#562)
discard facc6c8be56 [CARMEL-4488] Misleading DSQuotaExceededException Error
(#535)
discard f6e1e59cf2b [CARMEL-5024] Support task set summary for topN resource
usage (#560)
discard d06cb31600c [CARMEL-4959] optimize produer to handle skew when
serialize (#536)
discard e512c9e1703 [CARMEL-5005][FOLLOW-UP] Enable executor related events
when UI disabled (#559)
discard 4cbf52f650e [CARMEL-5016] Fix the display problem of access
non-parquet table to make jdbc checking in one execution (#558)
discard cc41abbbbd3 [CARMEL-4974] Optimize task result handling logic, move
the TaskSetManager status update logic to TaskResultGetter (#521)
discard 3f86b58fa44 [CARMEL-5000][FOLLOWUP] Restore origin AQE config after
ExplainOptimizeCommand
discard ed21b2566b3 [CARMEL-3166] Fully support DPP on AE
discard 3738d489276 [CARMEL-5020] Backport [SPARK-35286][SQL] Replace
SessionState.start with SessionState.setCurrentSessionState (#539)
discard 4a947763816 [CARMEL-5015][FOLLOW-UP]Fix shuffle metrics compute logic
(#555)
discard fc735def9ae [CARMEL-5015][FOLLOW-UP] Fix NPE and Concurrent issue for
shuffle metrics compute (#554)
discard 8f99e149d01 [CARMEL-5015][FOLLOW-UP] Fix NPE (#552)
discard ea3a8a36af5 [CARMEL-5014] Fix the memory leak of
InMemoryStore(kvstore) (#550)
discard e29565b8843 [CARMEL-5005] Support disable Spark UI (#548)
discard 884d6bb0c30 [CARMEL-5015] Move query metrics sink and shuffle size
check out of S… (#551)
discard 93ad9894e43 [CARMEL-4929][FOLLOWUP] Support spark driver runs as
Unmanaged AM (#545)
discard a5f0d5f1fea [CARMEL-5009] Backport [SPARK-33687][SQL] Support analyze
all tables in a specific database (#264)
discard 125891d3028 [CARMEL-4964] Make SparkSession purely serializable (#515)
discard a143acdd03d [CAREML-4766][FollowUp] Close all shuffle connections in
seperate threads (#544)
discard 6cb318d8023 [CARMEL-4978]Retain 7 days logs on server to avoid too
much disk usage (#530)
discard 1f6c7295f5a [CARMEL-4987] Remove useless logical relation in
InsertIntoDataSource (#541)
discard 00031209c72 [CARMEL-5000] Workaround to make explain works with AQE
(#543)
discard c2041f8680d [CARMEL-4998] Improve Rewrite NOT IN to NOT EXISTS (#540)
discard 2fc761241b4 [CARMEL-4996] Fix ExplainOptimizeCommand privilege issue
(#538)
discard 9d08b2e38c5 [CARMEL-4980] Cache the RejectedExecutionException if the
executor pool is shutdown (#528)
discard 4cf49255c98 [CARMEL-4898][follow up] reduce log size in kafka event
queue (#506)
discard fa757195b32 [CARMEL-4929] Support spark driver runs as Unmanaged AM
(#524)
discard 0c7fd6efceb [CARMEL-4880][FOLLOW-UP] Fix incorrect query state after
being cancel… (#520)
discard e3d6c4f76f9 [CARMEL-4900] Sql failed because of FetchFailedException
(#533)
discard 868c6e3ae34 [CARMEL-4886][follow up] fix npe when not get session info
(#525)
discard 6ebffcd331b [CARMEL-4981] Traffic shape handler thread leak in ut
(#529)
discard 262a0538b38 [CARMEL-4963] Support monitoring on active taskset (#512)
discard bc57147175f [CARMEL-4921] Correct the skew flag when the replacement
of shuffle reader skipped (#523)
discard dee3d23caec [CAREML-4766] Fix slow issue when shuffle tcp connection
only shared in stage (#518)
discard b8a7bcfca38 [CARMEL-4188] Support sinking operation log to MySQL (#504)
discard 9aa85d27e1e [CARMEL-4944] Avoid unnecessary RPC call to
BlockManagerMaster when submit Jobs/Stages (#519)
discard d047cd48eed [CARMEL-4944] Construct the taskMetrics early and offload
the work from DAGScheduler event process loop (#516)
discard fde10f2388f [CARMEL-4965] maxNumConcurrentTasks should be defined to
lazy (#513)
discard 49207ca9e95 [CARMEL-4955] Fix 'Automatically cleanup broadcast' flaky
test in ContextCleanerSuite (#507)
discard 8d5bdefc6ab [CARMEL-4944] Improve Spark DAGScheduler performance by
offload heavy… (#505)
discard 93d8c755725 [SPARK-35273][SQL] CombineFilters support
non-deterministic expressions (#509)
discard c8644981cd6 [CARMEL-4941] IllegalStateException in spark ui (#497)
discard a19286a5d50 [CARMEL-4886] Fix npe when not get session info (#498)
discard 995b5b64e92 [HOT-FIX] Fix scala style issue (#503)
discard f36613a8d62 [CARMEL-4898] make producer multi-thread (#495)
discard d622b8beb01 [CARMEL-4230] add error code to error msg (#487)
discard d02442e3560 [CARMEL-4880] Fix the InterruptedException error for
Cancelled query (#480)
discard 0b236fa4154 [CARMEL-4916] Refine top running API to reduce potential
connection leak (#494)
discard 5594781393f [CARMEL-4833] Query failed due to scala.MatchError (#468)
discard 7a47f76bff4 [CARMEL-4101] Close InputStream when getting all indexes
(#496)
discard 1c1d58cdab3 [SPARK-32234][SQL] Spark sql commands are failing on
selecting the orc tables
discard be8236fc54f [SPARK-34897][SQL] Support reconcile schemas based on
index after nested column pruning
discard f8d3726df2d [SPARK-35096][SQL] SchemaPruning should adhere
spark.sql.caseSensitive config
discard 9b0464568ec Backport [SPARK-34949][CORE] Prevent BlockManager
reregister when Executor is shutting down (#493)
discard 7ec70559244 [CARMEL-4838] Shuffle throttling tracing and tcp
connection isolation (#456)
discard 60d597ad6c5 [CARMEL-4914] Add a callback map to ensure the session
resources closed (#488)
discard 23254594e85 [CARMEL-4823] Critical queue events analysis and
monitoring (#483)
discard 29bbab142c8 [CARMEL-2483] Choose non-bucket side as the build side of
NestLoopBroadCastJoin (#476)
discard 2d1ebdfca19 [CARMEL-4884] Pypass the data columns match exception for
legacy empty delta table insertion (#478)
discard 770fa22a598 [CARMEL-4896] Add a dedicated event queue for query log to
avoid losing query log (#482)
discard b14b063d25f [CARMEL-4907] OptimizeSkewedInsert should work for delta
table (#486)
discard e42d1fd3aa2 [CARMEL-4874] Role grant with admin option failing on
spark 3.0 (#473)
discard 673bb063109 [CARMEL-4905] Remove the leaked live entities with a TTL
map (#484)
discard 4e1f5ec2b8b [CARMEL-4824] Cache GetLocationsAndStatus to reduce
overhead of seria… (#459)
discard c5a7df2ed44 [CARMEL-4895] ExternalCatalogListener make SparkSession
leak (#481)
discard 565bcae11a4 [CARMEL-4887] Consider driver slow impact when expire
executor (#475)
discard 4f9845d3708 [CARMEL-4867] Add DDLUtils.DDL_TIME into tables missed in
3.0 which may be used in Delta (#470)
discard b74db14e9fc [CARMEL-4881] Fix the slowness of makeOffer in
TaskScheduler (#474)
discard 90c745bdb72 Backport [SPARK-34087][SQL] Fix memory leak of
ExecutionListenerBus (#469)
discard a6420a6fbb1 [CARMEL-3408][FollowUP] fix missing metrics (#467)
discard 6f86978518c [CARMEL-4841] Fix the issue of qualify window function
with group by (#462)
discard 4f6c94317a3 [CARMEL-4842][CARMEL-4238][FOLLOWUP] craete table stored
as parquet not work when disable non-parquet (#460)
discard 1777fba64ea [CARMEL-4847] Allow create managed table with existing
files (#463)
discard 0a9e82ffcea [CARMEL-4816] Remove external heavy accumulator from event
listener bus (#455)
discard 25eaf530b55 [CARMEL-4750] Support create table like delta table (#458)
discard 8760b7c7783 [CARMEL-4839] Refactor DynamicPartitionPruningSuite for
DPP on AE (#457)
discard 46183455f6b [CARMEL-4780] Allow disabling logs for task status update
(#1748) (#449)
discard 1241ab912a6 [CARMEL-2421] HostToRack cache in TaskScheduler (#1096)
(#450)
discard 4b38d9bc616 [CARMEL-4815] Stop JVM if spark context stopped (#447)
discard 08a6e1f89f8 [CARMEL-4821] Support bad disk detection using Exception
(#446)
discard 31b15fdaec1 [CARMEL-1379][FOLLOW-UP] Fix ConcurrentHashMap misusage
and remove some logs (#431)
discard 9acdc8b734c [CARMEL-4814] Use proper file system (#444)
discard efbec18ec35 [CARMEL-4834] Remove assert partition numbers in
ParquetFileFormat
discard 2f5053598e8 [CARMEL-4832] Backport SPARK-32884 and SPARK-32855 Improve
dynamic partition pruning evaluation (#441)
discard 638818a0503 [CARMEL-4812] Allow run bin/spark-sql (#440)
discard dabdf3ca036 [CARMEL-3408] ListenerBus should send event through kafka
in multi-th… (#427)
discard e5da672aa61 [CARMEL-4806] Support check query DAG by query id (#439)
discard 376f8d8fd2b [CARMEL-4805] Simplify time calculation in ListenerBus
(#438)
discard 61916bf3b6c [CARMEL-4787] Misleading error message for make offers in
3.0 (#437)
discard e5ddf15042f CARMEL-4685: Allow admin_readonly role unblock user (#435)
discard 127c4339b66 [CARMEL-4238][FOLLOWUP] Friend error message of blocking
non-parquet (#434)
discard 71880b18fb4 [CARMEL-4796] Fix ACL issue for CTAS of delta table with
subquery (#436)
discard 97b3a003843 [CARMEL-4784][CARMEL-4767] Fix create and grant role issue
(#433)
discard 873248121d2 [CARMEL-4701] Remove heavy monitor lock in CatalogManager
(#409)
discard c6b92d24fa9 [CARMEL-4759] disable set location (#425)
discard bcc56b80598 [CARMEL-4624] Unblock users for zeta scheduler sessions
(#406)
discard 1ad883fc61c [CARMEL-4743] Support monitoring on zombie tasks (#416)
discard ff7619bffcf [CARMEL-4768][CARMEL-4775] Fix NPE issue when `show grant`
(#426)
discard 0d4ef45e9ec CARMEL-4773: Fix role name with upper case (#423)
discard 4904685635e [CARMEL-4361][SPARK-34772][SQL] RebaseDateTime
loadRebaseRecords should use Spark classloader instead of context (#421)
discard 14c15a4d4e0 [CARMEL-4728] Allow executing AlterTableDropPartition
inside DeltaCommand (#411)
discard cf688f233f2 [CARMEL-4770] Executor is removed long time ago but the
Spark UI still show it is active (#422)
discard 5f9771d3a49 [CARMEL-4769] Disable logSlowEvent in 3.0 (#420)
discard fb91f2d08c5 [CARMEL-4753][backport][CARMEL-1671] Add Queue Info into
Metrics in Spark Driver (#418)
discard cfb1dee6632 [CARMEL-4729] Restrict the paths used as table location
(#412)
discard 7217f14cdb4 [CARMEL-4739] Avoid to use SNAPSHOT to improve download
jar from eBay central repository (#414)
discard d9b76d9a738 [CARMEL-3836][FOLLOW-UP] Bucket table scan
outputpartitioning may be incorrect (#408)
discard 554c7df3246 [CARMEL-4660] Limit max cube expression size to 12 (#407)
discard 782be3a9067 [CARMEL-4679] Expose runtime metrics of query (#403)
discard 7f7998efd3e [CARMEL-4674] Correct the total cores number leveraging
ExecutorAllocationManager (#404)
discard c555ac6a5ef [CARMEL-4664] Lazy listing support for only have dynamic
partition filter (#399)
discard 4a32994667c [CARMEL-4680][SPARK-30115][SQL] Improve limit only query
(#405)
discard ba18f5e8b0e [CARMEL-4352] forbid to set
spark.sql.autoBroadcastJoinThreshold (#402)
discard f1af811d46a [CARMEL-4537] Manage and expose broadcast info in spark
driver (#378)
discard b12bb8a36b3 [CARMEL-4425] scala.MatchError when spilling large result
set (#394)
discard fea03cafc82 [CARMEL-4661] Shuffle client produce too many logs (#396)
discard 36138afcb8e [CARMEL-4569][DLETA] Invalidate the delta table cache when
drop table (#384)
discard db3c9f929fd CARMEL-4662 Download query state is incorrect. (#398)
discard f2c3aa915cc [CARMEL-4667] Configurable the max length of abbreviate
plan string (#400)
discard b36bb22e5e0 [CARMEL-4646] [FOLLOWUP] SQL level cache for
HiveExternalCatalog (#397)
discard 4d6b293c6aa [SPARK-34212][SQL][FOLLOWUP] Parquet vectorized reader can
read decimal fields with a larger precision (#392)
discard 97c4da845fa [CARMEL-4238][CARMEL-728] Block non-parquet/orc file
format data consuming with configuration (#362)
discard cfa5ff9ce92 [CARMEL-4415] Task result too large make the driver full
gc (#373)
discard 1bf62fe2681 [CARMEL-4510] Limit ExpressionSet size to avoid driver OOM
(#376)
discard eb5081edaa5 [CARMEL-4185] Too many rpc fail message in driver log
(#370)
discard f1d0950cf73 [CARMEL-4172] Restrict max threads of spark ui (#382)
discard e19afc29d3a [CARMEL-4653] Backport [SPARK-34129][SQL] Add table name
to LogicalRelation.simpleString
discard 719160e3999 [CARMEL-4229] Fast fail if query takes much time for
analysis (#366)
discard db6bd321204 [CARMEL-4202] Max task time check should only apply to
active stages (#381)
discard 7032cdf396c [CARMEL-4646] SQL level cache for HiveExternalCatalog
(#390)
discard 82158eb1fd9 [CARMEL-4652] Backport [SPARK-34515][SQL] Fix NPE if InSet
contains null value during getPartitionsByFilter (#391)
discard 608846639d5 [CARMEL-4011] Provide specific reason when call
cancalJobGroup (#1613) (#385)
discard 6a50ac192f3 [CARMEL-4344] add max split limitations for cartesian rdd
(#374)
discard 6f17008a6a6 [CARMEL-4208] Session may leak when open session with
wrong database (#371)
discard a9f83562f53 [CARMEL-4572] Backport[SPARK-34212][SQL] Fix incorrect
decimal reading from Parquet files
discard 6cb4aff2cd3 [CARMEL-4643] Backport [CARMEL-536][CARMEL-3295] cache
broadcast configuration(#388)
discard 3a3084be37b [CARMEL-4210] Improve UserResourceConsumption in
TaskSetManag (#351)
discard 47c6cf42a91 [CARMEL-3462][FOLLOWUP] Another lazy listing support for
spark 3.0 (#386)
discard 4701df54959 [CARMEL-4644] Backport [SPARK-24994] Add
UnwrapCastInBinaryComparison optimizer to simplify literal types (#387)
discard edce2f85357 [CARMEL-3462][FOLLOWUP] Fix failed test
discard 3e8efc70e18 [CARMEL-4517] Possible memory leak for ResultTask (#377)
discard 0d36230d89f [CARMEL-4637] Backport [SPARK-32788] non-partitioned table
scan should not have partition filter (#380)
discard 71e0dd735be [CARMEL-4214] Fix cancel/closed query marked as failed in
query log (#349)
discard 18575aa861d [CARMEL-4112] Spark may launch too many jobs for a query
with a limit… (#346)
discard d96d144649b [CARMEL-4258] Add max split number per skewed partition
(#344)
discard 00fcfa90090 [CARMEL-3462] Another lazy listing support for spark 3.0
(#365)
discard 18898f5c132 [CARMEL-4598] Refine Context Cleaner to improve the clean
performance (#369)
discard fd647741c63 [CARMEL-4349] Configurations set via JDBC connection
string cannot be tracked (#367)
discard 1620e07eb8f CARMEL-4512] Add session and query info when expose top
resource usage (#368)
discard d0c44dea518 [CARMEL-4339] Revert "[SPARK-31147][SQL] Forbid CHAR type
in non-Hive-Serde tables" (#317)
discard e57e72ec371 [CARMEL-4620] Backport [SPARK-32717] Add a AQEOptimizer
for AdaptiveS… (#364)
discard 37617b6ddc7 [CARMEL-4616] Upgrade Avro to 1.10.1-carmel1-SNAPSHOT
(#363)
discard d1265496cdc [CARMEL-2402] Task thread/FileScan threads are hung for
ever for waiting for BlockInfoManager write lock (#262)
discard 6590b1c3af8 [CARMEL-4497] Backport [SPARK-33612] and [SPARK-33621]
(#359)
discard 827e10d5139 [CARMEL-4614] Infer filters from DPP (#361)
discard 0527452cbb1 [SPARK-33306] Timezone is needed when cast date to string
(#360)
discard e59d53445d5 [CARMEL-4449] Refactor code for compact table command
(#358)
discard 826bd65706a [CARMEL-4592] Prevent user from repairing a table with too
many partitions (#353)
discard c30cd21209b [CARMEL-4560] Support creating delta table on existing
data (#352)
discard b29decc56cc [CARMEL-3675][FOLLOW-UP] Add metric of total task number
and shuffle read size (#348)
discard 87b7510a0f8 [CARMEL-4287][CARMEL-4288] Detect Join/Window function
data skew
discard e6b5a471595 [CARMEL-4604] Backport [SPARK-34342] Format DateLiteral
and TimestampLiteral toString
discard 1e79ea819ad [CARMEL-4333] Runtime filter for spark 3.0 (#354)
discard 87f44a2cc51 [CARMEL-636] Invalid url viewfs in Hive UDF (#333)
discard dff3ed9623f [CARMEL-2543][FOLLOW-UP] Fix memory leak (#345)
discard c2a0bd41909 [CARMEL-3594][CARMEL-3652][CARMEL-4519] Improve broadcast
stability (#330)
discard b7c88987de3 [CARMEL-3279] Add config to set max result size in session
level (#332)
discard 735aa81fb34 [CARMEL-4571] Backport [SPARK-32817] DPP throws error when
broadcast side is empty (#340)
discard ff6ce3254a7 [CARMEL-4568] Backport [SPARK-26345] Parquet support
Column indexes (#339)
discard d455863f63f [CARMEL-4308] Another support Dynamic Partition Pruning in
Adaptive Execution (#334)
discard d52e9ef8ce3 [CARMEL-2631] Add root cause information to query log when
query error (#338)
discard df1ee2abbad [CARMEL-4387] Fix toDnf java.lang.StackOverflowError (#336)
discard 494e5609cd8 [CARMEL-3278] Fix AvroTypeException issue (#331)
discard f1884e0e406 [CARMEL-2814] enhance log to find sql with stage easily
(#328)
discard a23ae52808f [CARMEL-4225][SPARK-32680][SQL][3.0] Don't Preprocess V2
CTAS with Unresolved Query (#329)
discard dcf11fe7dc2 [CARMEL-4463] Backport [SPARK-32816] Planner error when
aggregating multiple distinct DECIMAL columns (#313)
discard a74906b91b4 [CARMEL-4499] Backport [SPARK-28067][SPARK-32018] Fix
UnsafeRow set overflowed decimal (#325)
discard dba6e1d5986 [CARMEL-4470][SPARK-29166][FOLLOWUP] Correct the max total
file limitation checker (#320)
discard edc0c5c4eb1 [CARMEL-3913] qualify row_number() over [window] = 1 can
stop early (#240)
discard f0393d552cb [CARMEL-4490] Backport [SPARK-34119] Keep necessary stats
after partition pruning (#323)
discard 8daa7907e04 [CARMEL-4487] Backport [SPARK-32761] Planner error when
aggregating multiple distinct Constant columns (#322)
discard 0e51280217a [CARMEL-4477] Backport [SPARK-34081][SQL] Only pushdown
LeftSemi/LeftAnti over Aggregate if join can be planned as broadcast join
discard 615eaf416a2 [CARMEL-4475] Backport [SPARK-34031][SQL] Union operator
missing rowCount when CBO enabled
discard f4516d551de CARMEL-4193 spark3.0 should not allow create table with
location (#254)
discard 06e8e39e73b [CARMEL-4468] Backport SPARK-34110 Upgrade Zookeeper to
3.6.2 (#314)
discard 0f4c4adb547 [CARMEL-4351][DELTA] Delta UI collapse tables (#312)
discard 9646485df9d [CARMEL-4455] Backport [SPARK-33428] Conv UDF use BigInt
to avoid Long value overflow
discard e7439b05e9c [CARMEL-4286][CARMEL-4380][FOLLOWUP] Simplify conditional
in predicate should consider deterministic
discard 1b3a95d55d1 [CARMEL-4409] Make subtract date backward compatible with
spark2.3 (#302)
discard 271a6a9286f [CARMEL-4419] Broadcast exchange job leak when statement
canceled (#306)
discard 5a03e5b6c60 [SPARK-33954][SQL] Some operator missing rowCount when
enable CBO (#308)
discard a38a6d57f73 [CARMEL-3685][FOLLOWUP] Add SQL rewrite suggestion in
exception message for QUALIFY keyword (#304)
discard ba19d183c2a [CARMEL-4122] Backport [SPARK-27194][SPARK-29302][SQL] Fix
commit collision in dynamic partition overwrite mode (#300)
discard 119731b9d55 [CARMEL-456] condense precision for decimal expression to
improve aggregate performance (#298)
discard 111377cce18 [CARMEL-2758][CARMEL-3256] Adjust the max file number when
AE enabled (#301)
discard 6aa74a4c7a0 [MINOR] mute annoying logs (#303)
discard 6b1c0fb04b1 [CARMEL-1772] Make the same SQL could share the result of
running SQL… (#297)
discard 7465dbda6fb [CARMEL-4308] Support Dynamic Partition Pruning in
Adaptive Execution (#282)
discard 93f6a564bbf [CARMEL-3462] Lazy Listing Support (#284)
discard 14886e8ae6c [CARMEL-4393][SPARK-34000] fix NoSuchElementException in
stageAttemptToNumSpeculativeTasks (#299)
discard 7758f64d4e7 [CARMEL-4366] Fix the indexing pruning issue due to empty
iterator (#295)
discard 708aa8fdac5 [CARMEL-3014] Fix query failed with exception 'zip RDDs
with unequal numbers of partitions (#290)
discard 5d0914ea705 [MINOR] Mute annoying logs (#292)
discard 9d207f327eb CARMEL-4207 State in query log incorrect (#287)
discard f4ebeb35fa7 CARMEL-4154 spark3.0 have no queue info in queryLog
exception (#294)
discard e567be3abb3 [CARMEL-1626] view output resolved from sql text instead
of dataType stored in view definition (#1023) (#293)
discard e2faf1a56c1 [CARMEL-4286][CARMEL-4380] Backport SPARK-33910
Simplify/Optimize conditional expressions (#291)
discard 7811c34d1f2 [CARMEL-3356][FOLLOW-UP] Order by may introduce additional
shuffle (#289)
discard 195104cfce1 [CARMEL-2475] Long running query in cac queue, table
partitioned by Id (#278)
discard 46b5e49dc65 [CARMEL-4348] UnsupportedOperationException in Spark 3.0
(#288)
discard ed36c361de7 [CARMEL-3836][FOLLOW-UP] Bucket table scan
outputpartitioning may be incorrect (#286)
discard 69e139edb5e [CARMEL-4175][DELTA] Add a conf to allow creating V2Table
with sort columns (#285)
discard 85a9b9d861e [CARMEL-4276][FOLLOWUP] Refactor the ResolveInsertInto to
support user specified columns (#283)
discard 5bad04ad89e [CARMEL-4207] State in query log incorrect (#247)
discard 094b6339e6d [CARMEL-4323] AQE should work for basic
insert/update/delete operations (#280)
discard a705a31f0af [CARMEL-4333] Runtime filter (#275)
discard 337a1e225bb [CARMEL-4321] Backport [SPARK-33273] Fix a race condition
in subquery execution
discard 8f966305bd5 [CARMEL-4317] Backport [SPARK-33582][SPARK-33581] Hive
Metastore support filter by not-equals (#277)
discard 7c1889d9cee [CARMEL-4277] Fix the incorrect error message for scan
limit (#270)
discard 0df38c51b2e [CARMEL-4276][DELTA] ResolveInsertInto should consider the
specified columns (#274)
discard d1c4d0e90a8 [CARMEL-4149] Fix make offer issues and add more log (#271)
discard 48d8522e54d [CARMEL-4273]Always call alterTableStats to create a hive
metastore e… (#268)
discard e2043ac9fb6 [SPARK-32792][SQL] Improve InSet filter pushdown (#273)
discard dc78eae7870 [CARMEL-3375] Introduce
spark.sql.arithmeticOperations.failOnOverFlow (#142)
discard 645184b3a25 [CARMEL-4270][DELTA] Add SkipOptimizingSubquery node for
Delta command (#269)
discard 38d10cb4e84 [CARMEL-4237][FOLLOWUP] Fix bug in rest api for refresh
metadata (#267)
discard 6448af7ae22 [CARMEL-4109][DELTA] Assignment should only be resolved
with target table (#266)
discard 591d04e57f2 [CARMEL-4251] Backport [SPARK-33617] Add default
parallelism configuration for Spark SQL queries (#265)
discard d524c259292 [CARMEL-1376][CARMEL-4216] Limit the max numbers of tasks
that one stage could generat (#252)
discard 117ea1d2135 [CARMEL-4136] backport [SPARK-33183][SPARK-33472] Fix
Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts
(#261)
discard 0d5550d7879 [CARMEL-2353] Read broadcast take too much time (#256)
discard 66474aebed3 [CARMEL-3719] OOM in sort merge join (#125)
discard d8129bb9415 [CARMEL-4237][FOLLOWUP] Fix bug in rest api for refresh
metadata (#263)
discard e7546dfb5f2 [CARMEL-3965] Add another max result size threshold for
non-spillable result (#260)
discard 87e425f1c7e [CARMEL-3130] Better fetch fail handling and
MapOutputTracker improvement (#257)
discard d0d0195497a [CARMEL-4239] Backport [SPARK-32276] Remove redundant
sorts before re… (#259)
discard fc75a3e048f [CARMEL-2996] check if file exists earlier before response
in external shuffle service to decrease channel (#182)
discard 41cc3437456 [CARMEL-4237] Fix bug in rest api for refresh metadata
(#258)
discard e40f92954ff [CARMEL-2976] More logs to track shuffle output
information (#255)
discard 9315340bcd1 [SPARK-33026] Add numRows to metric of
BroadcastExchangeExec (#158)
discard fc6835d0ac7 [CARMEL-3831] Enable kill executor in UI (#170)
discard 6870e40de02 catch hdfs token exception (#115)
discard b63b10f1bf9 [CARMEL-3490] Provide json support for filter function
(#253)
discard 8ab5e748ae7 [CARMEL-4014] Fix query failed with exception 'zip RDDs
with unequal numbers of partitions' (#250)
discard 4b3d3aaa55c [CARMEL-2462] Prevent user from dropping a table with too
many partitions (#139)
discard cb2c4ba23a1 [CARMEL-1379][FOLLOW-UP] Remove session after session
closed (#246)
discard 342e7c215a1 [CARMEL-3245] More general Skew Join pattern match (#230)
discard d45cf1eb4cc [CARMEL-3356] Support some AE optimizations even it may
introduce new… (#95)
discard 1e495e73c56 [CARMEL-4213][SPARK-30768][SQL] Constraints inferred from
inequality attributes (#249)
discard 323037f89b9 [CARMEL-2445][FOLLOWUP] Remove duplicate setNumOutputRows
(#245)
discard 7c99bf18e95 [CARMEL-3282] admin readonly (#244)
discard 8474661370d [CARMEL-4058] Workload Circuit Breaker - Block user if
task max time … (#239)
discard 02f13d78bde [CARMEL-2543] Do not submit task on empty partitions (#202)
discard d1f6f684028 [CARMEL-4003] Configure query audit log with log4j (#223)
discard 23d6f975e93 [CARMEL-3438] Add expansion check in SortMergeJoinExec
(#242)
discard 0a5fa4f44f7 [CARMEL-4144] Backport
[SPARK-32879][SPARK-33140][SPARK-32056]SPARK-33432][SPARK-33389] (#229)
discard 0b96aced21f [CARMEL-3694] compressionFactor takes effect with stats
(#227)
discard c00e06d8d20 [CARMEL-4194] Backport [SPARK-33477][SQL] Hive Metastore
support filter by date type (#241)
discard 41023b6c961 [CARMEL-1379][FOLLOW-UP] Fix download NPE (#238)
discard fb189a9b0f6 [CARMEL-4017] Restrict on total broadcast size per query
(#232)
discard a374aa2b4b6 CARMEL-3905 too many RPC calls to hadoop nn when reading
index (#1611) (#237)
discard 553afb1ce85 [CARMEL-3728] Check memory in TaskReaper Support (#128)
discard 68d7eaddb36 [CARMEL-3821] Expose basic queue usage info to zeta (#224)
discard 5bda62abcda [CARMEL-3888] Zombie tasks can not be cancelled and
killed, those tasks might consume and waste system resources (#225)
discard 227cb39833f [CARMEL-3765] Check BDP 'Top Resource Usage Jobs' Active
Task accuracy (#233)
discard 8115a6737fb [CARMEL-4140] Implement built-in LikeAny and LikeAll (#213)
discard 644109039fb [CARMEL-4138] Port latest download to 3.0 (#210)
discard 1eb1bfb1e68 [CARMEL-3638] Work queue metrics (#231)
discard 81d0a5425a6 [CARMEL-3940] Add session start/close information to query
log (#222)
discard 0a012451786 [CARMEL-4164] Fix query failed with
IllegalArgumentException after Rebucket enabled (#236)
discard e9cbab90c51 [CARMEL-3285] Forbid creating function with a static
configuration (#234)
discard 512c51b54d3 [CARMEL-2445] Upgrade Thrift to Support Modified Rows and
Query Progress (#219)
discard 9301134e4e6 [CARMEL-3784][DELTA] Update external heavy accumulators
before they entering into listener event loop (#145)
discard 7e538219ac3 [CARMEL-3673][FOLLOW-UP] remove flaky UT (#235)
discard 2a0f1bb346f [MINOR] Fix the dependency of viewpoint (#197)
discard 9b91aa2d8ed [CARMEL-3716] Use partitions instead of getPartitions to
avoid re-fetch partition (#226)
discard ea7c6086dee [CARMEL-3697] Task Summary Support in TaskScheduler (#228)
discard f65adc3385d [CARMEL-3657] Add query metrics in query log (#218)
discard 5e5702da2d9 [CARMEL-3673] Limit the max shuffle data size in a single
query (#220)
discard 02e06ae9d72 [CARMEL-2969] Auto retry on FileNotFoundException (#215)
discard 60fb774a167 [CARMEL-4133] Backport [SPARK-33473] Extend interpreted
subexpression elimination to other interpreted projections (#216)
discard caa3aac162e [CARMEL-4139] Support GROUP BY use Separate columns and
CUBE/ROLLUP
discard 9a7c9d02846 [CARMEL-4141] Backport [SPARK-32948][SQL] Optimize to_json
and from_json expression chain (#217)
discard d502a60cd5f [CARMEL-3225] Support AE skew join with Union (#214)
discard 0c2070cff9b [CARMEL-4132] Backport [SPARK-33427] Add subexpression
elimination fo… (#211)
discard 40bd447b655 [CARMEL-4137] Backport [SPARK-27421][SQL] Fix filter for
int column and value class java.lang.String when pruning partition column (#209)
discard 3a3656de53e [CARMEL-4126] Backport [SPARK-32332] Support columnar
exchanges (#208)
discard a11591bc588 [CARMEL-4123]Backport [SPARK-32430] Extend
SparkSessionExtensions to inject rules into AQE query stage preparation (#207)
discard d3d218b94d0 [CARMEL-4118] Backport [SPARK-33416][SQL] Avoid Hive
metastore stack overflow when InSet predicate have many values
discard f72d7c39f12 [CARMEL-4114] Bckport [SPARK-33458][SQL] Hive partition
pruning support Contains, StartsWith and EndsWith predicate
discard 05ac4f395ec [CARMEL-4114] Backport [SPARK-33385][SQL] Support bucket
pruning for IsNaN
discard b109b4e31bb [CARMEL-4113] Backport [SPARK-33372][SQL] Fix InSet bucket
pruning
discard 76790453bf6 [CARMEL-3836] Add config to dynamically determine whether
to apply bu… (#190)
discard 44e54647970 [CARMEL-4099] Backport [SPARK-32740] Refactor common
partitioning/distribution logic to BaseAggregateExec (#201)
discard 813f33ff865 [CARMEL-4094] Backport [SPARK-33302] Failed to push down
filters through Expand (#198)
discard 38fbcd80d17 [CARMEL-4100] Backport [SPARK-32031] Fix the wrong
references of PartialMerge/Final AggregateExpression (#200)
discard edd6f98ff2e [CARMEL-4086] Backport [SPARK-33337][SQL] Support
subexpression elimination in branches of conditional expressions
discard 55cd8d1310f [CARMEL-3917] Backport [SPARK-31253] Add metrics to AQE
shuffle reader (#193)
discard 68b8217f658 [CARMEL-3945] Extend Show Session Command to include queue
info (#189)
discard 9d777e423bf [CARMEL-4056]Backport [SPARK-33290][SQL] REFRESH TABLE
should invalid… (#192)
discard bdd337c52db [CARMEL-4035] fix for missing output for qualify (#191)
discard 738cd6af078 CARMEL-3863 only allow admin to create/rebuild index (#188)
discard c3ae3bc3787 [CARMEL-3985] Improve CoarseGrainedSchedulerBackend
performance when there are lots of events Support (#178)
discard c94b6b25d0c [CARMEL-3851] Upload not work in delta table (#187)
discard 3533fdc9298 [CARMEL-3474] BuiltIn UDF java_method() is dangerous (#173)
discard 1da3aaa1819 CARMEL-3834 make mysql-connector included in the package
(#155)
discard a8158dbf14b [CARMEL-3862][FOLLOWUP] Make HiveClientImpl share nothing
(#181)
discard 0d724433a72 [CARMEL-3853][DELTA] Rowcount for insert, update & delete
(#177)
discard ea72be6a708 [CARMEL-3513] fix flaky CompactTableSuite (#185)
discard 5df38860dd5 [CARMEL-3941] Backport [SPARK-29606][SQL] Improve
EliminateOuterJoin performance
discard b4a0e1d87de [CARMEL-3986] Avoid sending many heartbeats frequently
after recovering from cpu pressure Support (#179)
discard 6b66a100e84 [CARMEL-3494] Parquet read performance improvement (#123)
discard 4d3f133d995 [CARMEL-3983] CollectAsIterator when listing files in job
Support (#175)
discard 1bff7c00290 [CARMEL-3962]Add api to export driver thread dump (#171)
discard 372a98b5e99 [CARMEL-3963] Thread dump when driver crash due to
RpcTimeout in DAGScheduler (#172)
discard 7a2a964191a [CARMEL-3150] Integration with admin and metadata cache
notification (#21)
discard df12d154a03 [CARMEL-3862][SPARK-32118][SQL] Use fine-grained read
write lock for each database in HiveExternalCatalog (#166)
discard b471b5a4ca0 [CARMEL-3950][CARMEL-3916][CARMEL-1297] Add precheck for
data writing command before AQE (#168)
discard 46b7d5cd7b3 [CARMEL-3833] enable rule stats on UI (#163)
discard a87c1fd7d38 [CARMEL-3847] Rename temporary table name should case
insensitive (#169)
discard de02dd64cff [CARMEL-3797]Backport [SPARK-32730][SQL] Improve LeftSemi
and Existence SortMergeJoin right side buffering
discard c7db97f538c [CARMEL-3871] Missing local number files
discard 4f27c0f9d9c [CARMEL-3912] Create temporary table without using should
use parquet by default (#160)
discard 47dd50ff69d [CARMEL-3751][FOLLOWUP][DELTA] Check ACL for creating data
source V2 table (#164)
discard d3422b16208 [CARMEL-3499][FOLLOWUP] Change vCores to 1 in
ContainerRequest for hadoop limit (#159)
discard 6799b2747d2 [SPARK-33018][SQL] Fix estimate statistics issue if child
has 0 bytes (#157)
discard c7e7b8945d2 [CARMEL-3856] Backport [SPARK-32594] Insert wrong dates to
Hive tables (#156)
discard 7ca49df7414 [SPARK-32914][SQL] Avoid constructing dataType multiple
times (#151)
discard 119ba4fc71b [CARMEL-3796] Backport [SPARK-32629][SQL] Track metrics of
BitSet/OpenHashSet in full outer SHJ (#148)
discard 8f5c5745dee [CARMEL-3463] Transparent Decryption Support (#70)
discard 101a0b38c65 [CARMEL-3668] Implement the equivalent TD functions for
get the current authorized user and session user Support (#153)
discard afe897297e0 [CARMEL-1571][FOLLOWUP] Partially revert CARMEL-1571 (#154)
discard 71e4fdc658a [CARMEL-3812][SPARK-33014][SQL] Support multiple bucket
columns in DataSourceV2 table (#150)
discard 119a70f8db8 [CARMEL-3158][FOLLOW-UP] Fix metric key not found
exceeption
discard 0094014e424 [CARMEL-3572] reduce information in
SparkListenerThriftServerQueryExit event (#138)
discard a2bd457acdd [CARMEL-1571] Disable select from PATH by default (#79)
discard 6d49b429bb6 [CARMEL-1206][SPARK-29166][SQL] Add parameters to limit
the number of dynamic partitions for data source table (#102)
discard 1859606f3fd [CARMEL-3803] Backport [SPARK-32767] Bucket join should
work if spark.sql.shuffle.partitions larger than bucket number
discard b7a6b8907db [CARMEL-3791] Backport [SPARK-32705][SQL] Fix
serialization issue for EmptyHashedRelation (#147)
discard 640ad31642d [CARMEL-3790] Backport [SPARK-32649][SQL] Optimize BHJ/SHJ
inner/semi join with empty hashed relation (#146)
discard 204a64c3d21 [CARMEL-3789] Backport [SPARK-32678][SQL] Rename
EmptyHashedRelationWithAllNullKeys and simplify NAAJ generated code (#144)
discard 7b7fbffe907 [CARMEL-3170][FOLLOWUP][DELTA] Add a unit test for
creating bucket table (#143)
discard 7f3543f8bf4 [CARMEL-3170][FOLLOWUP][DELTA] Delta table metrics (#137)
discard 2dea9061189 [CARMEL-3170][FOLLOWUP] Move DeltaSQLQuerySuite to slow
test (#140)
discard c8f9b5c268e [CARMEL-3731]Backport [SPARK-32399] Support full outer
join in shuffled hash join (#136)
discard d60d7b5a32e [CARMEL-3158] HDFS cache and other remote HDFS support
changes (#117)
discard 67bf5420bfd [CARMEL-3513] Add compact table command (#22)
discard 8ba435471e0 [CARMEL-3741] Backport [SPARK-32706][SQL] Improve cast
string to decimal type (#133)
discard 8f787c07820 [CARMEL-3752] Backport [SPARK-31511] Make BytesToBytesMap
iterator() thread-safe (#135)
discard cb2b033b8fa [CARMEL-3751][DELTA] ACL for delta (#134)
discard 839062c80cb [CARMEL-3556] Backport [SPARK-32573] Anti Join Improvement
with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys (#131)
discard ac6ad14b680 [CARMEL-3729] Do not re-register if executor is recently
removed Support (#129)
discard 05035e40cb5 [CARMEL-3170][DELTA] Delta supports SparkSQL 3.0 (#130)
discard a254d0a1b5c [CARMEL-3687] view database
discard 82fdbc40049 [CARMEL-3739]Backport [SPARK-31865][SQL] Fix complex AQE
query stage … (#132)
discard 747e8dc4f72 CARML-3709 Add more info to query log (#121)
discard 4adeaed04cf [CARMEL-3727]Keep necessary stats after partition pruning
(#127)
discard f26a760a6a1 [CARMEL-3725]Add stats including row count for more
operators (#126)
discard 96e5972deb2 [CARMEL-3172] Upload Support (#62)
discard 4b05d88ffaa [CARMEL-3717] Update histogram height value if the number
of rows expanded for Join operation (#124)
discard d28531765ad [CARMEL-3698] implement deny for auth (#118)
discard 780816ab719 [CARMEL-3685] implement qualify (#112)
discard a5bfe3bf9a1 [CARMEL-3713]Backport [SPARK-32615] Fix AQE
aggregateMetrics java.util.NoSuchElementException (#122)
discard 614fa94a8a1 [CARMEL-3601] Fix remote table path based on database
location (#68)
discard 105a94e6ceb [CARMEL-3710]Backport [SPARK-31967] Loading jobs UI page
takes 40 seconds (#120)
discard a228a49d5e9 [CARMEL-3667] Show Session Command Support (#105)
discard 20617eefa67 [CARMEL-3580] Backport [CARMEL-1603]ViewPoint - RunTime
Query Glance (#59)
discard a40575297a6 [CARMEL-3666] Zombie Stage/Tasks Check Support (#104)
discard beb8f1952db [CARMEL-3700][FOLLOWUP] Add ExtendedSQLTest to
ParquetV1FilterSuite and ParquetV2FilterSuite
discard e6942fc609c [CARMEL-3362] Enable Query Log in spark3.0 (#25)
discard b6feb5a46af [CARMEL-3700] Port community spark/master/dev to support
test with sbt (#116)
discard 7716b6cbe00 [CARMEL-1599][SPARK-30069][CORE][YARN] Clean up
non-shuffle disk bloc… (#80)
discard a2ad88b36c3 add executor id and detail info into task thread name
(#111)
discard 6c46f9a8d2d [MINOR] Fix sbt build missing httpclient issue: (#107)
discard 92bbafdda56 [CARMEL-3671][FOLLOWUP] Make spark.sql.hive.metastore.jars
support hdfs path (#113)
discard 844898b6ec4 [CARMEL-3602][FOLLOWUP]HiveThriftServer kerberos login
should use the… (#109)
discard b2edbe6f423 [CARMEL-3671] Make spark.sql.hive.metastore.jars support
hdfs path (#108)
discard 3c7967e2153 [CARMEL-3682] Add missing dependency for log4j-extras
(#110)
discard 07b46333007 Revert "[CARMEL-2462] Prevent user from dropping a table
with too many partit… (#72)" (#106)
discard b519327ecb9 [CARMEL-2813] index upgrade 3.0 - index metrics improvement
discard 0c18c1a1a71 [CARMEL-3645] Scan size limit support (#91)
discard 7aa305b9f62 [CARMEL-2462] Prevent user from dropping a table with too
many partit… (#72)
discard abb9da52c46 [CARMEL-2480] Add restriction on the bucket number of a
DDL (#71)
discard a3d02bc859e [CARMEL-1106][CARMEL-1191] Tune target shuffle partition
size to acce… (#89)
discard b7616699f13 [CARMEL-2718] Need respect case sensitive configuration
when parsing bucket information (#98)
discard 79b6fbf2334 [CARMEL-1379] User Level Resource Management Phase (#100)
discard 8182cdaf74d [CARMEL-1732] Create temporary table should not check the
database privilege (#84)
discard e5be66bc11a [CARMEL-3602] [FOLLOWUP]HiveThriftServer kerberos login
should use the value from spark.kerberos.keytab (#101)
discard 07ca2ae2e4b [CARMEL-3343] Clear the physical plan after the stage it
belongs to finished (#92)
discard bb99536e637 [CARMEL-3236] Prevent a skewed join from producing too
many splits (#97)
discard 1c1bd059ac3 [CARMEL-667] Bucket pruning may fail to apply for IN
clause (#99)
discard d58a34d0b0a [NIMOR] Fix sbt build issue: (#56)
discard 49eeb1471ef [MINOR] Fix HiveThriftServer2 NPE issue (#94)
discard 8ebbf539e0e [CARMEL-1199] Optimize executeTake() in SparkPlan when
submit multiple jobs and but progress slowly
discard 5d890840887 [CARMEL-3339] Support optimize skewed join with only one
query stage (#90)
discard 2768742375f [CARMEL-3602][FOLLOWUP] HiveThriftServer kerberos login
should use the value from spark conf (#93)
discard b1be56ce184 [CARMEL-2593] Make RowEncoder checkOverflow configurable
(#85)
discard f42b52b7c1a [CARMEL-3525] support spillable large result set (#31)
discard ec49233a9c9 [CARMEL-3166] dpp to support AE and enhancement (#77)
discard a3a9dd7f34a [CARMEL-1433] Add max expansion rate for stage with
shuffle write (#87)
discard 7902bc35276 [CARMEL-1631] Make executor metrics sources configurable
(#73)
discard b3dd4460754 [CARMEL-655] Limit return count for a single query to
avoid driver OOM (#46)
discard 2cc8578ccb0 [CARMEL-1661] backport and enhance
[SPARK-14922][SPARK-17732][SPARK-23866] Support partition filter in ALTER TABLE
DROP PARTITION and batch dropping PARTITIONS (#76)
discard 6613ceb8e63 [CARMEL-3590] zeta support in viewpoint server (#64)
discard b74dc7adbf2 [CARMEL-3545] Shuffle memory usage improvement - Fix NPE
issue (#88)
discard 8c22d1d6635 [CARMEL-3157] Index upgrade to 3.0 (#52)
discard 40789c4917a [CARMEL-3152] driver scheduling enhancement (#58)
discard e1ffe931f91 [CARMEL-2335] Add new operator InsertIntoDataSource for
delta tables (#78)
discard 2b96b3fafed [CARMEL-2897] API for adding Spark listener to custom a
queue (#86)
discard a889b0f143a [SPARK-32663][CORE] Avoid individual closing of pooled
TransportClient (#82)
discard 3a22daec0cb CARMEL-3571 Support ViewPointServer in Spark Command
Builder (#54)
discard 861f5043d7f [CARMEL-3620][SPARK-32659][SQL] Fix the data issue when
pruning DPP on non-atomic type (#75)
discard a923d889541 [CARMEL-1298][SQL] Set CREATE TABLE LIKE command file
format configurable (#74)
discard 45c0f60deb1 [CARMEL-3564] Disable LOAD DATA INPATH & INSERT INTO PATH
discard ac70e7b2d08 [CARMEL-3523] Auto repartition when writing to a table
discard 474ac1cf34c [CARMEL-3167] download support (#53)
discard 9484b60c50d [CARMEL-484][FOLLOW-UP] Add more metrics (#66)
discard 36a6446f08b [CARMEL-3548] Shuffle OOM error handling enhancement and
slow rpc monitoring (#37)
discard 258a5572ef5 [CARMEL-3602] HiveThriftServer kerberos login should use
the value from spark.kerberos.keytab (#69)
discard 49d71a6b917 [CARMEL-3359][SPARK-31377][SPARK-31389][SPARK-32421][SQL]
Add code-gen for shuffled hash join (#65)
discard 8d0dea0d9b8 [CARMEL-640] add max memory limit for result tasks (#45)
discard 7c460f1fea6 [CARMEL-3535] Create temporary table like and analyze a
temporary table (#61)
discard 232343ed176 [CARMEL-3592][SPARK-32420] Add handling for unique key in
non-codegen hash join
discard bfc42d64f50 [CARMEL-3586][SPARK-32383] Preserve hash join (BHJ and
SHJ) stream side ordering
discard 4bcf1783584 [CARMEL-3545] Shuffle memory usage improvement (#33)
discard d4d52c303b2 [CARMEL-484] Collect and expose metrics in
HiveExternalCatalog to monitor the performance of HiveMetastore (#43)
discard 886badefe13 [CARMEL-3529] Backport [SPARK-32437] Improve MapStatus
deserialization speed with RoaringBitmap 0.9.0 (#26)
discard fabbde4ab56 [CARMEL-3524] Create ReadWriteLock for each database in
HiveExternalC… (#24)
discard c173b47953c [CARMEL-3374] Support recursive SQL query
discard 89958d571e3 [CARMEL-3574][SPARK-32330] Preserve shuffled hash join
build side partitioning
discard 15a9ffd1ae9 [CARMEL-3579] Backport SPARK-27142 Provide REST API for
SQL information
discard 49f199f5e8a [CARMEL-3372] Support column list specification in insert
into statement
discard 3024e43a5cf [CARMEL-3555][SPARK-32290] NotInSubquery SingleColumn
Optimize
discard cebd93a2280 [CARMEL-3510][CARMEL-3511] Try to replace
ShuffleExchangeExec with CoalesceExec or RebucketingExec
discard c1a98863dec [CARMEL-3573][SPARK-31869] BroadcastHashJoinExec can
utilize the build side for its output partitioning
discard 1d832f298be [CARMEL-3568][SPARK-31719] Refactor JoinSelection
discard 39376528ea5 [CARMEL-3431] write to a bucket table
discard c795881efe1 [CARMEL-3433] Support bucket join if bucket columns are
subset of join keys
discard 3e50a8a7320 [CARMEL-3558] Possible deadlock for multi-thread scan when
task is killed (#38)
discard 2db2ee97711 [CARMEL-3569][SPARK-30842] Adjust abstraction structure
for join operators (#44)
discard 174366c085a [CARMEL-3553] Add upload and download api to thrift
protocol
discard b1fc9069ff7 CARMEL-3537 add ut for spark-26859 (#30)
discard 129e30de377 [CARMEL-3519] Viewpoint Support (#27)
discard 613513800c5 [CARMEL-3560] Increase retry times to avoid flaky test
discard 0ceddbf4c20 [CARMEL-3559][SPARK-25154] Support NOT IN sub-queries
inside nested OR conditions
discard 7663d1a9477 [CARMEL-3451][SPARK-31705][SPARK-28169][SPARK-32302]
Extract additional OR clause from Join condition and push down to query
discard 2a73041682e [CARMEL-3533] Make the Column Batch number configurable in
VectorizedParquetRecordReader
discard 13d2f2e0923 [CARMEL-3532] Enable multi-thread file scan
discard e1ee4351a3f [CARMEL-3536] Ignore HiveExternalCatalogVersionsSuite (#29)
discard ad6d8000a6a Backport [SPARK-30616][SQL] Introduce TTL config option
for SQL Metadata Cache (#14)
discard 5d43bfee5fd [SPARK-30525][SQL] HiveTableScanExec do not need to prune
partitions again after pushing down to SessionCatalog for partition pruning
discard 73824be915e [SPARK-32302][SQL] Partially push down disjunctive
predicates through Join/Partitions
discard 9059eafeb63 [SPARK-28169][SQL] Convert scan predicate condition to CNF
discard 57f1613366b [SPARK-31705][SQL][FOLLOWUP] Avoid the unnecessary CNF
computation for full-outer joins
discard 1fa88fcdc61 [SPARK-31705][SQL] Push more possible predicates through
Join via CNF conversion
discard 9a229155919 [CARMEL-3477] Backport [SPARK-31047][SQL] Improve file
listing for ViewFileSystem (#15)
discard 104a62bb53a [CARMEL-3381] Consolidate avro format support in Hermes
(#19)
discard 2ca1fbffa98 [SPARK-31793][SQL] Reduce the memory usage in file scan
location metadata (#8)
discard dcd6c6a409e [CARMEL-3500] Catch potential exception for renew kerberos
ticket (#18)
discard 88e9d8b7440 [SPARK-29231][SQL] Constraints should be inferred from
cast equality constraint (#13)
discard e933ef6b02a [CARMEL-3499] Change vCores to 1 in ContainerRequest for
hadoop limit (#17)
discard 171868b0162 [CARMEL-3154] authorization (#4)
discard 81d384a4c40 [SPARK-30724][SQL] Support 'LIKE ANY' and 'LIKE ALL'
operators (#5)
discard 91f1a3acd0e [SPARK-31220][SQL] repartition obeys initialPartitionNum
when adaptiveExecutionEnabled (#7)
discard adbb0533a07 [SPARK-28481][SQL] More expressions should extend
NullIntolerant (#9)
discard c77e263d3e9 [CARMEL-2691] Support % as well in the pattern spec for
show like table 'xxx' (#10)
discard 1930e50cfff [CARMEL-2691] Support % as well in the pattern spec for
show like table 'xxx' (#6)
discard 3e23e3d24e8 [CARMEL-3156][SPARK-32064][SQL] Supporting create
temporary table (#3)
discard f6570569dd3 Merge pull request #2 from xuluan/am-onyarn
discard d1515a1c88d register thrift server to zookeeper
discard 42beda25d54 ignore some test cases to enable ci
discard 753245ec22c Delete 1.txt
discard dc955ad7e6f ignore some ut
discard cfa382f53fe test
discard 5c309cc3e83 [CARMEL-3186] Run spark thrift server in yarn cluster mode
for spark3
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
