jzhuge commented on issue #23507: [SPARK-26576][SQL] Broadcast hint not applied to partitioned table URL: https://github.com/apache/spark/pull/23507#issuecomment-453879185 Thanks Sean for reporting the issue. StreamingOuterJoinSuite passed on my mac, and in my company's branch-2.3 testing (based on v2.3.2 though). However, all 4 flavors of upstream branch-2.3 tests started failing after my commit, in slightly ways: Jan 11, 2019 9:23:22 AM on amp-jenkins-worker-03 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.6/577/ Test Result <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.6/577/testReport/> (1 failure / +1) - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.left outer join with non-key condition violated <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.6/577/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/left_outer_join_with_non_key_condition_violated/> Jan 11, 2019 9:23:22 AM on amp-jenkins-worker-03 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/572/ Test Result <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/572/testReport/> (2 failures / +2) - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.windowed right outer join <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/572/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/windowed_right_outer_join/> - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.left outer join with non-key condition violated <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/572/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/left_outer_join_with_non_key_condition_violated/> Jan 11, 2019 9:24:12 AM on amp-jenkins-worker-02 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.6/547/ Test Result <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.6/547/testReport/> (3 failures / +3) - org.apache.spark.sql.kafka010.KafkaRelationSuite.default starting and ending offsets <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.6/547/testReport/junit/org.apache.spark.sql.kafka010/KafkaRelationSuite/default_starting_and_ending_offsets/> - org.apache.spark.sql.kafka010.KafkaRelationSuite.explicit offsets <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.6/547/testReport/junit/org.apache.spark.sql.kafka010/KafkaRelationSuite/explicit_offsets/> - org.apache.spark.sql.kafka010.KafkaRelationSuite.reuse same dataframe in query <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.6/547/testReport/junit/org.apache.spark.sql.kafka010/KafkaRelationSuite/reuse_same_dataframe_in_query/> Jan 11, 2019 9:24:12 AM on amp-jenkins-worker-04 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/ Test Result <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/> (58 failures / +58) - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.windowed left outer join <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/windowed_left_outer_join/> - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.windowed right outer join <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/windowed_right_outer_join/> - org.apache.spark.sql.streaming.StreamingOuterJoinSuite.left outer join with non-key condition violated <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.streaming/StreamingOuterJoinSuite/left_outer_join_with_non_key_condition_violated/> - org.apache.spark.sql.hive.client.HiveClientSuites.(It is not a test it is a sbt.testing.SuiteSelector) <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/HiveClientSuites/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/> - org.apache.spark.sql.hive.client.VersionsSuite.0: create client <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/VersionsSuite/0__create_client/> - org.apache.spark.sql.hive.client.VersionsSuite.0: createDatabase <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/VersionsSuite/0__createDatabase/> - org.apache.spark.sql.hive.client.VersionsSuite.0: createDatabase with null description <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/VersionsSuite/0__createDatabase_with_null_description/> - org.apache.spark.sql.hive.client.VersionsSuite.0: setCurrentDatabase <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/VersionsSuite/0__setCurrentDatabase/> - org.apache.spark.sql.hive.client.VersionsSuite.0: getDatabase <https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/542/testReport/junit/org.apache.spark.sql.hive.client/VersionsSuite/0__getDatabase/> Still clueless why removing case ResolvedHint in PhysicalOperation.collectProjectsAndFilters in my commit would affect StreamingOuterJoinSuite. I'd suggest reverting my commit in branch-2.3 for now in order to unblock tests and confirm it is the culprit in some way. Thanks, On Sun, Jan 13, 2019 at 6:14 AM Sean Owen <[email protected]> wrote: > I'm not 100% sure, but I think this is causing StreamingQuerySuite to fail > consistently in 2.3, but not 2.4. See failures from > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ > and drill down to things like > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.6/577/testReport/ > . Most of the subsequent build failures seem to be related to this test. > > CC @maropu <https://github.com/maropu> as this probably affects a 2.3.3 > release. > @jzhuge <https://github.com/jzhuge> @gatorsmile > <https://github.com/gatorsmile> what do you think, is it an issue, and > should we fix-forward or just revert from 2.3? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/23507#issuecomment-453833268>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ABy-pElCg5Zk5nm3rmk93znWrHd3KIhHks5vCz9YgaJpZM4Z4-eK> > . > -- John Zhuge
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
