Hi Holden, This is the digest for commits merged between *June 3 and June 16.* The commits you mentioned would be included in the future digests.
Cheers, Xingbo On Tue, Jul 21, 2020 at 11:13 AM Holden Karau <hol...@pigscanfly.ca> wrote: > I'd also add [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are > being shutdown & > > [SPARK-21040][CORE] Speculate tasks which are running on decommission > executors two of the PRs merged after the decommissioning SPIP. > > On Tue, Jul 21, 2020 at 10:53 AM Xingbo Jiang <jiangxb1...@gmail.com> > wrote: > >> Hi all, >> >> This is the bi-weekly Apache Spark digest from the Databricks OSS team. >> For each API/configuration/behavior change, an *[API] *tag is added in >> the title. >> >> CORE >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#70spark-31923core-ignore-internal-accumulators-that-use-unrecognized-types-rather-than-crashing-63--5>[3.0][SPARK-31923][CORE] >> Ignore internal accumulators that use unrecognized types rather than >> crashing (+63, -5)> >> <https://github.com/apache/spark/commit/b333ed0c4a5733a9c36ad79de1d4c13c6cf3c5d4> >> >> A user may name his accumulators using the internal.metrics. prefix, so >> that Spark treats them as internal accumulators and hides them from UI. We >> should make JsonProtocol.accumValueToJson more robust and let it ignore >> internal accumulators that use unrecognized types. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#api80spark-31486core-sparksubmitwaitappcompletion-flag-to-control-spark-submit-exit-in-standalone-cluster-mode-88--26>[API][3.1][SPARK-31486][CORE] >> spark.submit.waitAppCompletion flag to control spark-submit exit in >> Standalone Cluster Mode (+88, -26)> >> <https://github.com/apache/spark/commit/6befb2d8bdc5743d0333f4839cf301af165582ce> >> >> This PR implements an application wait mechanism that allows spark-submit to >> wait until the application finishes in Standalone mode. This will delay the >> exit of spark-submit JVM until the job is completed. This implementation >> will keep monitoring the application until it is either finished, failed, >> or killed. This will be controlled via the following conf: >> >> - >> >> spark.standalone.submit.waitAppCompletion (Default: false) >> >> In standalone cluster mode, controls whether the client waits to exit >> until the application completes. If set to true, the client process >> will stay alive polling the driver's status. Otherwise, the client process >> will exit after submission. >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#sql> >> SQL >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#71spark-31220sql-repartition-obeys-initialpartitionnum-when-adaptiveexecutionenabled-27--12>[3.0][SPARK-31220][SQL] >> repartition obeys initialPartitionNum when adaptiveExecutionEnabled (+27, >> -12)> >> <https://github.com/apache/spark/commit/1d1eacde9d1b6fb75a20e4b909d221e70ad737db> >> >> AQE and non-AQE use different configs to set the initial shuffle >> partition number. This PR fixes repartition/DISTRIBUTE BY so that it >> also uses the AQE config >> spark.sql.adaptive.coalescePartitions.initialPartitionNum to set the >> initial shuffle partition number if AQE is enabled. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#70spark-31867sqlfollowup-check-result-differences-for-datetime-formatting-51--8>[3.0][SPARK-31867][SQL][FOLLOWUP] >> Check result differences for datetime formatting (+51, -8)> >> <https://github.com/apache/spark/commit/fc6af9d900ec6f6a1cbe8f987857a69e6ef600d1> >> >> Spark should throw SparkUpgradeException when getting DateTimeException for >> datetime formatting in the EXCEPTION legacy Time Parser Policy. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#api70spark-31879spark-31892sql-disable-week-based-pattern-letters-in-datetime-parsingformatting-1421--171-102--48>[API][3.0][SPARK-31879][SPARK-31892][SQL] >> Disable week-based pattern letters in datetime parsing/formatting (+1421, >> -171)> >> <https://github.com/apache/spark/commit/9d5b5d0a5849ac329bbae26d9884d8843d8a8571> >> (+102, >> -48)> >> <https://github.com/apache/spark/commit/afe95bd9ad7a07c49deecf05f0a1000bb8f80caa> >> >> Week-based pattern letters have very weird behaviors during datetime >> parsing in Spark 2.4, and it's very hard to simulate the legacy behaviors >> with the new API. For formatting, the new API makes the start-of-week >> localized, and it's not possible to keep the legacy behaviors. Since the >> week-based fields are rarely used, we disable week-based pattern letters in >> both parsing and formatting. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#70spark-31896sql-handle-am-pm-timestamp-parsing-when-hour-is-missing-39--3>[3.0][SPARK-31896][SQL] >> Handle am-pm timestamp parsing when hour is missing (+39, -3)> >> <https://github.com/apache/spark/commit/afcc14c6d27f9e0bd113e0d86b64dc6fa4eed551> >> >> This PR sets the hour field to 0 or 12 when the AMPM_OF_DAY field is AM >> or PM during datetime parsing, to keep the behavior the same as Spark 2.4. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#api80spark-31830sql-consistent-error-handling-for-datetime-formatting-and-parsing-functions-126--580>[API][3.1][SPARK-31830][SQL] >> Consistent error handling for datetime formatting and parsing functions >> (+126, -580)> >> <https://github.com/apache/spark/commit/6a424b93e5bdb79b1f1310cf48bd034397779e14> >> >> When parsing/formatting datetime values, it's better to fail fast if the >> pattern string is invalid, instead of returning null for each input record. >> The formatting functions such as date_format already do it, this PR >> applies the fail-fast behavior to parsing functions: from_unixtime, >> unix_timestamp,to_unix_timestamp, to_timestamp and to_date. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#80spark-31910sql-enable-java-8-time-api-in-thrift-server-23--0>[3.1][SPARK-31910][SQL] >> Enable Java 8 time API in Thrift server (+23, -0)> >> <https://github.com/apache/spark/commit/2c9988eaf31b7ebd97f2c2904ed7ee531eff0d20> >> >> This PR enables Java 8 time API in thriftserver, so that we use the >> session timezone more consistently. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#55spark-31935sql-hadoop-file-system-config-should-be-effective-in-data-source-options-52--7>[2.4][SPARK-31935][SQL] >> Hadoop file system config should be effective in data source options (+52, >> -7)> >> <https://github.com/apache/spark/commit/f3771c6b47d0b3aef10b86586289a1f675c7cfe2> >> >> This PR fixes a bug that the hadoop configs in read/write options are not >> respected in data source V1. >> [API][2.4][SPARK-31968][SQL] Duplicate partition columns check when >> writing data (+12, -1)> >> <https://github.com/apache/spark/commit/a4ea599b1b9b8ebaae0100b54e6ac1d7576c6d8c> >> >> Add a check for duplicate partition columns when writing built-in file >> sources. After the change, when the DataFrame has duplicate partition >> columns, the users get an AnalysisException when writing it. Previously, >> the writing would succeed, but reading the files with duplicate columns >> will fail. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#api71spark-26905sql-add-type-in-the-ansi-non-reserved-list-2--0>[API][3.0][SPARK-26905][SQL] >> Add TYPE in the ANSI non-reserved list (+2, -0)> >> <https://github.com/apache/spark/commit/e14029b18df10db5094f8abf8b9874dbc9186b4e> >> >> Add TYPE in the ANSI non-reserved list to follow the ANSI/SQL standard. >> The change impacts the behavior only when ANSI mode is on >> (spark.sql.ansi.enabled=true) >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#api71spark-26905sql-follow-the-sql2016-reserved-keywords-429--5>[API][3.0][SPARK-26905][SQL] >> Follow the SQL:2016 reserved keywords (+429, -5)> >> <https://github.com/apache/spark/commit/3698a14204dd861ea3ee3c14aa923123b52caba1> >> >> Move keywords ANTI, SEMI, and MINUS from reserved to non-reserved to >> comply with the ANSI/SQL standard. The change impacts the behavior only >> when ANSI mode is on (spark.sql.ansi.enabled=true) >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#api70spark-31939sqltest-java11-fix-parsing-day-of-year-when-year-field-pattern-is-missing-465--3>[API][3.0][SPARK-31939][SQL][TEST-JAVA11] >> Fix Parsing day of year when year field pattern is missing (+465, -3)> >> <https://github.com/apache/spark/commit/22dda6e18e91c6db6fa8ff9fafaafe09a79db4ea> >> >> When a datetime pattern does not contain a year field (ie. 'yyyy') but >> contains the day of year field (ie. 'DD'), Spark should still be able to >> respect the datetime pattern and parse the constants. >> >> Before the change: >> >> spark-sql> select to_timestamp('31', 'DD'); >> 1970-01-01 00:00:00 >> spark-sql> select to_timestamp('31 30', 'DD dd'); >> 1970-01-30 00:00:00 >> >> After the change: >> >> spark-sql> select to_timestamp('31', 'DD'); >> 1970-01-31 00:00:00 >> spark-sql> select to_timestamp('31 30', 'DD dd'); >> NULL >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#70spark-31956sql-do-not-fail-if-there-is-no-ambiguous-self-join-7--2>[3.0][SPARK-31956][SQL] >> Do not fail if there is no ambiguous self join (+7, -2)> >> <https://github.com/apache/spark/commit/c40051932290db3a63f80324900a116019b1e589> >> >> df("col").as("name") is not a column reference anymore, and should not >> have the special column metadata that is used to identify the root >> attribute (e.g., Dataset ID and col position). This PR fixes the >> corresponding regression that could cause a DataFrame could fail even when >> there is no ambiguous self-join. Below is an example, >> >> val joined = df.join(spark.range(1)).select($"a") >> joined.select(joined("a").alias("x"), sum(joined("a")).over(w)) >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#70spark-31958sql-normalize-special-floating-numbers-in-subquery-18--4>[3.0][SPARK-31958][SQL] >> normalize special floating numbers in subquery (+18, -4)> >> <https://github.com/apache/spark/commit/6fb9c80da129d0b43f9ff5b8be6ce8bad992a4ed> >> >> The PR fixes a bug that special floating numbers in non-correlated >> subquery expressions are not handled, now the subquery expressions will be >> handled by OptimizeSubqueries. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#api80spark-21117sql-built-in-sql-function-support---width_bucket-431--30>[API][3.1][SPARK-21117][SQL] >> Built-in SQL Function Support - WIDTH_BUCKET (+431, -30)> >> <https://github.com/apache/spark/commit/b1adc3deee00058cba669534aee156dc7af243dc> >> >> Add a built-in SQL function WIDTH_BUCKET, that returns the bucket number >> to which value would be assigned in an equiwidth histogram with >> num_bucket buckets, in the range min_value to max_value. Examples: >> >> > SELECT WIDTH_BUCKET(5.3, 0.2, 10.6, 5); >> 3 >> > SELECT WIDTH_BUCKET(-2.1, 1.3, 3.4, 3); >> 0 >> > SELECT WIDTH_BUCKET(8.1, 0.0, 5.7, 4); >> 5 >> > SELECT WIDTH_BUCKET(-0.9, 5.2, 0.5, 2); >> 3 >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-27217sql-nested-column-aliasing-for-more-operators-which-can-prune-nested-column-190--10>[3.1][SPARK-27217][SQL] >> Nested column aliasing for more operators which can prune nested column >> (+190, -10)> >> <https://github.com/apache/spark/commit/43063e2db2bf7469f985f1954d8615b95cf5c578> >> >> Support nested column pruning from an Aggregate or Expand operator. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-27633sql-remove-redundant-aliases-in-nestedcolumnaliasing-43--1>[3.1][SPARK-27633][SQL] >> Remove redundant aliases in NestedColumnAliasing (+43, -1)> >> <https://github.com/apache/spark/commit/8282bbf12d4e174986a649023ce3984aae7d7755> >> >> Avoid generating redundant aliases if the parent nested field is aliased >> in the NestedColumnAliasing rule. This slightly improves the performance. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31736sql-nested-column-aliasing-for-repartitionbyexpressionjoin-197--16>[3.1][SPARK-31736][SQL] >> Nested column aliasing for RepartitionByExpression/Join (+197, -16)> >> <https://github.com/apache/spark/commit/ff89b1114319e783eb4f4187bf2583e5e21c64e4> >> >> Support nested column pruning from a RepartitionByExpression or Join >> operator. >> ML >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31925ml-summarytotaliterations-greater-than-maxiters-43--12>[3.1][SPARK-31925][ML] >> Summary.totalIterations greater than maxIters (+43, -12)> >> <https://github.com/apache/spark/commit/f83cb3cbb3ce3f22fd122bce620917bfd0699ce7> >> >> The PR fixes a correctness issue in LogisticRegression and >> LinearRegression, that the actual round of training iterations was larger >> by 1 than the specified maxIter. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31944-add-instance-weight-support-in-linearregressionsummary-56--24>[3.1][SPARK-31944] >> Add instance weight support in LinearRegressionSummary (+56, -24)> >> <https://github.com/apache/spark/commit/89c98a4c7068734e322d335cb7c9f22379ff00e8> >> >> The PR adds instance weight support in LinearRegressionSummary, instance >> weight is already supported by LinearRegression and RegressionMetrics. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#ss> >> SS >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#71spark-31593ss-remove-unnecessary-streaming-query-progress-update-58--7>[3.0][SPARK-31593][SS] >> Remove unnecessary streaming query progress update (+58, -7)> >> <https://github.com/apache/spark/commit/1e40bccf447dccad9d31bccc75d21b8fca77ba52> >> >> The PR fixes a bug that sets incorrect metrics in Structured Streaming. >> We should make a progress update every 10 seconds when a stream doesn't >> have any new data upstream. Without the fix, we zero out the input >> information but not the output information when making the progress update. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#70spark-31990ss-use-tosettoseq-in-datasetdropduplicates-3--1>[3.0][SPARK-31990][SS] >> Use toSet.toSeq in Dataset.dropDuplicates (+3, -1)> >> <https://github.com/apache/spark/commit/7f7b4dd5199e7c185aedf51fccc400c7072bed05> >> >> The PR proposes to preserve the input order of colNames for groupCols in >> Dataset.dropDuplicates, because the Streaming's state store depends on >> the groupCols order. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-24634ss-add-a-new-metric-regarding-number-of-inputs-later-than-watermark-plus-allowed-delay-94--29>[3.1][SPARK-24634][SS] >> Add a new metric regarding number of inputs later than watermark plus >> allowed delay (+94, -29)> >> <https://github.com/apache/spark/commit/84815d05503460d58b85be52421d5923474aa08b> >> >> Add a new metrics numLateInputs to count the number of inputs which are >> later than watermark ('inputs' are relative to operators). The new metrics >> will be provided both on the SparkUI - SQL Tab - query execution details >> page, and on the Streaming Query Listener. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#python> >> PYTHON >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#api70spark-31895pythonsql-support-dataframeexplainextended-str-case-to-be-consistent-with-scala-side-24--11>[API][3.0][SPARK-31895][PYTHON][SQL] >> Support DataFrame.explain(extended: str) case to be consistent with Scala >> side (+24, -11)> >> <https://github.com/apache/spark/commit/e1d52011401c1989f26b230eb8c82adc63e147e7> >> >> Improves DataFrame.explain in PySpark, so that it takes the explain mode >> string as well, which is consistent with the Scala API. >> [3.0][SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per >> the case sensitivity in grouped and cogrouped pandas UDFs (+37, -8)> >> <https://github.com/apache/spark/commit/00d06cad564d5e3e5f78a687776d02fe0695a861> >> >> The PR proposes to resolve grouping attributes separately first so it can >> be properly referred to when FlatMapGroupsInPandas and >> FlatMapCoGroupsInPandas are resolved without ambiguity. Example: >> >> from pyspark.sql.functions import *df = spark.createDataFrame([[1, 1]], >> ["column", "Score"])pandas_udf("column integer, Score float", >> PandasUDFType.GROUPED_MAP)def my_pandas_udf(pdf): >> return pdf.assign(Score=0.5) >> df.groupby('COLUMN').apply(my_pandas_udf).show() >> >> df1 = spark.createDataFrame([(1, 1)], ("column", "value"))df2 = >> spark.createDataFrame([(1, 1)], ("column", "value")) >> df1.groupby("COLUMN").cogroup( >> df2.groupby("COLUMN") >> ).applyInPandas(lambda r, l: r + l, df1.schema).show() >> >> Before: >> >> pyspark.sql.utils.AnalysisException: Reference 'COLUMN' is ambiguous, could >> be: COLUMN, COLUMN.; >> >> pyspark.sql.utils.AnalysisException: cannot resolve '`COLUMN`' given input >> columns: [COLUMN, COLUMN, value, value];; >> 'FlatMapCoGroupsInPandas ['COLUMN], ['COLUMN], <lambda>(column#9L, >> value#10L, column#13L, value#14L), [column#22L, value#23L] >> :- Project [COLUMN#9L, column#9L, value#10L] >> : +- LogicalRDD [column#9L, value#10L], false >> +- Project [COLUMN#13L, column#13L, value#14L] >> +- LogicalRDD [column#13L, value#14L], false >> >> After: >> >> +------+-----+ >> |column|Score| >> +------+-----+ >> | 1| 0.5| >> +------+-----+ >> >> +------+-----+ >> |column|value| >> +------+-----+ >> | 2| 2| >> +------+-----+ >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31945sqlpyspark-enable-cache-for-the-same-python-function-25--4>[3.1][SPARK-31945][SQL][PYSPARK] >> Enable cache for the same Python function (+25, -4)> >> <https://github.com/apache/spark/commit/032d17933b4009ed8a9d70585434ccdbf4d1d7df> >> >> This PR proposes to make PythonFunction hold Seq[Byte] instead of >> Array[Byte]. After the change, it can compare if the byte array has the >> same values. With the proposed change, the cache manager will detect the >> same function and use the cache for it if it exists. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31964python-use-pandas-is_categorical-on-arrow-category-type-conversion-2--5>[3.1][SPARK-31964][PYTHON] >> Use Pandas is_categorical on Arrow category type conversion (+2, -5)> >> <https://github.com/apache/spark/commit/b7ef5294f17d54e7d90e36a4be02e8bd67200144> >> >> When using PyArrow to convert a Pandas categorical column, use >> is_categorical instead of trying to import CategoricalDtype, because the >> former is a more stable API. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#ui> >> UI >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#70spark-31903sqlpysparkr-fix-topandas-with-arrow-enabled-to-show-metrics-in-query-ui-4--4>[3.0][SPARK-31903][SQL][PYSPARK][R] >> Fix toPandas with Arrow enabled to show metrics in Query UI (+4, -4)> >> <https://github.com/apache/spark/commit/632b5bce23c94d25712b43be83252b34ebfd3e72> >> >> In Dataset.collectAsArrowToR and Dataset.collectAsArrowToPython, since >> the code block for serveToStream is run in the separate thread, >> withAction finishes as soon as it starts the thread. As a result, it >> doesn't collect the metrics of the actual action and Query UI shows the >> plan graph without metrics. This PR fixes the issue. >> >> The affected functions are: >> >> - collect() in SparkR >> - DataFrame.toPandas() in PySpark >> >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#70spark-31886webui-fix-the-wrong-coloring-of-nodes-in-dag-viz-33--3>[3.0][SPARK-31886][WEBUI] >> Fix the wrong coloring of nodes in DAG-viz (+33, -3)> >> <https://github.com/apache/spark/commit/8ed93c9355bc2af6fe456d88aa693c8db69d0bbf> >> >> In the Job Page and Stage Page, nodes which are associated with "barrier >> mode" in the DAG-viz will be colored pale green. But, with some types of >> jobs, nodes which are not associated with the mode will also be colored. >> This PR fixes it. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-3-~-June-9,-2020#80spark-29431webui-improve-web-ui--sql-tab-visualization-with-cached-dataframes-46--0>[3.1][SPARK-29431][WEBUI] >> Improve Web UI / Sql tab visualization with cached dataframes (+46, -0)> >> <https://github.com/apache/spark/commit/e4db3b5b1742b4bdfa32937273e5d07a76cde79b> >> >> Display the query plan of cached DataFrames as well in the web UI. >> [2.4][SPARK-31967][UI] Downgrade to vis.js 4.21.0 to fix Jobs UI loading >> time regression (+49, -86)> >> <https://github.com/apache/spark/commit/f535004e14b197ceb1f2108a67b033c052d65bcb> >> >> Fix the serious perf issue in web UI by falling back vis-timeline-graph2d >> to 4.21.0. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-30119webui-support-pagination-for-streaming-tab-259--178>[3.1][SPARK-30119][WEBUI] >> Support pagination for streaming tab (+259, -178)> >> <https://github.com/apache/spark/commit/9b098f1eb91a5e9f488d573bfeea3f6bfd9b95b3> >> >> The PR adds pagination support for the streaming tab. >> >> <https://github.com/databricks/runtime/wiki/OSS-Digest-June-10-~-June-16,-2020#80spark-31642followup-fix-sorting-for-duration-column-and-make-status-column-sortable-7--6>[3.1][SPARK-31642][FOLLOWUP] >> Fix Sorting for duration column and make Status column sortable (+7, -6)> >> <https://github.com/apache/spark/commit/f5f6eee3045e90e02fc7e999f616b5a021d7c724> >> >> The PR improves the pagination support in the streaming job, by fixing >> the wrong sorting result and making Status column sortable. >> >> >> > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
