GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/16955
[SPARK-19626]update cred using spark.yarn.credentials.updateTime
## What changes were proposed in this pull request?
update cred using spark.yarn.credentials.updateTime
## How
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/16955#discussion_r101491787
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/CredentialUpdater.scala
---
@@ -55,14 +55,10 @@ private[spark
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/16955
cc again @jerryshao
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/16955
@srowen @jerryshao Thanks for your comments. I have added some
descriptions both in the JIRA and here, please check whether OK or not.
---
If your project is set up for it, you can reply to this
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/16955
ping @srowen, would you plz verify this patch again? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/16955
@vanzin Thanks, all tests passed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/16445
[SPARK-19043][SQL]Make SparkSQLSessionManager more configurable
## What changes were proposed in this pull request?
To make SparkSQLSessionManager's background operation thread
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/16445
ping @srowen would you plz take a look at this prï¼
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/17333
[SPARK-19997] [SQL]fix proxy ugi could not get tgt to cause metastore
connecting problem
## What changes were proposed in this pull request?
Pass the real user ugi instead of proxy ugi
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17333
ping @vanzin , can you take a look at this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17333
@jerryshao ð
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
The dbs and tbls may be created on hdfs via the real userï¼so that the
proxy user may have no rights to things such as:
```
Error: java.lang.RuntimeException: Cannot create staging
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
with this creds provided by HiveCredentialProvider and configured by
`hive.metastore.kerberos.principal`, do we need to re-login with
`spark.yarn.principal` aiming to connect metastore
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
I have tested this with my kerberized hdfs and it works for me. LGTM,
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17333
see #17335 for more details, duplicated & closed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user yaooqinn closed the pull request at:
https://github.com/apache/spark/pull/17333
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/17387
[SPARK-20060][Deploy][Kerberos][Spark Shell] Obtain credentials for proxy
user before talking to hive metastore
## What changes were proposed in this pull request?
For **Spark on non
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
@jerryshao This may also fix issue SPARK-19995 and SPARK-19997 on yarn with
those apps which take `SparkSQLCLIDriver.main` as entrance, plz take a look.
@vanzin @tgravescs @mridulm @dongjoon
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
yesï¼this seem only fix the local modeï¼in standalone mode it still got
problems of hdfs token lost @tgravescs @jerryshao .
---
If your project is set up for it, you can reply to this email
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
https://issues.apache.org/jira/browse/SPARK-15754 will this patch cause
this problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
@subrotosanyal would you please help to describe
https://github.com/apache/spark/pull/13499 in detailï¼Thanks
---
If your project is set up for it, you can reply to this email and have your
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/17430
[SPARK-20096][Spark Submit][Minor]Expose the right queue name not null if
set by --conf or configure file
## What changes were proposed in this pull request?
while submit apps with -v
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17430
@srowen could please help me to verify this pr? thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108051067
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -307,7 +308,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108051074
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108055529
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108061061
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17387
cc @tgravescs, tested with a secured HDFS with standalone and works fine.
And this pr has a lot of yarn specified security arguments to rename.
cc @jerryshao plz take a look.
---
If
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108094996
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17430
ok to test. if i am in whitelist...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17430
ping @felixcheung, would you please take a look again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/17430#discussion_r108616336
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -148,6 +148,17 @@ class SparkSubmitSuite
appArgs.childArgs
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/11045
Hi @winningsix, I am interesting in your idea, and i am confusing about
the field `userName` used in your code, 1)where is it be initialized? 2) is it
used for privilege checking?
---
If your
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/11045
@winningsix Glad to hear from you. And we can only use this through Spark
ThriftServer, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9559#issuecomment-155252219
@rxin OpenHashSet replace HashSet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9559#issuecomment-155252403
@andrewor14 Thanks for your advices
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/9661
[SPARK-11583] [Core]MapStatus Using RoaringBitmap More Properly
1. test cases
1.1 sparse case: for each task 10 blocks contains data, others dont
sc.makeRDD(1 to 40950, 4095).groupBy(x=&g
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r44745422
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -154,15 +155,17 @@ private[spark] class HighlyCompressedMapStatus
private
Github user yaooqinn closed the pull request at:
https://github.com/apache/spark/pull/9559
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9559#issuecomment-156309802
OK, close this pr and see #9661
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9661#issuecomment-156327126
## test cases
sparse case: 4085 empty ---*S*
```scala
sc.makeRDD(1 to 40950, 4095).groupBy(x=>x).top(5)
```
dense case: 95 empty
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r44751632
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -154,15 +155,17 @@ private[spark] class HighlyCompressedMapStatus
private
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9661#issuecomment-156359312
For my questions in my last comment
### continuous
```scala
scala> import org.roaringbitmap._
import org.roaringbitmap._
scala> val r
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r44759326
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -154,15 +155,17 @@ private[spark] class HighlyCompressedMapStatus
private
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r45007852
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -21,24 +21,24 @@ import java.io.{EOFException, IOException
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r45007872
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -173,18 +172,15 @@ private[spark] object HighlyCompressedMapStatus
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9661#issuecomment-157253352
$ git push https://github.com/yaooqinn/spark.git mapstatus-roaring:test
Counting objects: 5581, done.
Delta compression using up to 4 threads.
Compressing
Github user yaooqinn commented on the pull request:
https://github.com/apache/spark/pull/9661#issuecomment-157300997
@davies thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user yaooqinn closed the pull request at:
https://github.com/apache/spark/pull/9661
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/20145
[SPARK-22950]Handle ChildFirstURLClassLoader's parent
## What changes were proposed in this pull request?
ChildFirstClassLoader's parent is set to null, so we can't get
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20145#discussion_r159585110
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUtilsSuite.scala ---
@@ -42,4 +47,29 @@ class HiveUtilsSuite extends QueryTest with
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20145#discussion_r159585279
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUtilsSuite.scala ---
@@ -42,4 +47,29 @@ class HiveUtilsSuite extends QueryTest with
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20145#discussion_r159587396
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUtilsSuite.scala ---
@@ -42,4 +47,41 @@ class HiveUtilsSuite extends QueryTest with
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20145
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/21290
[SPARK-24241][Submit]Do not fail fast when dynamic resource allocation
enabled with 0 executor
## What changes were proposed in this pull request?
```
~/spark-2.3.0-bin-hadoop2.7$ bin
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/21290
cc @srowen
The last code change seems tp be related to you, plz help to review, thanks
---
-
To unsubscribe, e-mail
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/21290#discussion_r187851318
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -76,6 +75,7 @@ private[deploy] class SparkSubmitArguments(args
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19840
@vanzin I am not very familiar with python part
[context.py#L191](https://github.com/yaooqinn/spark/blob/8ff5663fe9a32eae79c8ee6bc310409170a8da64/python/pyspark/context.py#L191),
so handle it at
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
@samartinucci thanks for reminding of this, i have fixed the conflicts.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
@gatorsmile would you plz take a look at this.
this pr mainly want to close HiveSessionState explicitly to delete
`hive.downloaded.resources.dir` which points to `"${system:java.io.t
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
@liufengdb its necessary to create these for `add jar` cmd
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18666
@liufengdb yes you are right. what's more, the Hive's `SessionState` is too
overstaffed for spark to create an hive client, we may only need an
`IMetaS
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20898
proxy or not, i only find such an issue with proxy
https://github.com/apache/spark/pull/20784
---
-
To unsubscribe, e-mail
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20784#discussion_r177288126
--- Diff:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
---
@@ -121,6 +134,25 @@ private[hive
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20784
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20784
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20784#discussion_r177610190
--- Diff:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
---
@@ -121,6 +123,11 @@ private[hive
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20784#discussion_r177610275
--- Diff:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
---
@@ -121,6 +123,11 @@ private[hive
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18983
besides, redirecting output here only is needed when there is an instance
of a CliSessionState, otherwise, it will be done during the SessionState
initializing in HiveClientImpl
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/20422
[SPARK-23253][Core][Shuffle]Only write shuffle temporary index file when
there is not an existing one
## What changes were proposed in this pull request?
Shuffle Index temporay file
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20422
thanks guys for reviewing. yes, this is just a minor improvement which I
guess code here seem not very logical when I was trying to do some
optimizations for my customer's heavy shuffle cas
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20422
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20422
@squito add a test for index file. plz check it again, thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/20469
[SPARK-23295][Build][Minor]Exclude Waring message when generating versions
in make-distribution.sh
## What changes were proposed in this pull request?
When we specified a wrong profile
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/20571
[SPARK-23383][Build][Minor]Make a distribution should exit with usage while
detecting wrong options
## What changes were proposed in this pull request?
```shell
./dev/make-distribution.sh
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/20571#discussion_r167440294
--- Diff: dev/make-distribution.sh ---
@@ -72,8 +76,15 @@ while (( "$#" )); do
--help)
exit_
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20571
sorry for not replying in time. the logic here is that firstly unrecognized
- -options show warning message and usage informationï¼secondly the - options
end the while loop and treat the current
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20571
OKï¼ i will push a commit very soon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20571
yesï¼ single dash means the coming ones are all maven optionsï¼they will
be handled later by mvn command. we do not parse each single option in this
while loop from then till now
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/20784
[SPARK-23639][SQL]Obtain token before init metastore client in SparkSQL CLI
## What changes were proposed in this pull request?
In SparkSQLCLI, SessionState generates before SparkContext
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20784
cc @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/20784
yarn @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/22180
[SPARK-25174][YARN]Limit the size of diagnostic message for am to
unregister itself from rm
## What changes were proposed in this pull request?
When using older versions of spark
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/22180
cc @gatorsmile @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/22180#discussion_r212161599
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -368,7 +369,11 @@ private[spark] class
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/18648
[SPARK-21428] Set IsolatedClientLoader off while using builtin Hive jars
for reusing CliSessionState
## What changes were proposed in this pull request?
Set isolated to false while using
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/18666
[SPARK-21449][SQL][Hive]Close HiveClient's SessionState to delete residual
dirs
## What changes were proposed in this pull request?
When sparkSession.stop() is called, clos
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/18668
[SPARK-21451][SQL]get `spark.hadoop.*` properties from sysProps to hiveconf
## What changes were proposed in this pull request?
get `spark.hadoop.*` properties from sysProps to
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/18668
ping @cloud-fan @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/18668#discussion_r128170401
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
---
@@ -404,6 +404,13 @@ private[spark] object HiveUtils extends Logging
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/18668#discussion_r128186557
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
---
@@ -404,6 +404,13 @@ private[spark] object HiveUtils extends Logging
Github user yaooqinn closed the pull request at:
https://github.com/apache/spark/pull/15071
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/15071
[WIP][SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec
## What changes were proposed in this pull request?
For current `BroadcastHashJoinExec`, we generate join code for
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/15071
@hvanhovell
For current `BroadcastHashJoinExec`, we generate join code for key is not
unique like this:
```
while (matches.hasnext) {
matched = matches.next
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/15071
@hvanhovell thanks very much for your suggestions. I have added my comments
to the description, and I will run benchmark later to see if it works
---
If your project is set up for it, you can
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/15071
@hvanhovell I have added a benchmark test for this, could you please help
me to review? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/15071#discussion_r79637407
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BroadcastHashJoinBenchmark.scala
---
@@ -0,0 +1,84 @@
+/*
+ * Licensed
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/15071
ping @hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/15071
@hvanhovell
I think unfixed length fields may lead to memory overlapping when
```BuildLeft```, since we are reusing the ```BufferHolder``` to avoid writing
the stream side repeatedly. In this
1 - 100 of 233 matches
Mail list logo
