GitHub user guoxiaolongzte reopened a pull request:
https://github.com/apache/spark/pull/17580
[20269][Structured Streaming] add class 'JavaWordCountProducer' to 'provide
java word count producer'.
## What changes were proposed in this pull request?
1.run example of streaming kafka,currently missing java word count
producer,not conducive to java developers to learn and test.
add a class JavaKafkaWordCountProducer.
2.run example of JavaKafkaWordCount.I find no java word count producer.
run example of KafkaWordCount.I find have scala word count producer.
I think we should provide the corresponding example code to facilitate java
developers to learn and test.
3.My project team develops spark applications,basically with java
statements and java API.
4.test case
1)java kafka word count producer,generate data for run example of
JavaKafkaWordCount.the class of JavaKafkaWordCountProducer add by me.
spark-submit ... --class
org.apache.spark.examples.streaming.JavaKafkaWordCountProducer --jars
/home/gxl/spark/libext/kafka-clients-0.8.2.1.jar
/home/gxl/spark/examples/jars/spark-examples_2.11-2.1.0.jar localhost:9092
topic1 3 4
2)java kafka streaming example
spark-submit ... --class
org.apache.spark.examples.streaming.JavaKafkaWordCount --jars
/home/gxl/kafka/kafka_2.11-0.10.2.0/libs/kafka-clients-0.10.2.0.jar --jars
/home/gxl/spark/libext/kafka_2.11-0.8.2.1.jar
/home/gxl/spark/examples/jars/spark-examples_2.11-2.1.0.jar localhost:2181
topic1 topic1 2
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-20269
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17580
----
commit d383efba12c66addb17006dea107bb0421d50bc3
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-03-31T13:57:09Z
[SPARK-20177]Document about compression way has some little detail changes.
commit 3059013e9d2aec76def14eb314b6761bea0e7ca0
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-01T01:38:02Z
[SPARK-20177] event log add a space
commit 46bb1ad3ddd9fb55b5607ac4f20213a90186cfe9
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-05T03:16:50Z
Merge branch 'master' of https://github.com/apache/spark into SPARK-20177
commit 0efb0dd9e404229cce638fe3fb0c966276784df7
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-05T03:47:53Z
[SPARK-20218]'/applications/[app-id]/stages' in REST API,add description.
commit 0e37fdeee28e31fc97436dabd001d3c85c5a7794
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-05T05:22:54Z
[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,remove
redundant description.
commit 52641bb01e55b48bd9e8579fea217439d14c7dc7
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-07T06:24:58Z
Merge branch 'SPARK-20218'
commit d3977c9cab0722d279e3fae7aacbd4eb944c22f6
Author: éå°é¾ 10207633 <[email protected]>
Date: 2017-04-08T07:13:02Z
Merge branch 'master' of https://github.com/apache/spark
commit 1279041f17f1f5d02c9aada5fb5af9c7d58f2423
Author: asmith26 <[email protected]>
Date: 2017-04-09T06:47:23Z
[MINOR] Issue: Change "slice" vs "partition" in exception messages (and
code?)
## What changes were proposed in this pull request?
Came across the term "slice" when running some spark scala code.
Consequently, a Google search indicated that "slices" and "partitions" refer to
the same things; indeed see:
- [This issue](https://issues.apache.org/jira/browse/SPARK-1701)
- [This pull request](https://github.com/apache/spark/pull/2305)
- [This StackOverflow
answer](http://stackoverflow.com/questions/23436640/what-is-the-difference-between-an-rdd-partition-and-a-slice)
and [this
one](http://stackoverflow.com/questions/24269495/what-are-the-differences-between-slices-and-partitions-of-rdds)
Thus this pull request fixes the occurrence of slice I came accross.
Nonetheless, [it would
appear](https://github.com/apache/spark/search?utf8=%E2%9C%93&q=slice&type=)
there are still many references to "slice/slices" - thus I thought I'd raise
this Pull Request to address the issue (sorry if this is the wrong place, I'm
not too familar with raising apache issues).
## How was this patch tested?
(Not tested locally - only a minor exception message change.)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: asmith26 <[email protected]>
Closes #17565 from asmith26/master.
commit b0c7b1dcacc25f3ddf5c6d27b84a6e02122dbb73
Author: Sean Owen <[email protected]>
Date: 2017-04-09T07:44:02Z
[SPARK-19991][CORE][YARN] FileSegmentManagedBuffer performance improvement
## What changes were proposed in this pull request?
Avoid `NoSuchElementException` every time `ConfigProvider.get(val,
default)` falls back to default. This apparently causes non-trivial overhead in
at least one path, and can easily be avoided.
See https://github.com/apache/spark/pull/17329
## How was this patch tested?
Existing tests
Author: Sean Owen <[email protected]>
Closes #17567 from srowen/SPARK-19991.
commit 4c4e1c11fb326e446b2b4f668edf96876db5b187
Author: guoxiaolongzte <[email protected]>
Date: 2017-04-09T07:48:34Z
Revert "[MINOR] Issue: Change "slice" vs "partition" in exception messages
(and code?)"
This reverts commit 34fc48fb5976ede00f3f6d8c4d3eec979e4f4d7f.
commit d7ab5c15650686d63dbea568b7e0881eb804d015
Author: guoxiaolongzte <[email protected]>
Date: 2017-04-09T10:55:06Z
Merge branch 'master' of https://github.com/apache/spark
commit fe3444df5311cff2092b02e5249310153ac8a060
Author: guoxiaolongzte <[email protected]>
Date: 2017-04-09T10:56:15Z
add JavaWordCountProducer in steaming examples
commit 6f0491d4f1c5647520aaebb75152ff0a8e38e8ef
Author: guoxiaolongzte <[email protected]>
Date: 2017-04-09T12:43:14Z
modify java code style.Use 2-space indentation in general
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
