[beam] 01/01: [BEAM-7461] Disalbe flaky tests
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ce09b8416c481ae8a4155931ec148e3d51773bd0 Merge: fe6891a 944646e Author: Lukasz Cwik AuthorDate: Mon Jun 17 22:58:20 2019 -0700 [BEAM-7461] Disalbe flaky tests .../java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java| 3 +++ .../beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java | 1 + 2 files changed, 4 insertions(+)
[beam] branch master updated (fe6891a -> ce09b84)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fe6891a Fixing file naming for windows (#8870) add 944646e [BEAM-7461] disalbe flaky tests due to flaky org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.checkIfAnySubscriptionExists] new ce09b84 [BEAM-7461] Disalbe flaky tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java| 3 +++ .../beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java | 1 + 2 files changed, 4 insertions(+)
[beam] branch master updated: Fixing file naming for windows (#8870)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fe6891a Fixing file naming for windows (#8870)
fe6891a is described below
commit fe6891ad0dd181b1df641e4df6671858598fac6c
Author: Pablo
AuthorDate: Tue Jun 18 07:09:46 2019 +0200
Fixing file naming for windows (#8870)
* Fixing file naming for windows
* Adding to fix.
* trailing whitespace aiaiai
---
sdks/python/apache_beam/io/fileio_test.py | 29 +++--
1 file changed, 19 insertions(+), 10 deletions(-)
diff --git a/sdks/python/apache_beam/io/fileio_test.py
b/sdks/python/apache_beam/io/fileio_test.py
index f87aa2f..75e4175 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -437,7 +437,7 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
# streaming mode.
WriteFilesTest.all_records = []
-dir = self._new_tempdir()
+dir = '%s%s' % (self._new_tempdir(), os.sep)
# Setting up the input (TestStream)
ts = TestStream().advance_watermark_to(0)
@@ -450,6 +450,10 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
ts.advance_processing_time(5)
ts.advance_watermark_to(timestamp)
+def no_colon_file_naming(*args):
+ file_name = fileio.destination_prefix_naming()(*args)
+ return file_name.replace(':', '_')
+
# The pipeline that we are testing
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
@@ -467,6 +471,7 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
_ = (res
| beam.io.fileio.WriteToFiles(path=dir,
+ file_naming=no_colon_file_naming,
max_writers_per_bundle=0)
| beam.Map(lambda fr: FileSystems.join(dir, fr.file_name))
| beam.ParDo(self.record_dofn()))
@@ -505,6 +510,10 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
.add_elements([next(input), next(input)])
.advance_watermark_to(40))
+def no_colon_file_naming(*args):
+ file_name = fileio.destination_prefix_naming()(*args)
+ return file_name.replace(':', '_')
+
with TestPipeline() as p:
_ = (p
| ts
@@ -515,7 +524,7 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
sink=lambda dest: (
WriteFilesTest.CsvSink(WriteFilesTest.CSV_HEADERS)
if dest == 'apache' else WriteFilesTest.JsonSink()),
- file_naming=fileio.destination_prefix_naming(),
+ file_naming=no_colon_file_naming,
max_writers_per_bundle=0,
))
@@ -534,25 +543,25 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
stringmatches.matches_regexp(
FileSystems.join(
dir,
-
'cncf-1970-01-01T00:00:00-1970-01-01T00:00:10--.*'
+
'cncf-1970-01-01T00_00_00-1970-01-01T00_00_10--.*'
)
),
stringmatches.matches_regexp(
FileSystems.join(
dir,
-
'cncf-1970-01-01T00:00:10-1970-01-01T00:00:20--.*'
+
'cncf-1970-01-01T00_00_10-1970-01-01T00_00_20--.*'
)
),
stringmatches.matches_regexp(
FileSystems.join(
dir,
-
'cncf-1970-01-01T00:00:20-1970-01-01T00:00:30--.*'
+
'cncf-1970-01-01T00_00_20-1970-01-01T00_00_30--.*'
)
),
stringmatches.matches_regexp(
FileSystems.join(
dir,
-
'cncf-1970-01-01T00:00:30-1970-01-01T00:00:40--.*'
+
'cncf-1970-01-01T00_00_30-1970-01-01T00_00_40--.*'
)
)
]),
@@ -562,19 +571,19 @@ class WriteFilesTest(_TestCaseWithTempDirCleanUp):
matches_all([
stringmatches.matches_regexp(FileSystems.join(
dir,
- 'apache-1970-01-01T00:00:00-1970-01-01T00:00:10--.*')
+ 'apache-1970-01-01T00_00_00-1970-01-01T00_00_10--.*')
),
stringmatches.matches_regexp(FileSystems.join(
dir,
- 'apache-1970-01-01T00:00:10-1970-01-01T00:00:20--.*')
+
[beam] branch master updated: [BEAM-7467] Add dependency classifier to published pom (#8868)
This is an automated email from the ASF dual-hosted git repository.
mauzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 77456c1 [BEAM-7467] Add dependency classifier to published pom (#8868)
77456c1 is described below
commit 77456c1a1bc42b0ffce4e8eb9290fa9b4e5d718b
Author: Manu Zhang
AuthorDate: Tue Jun 18 12:40:30 2019 +0800
[BEAM-7467] Add dependency classifier to published pom (#8868)
---
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 9 +
1 file changed, 9 insertions(+)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 2bccfbe..699344f 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1127,17 +1127,26 @@ class BeamModulePlugin implements Plugin {
def generateDependenciesFromConfiguration = { param ->
project.configurations."${param.configuration}".allDependencies.each {
def dependencyNode =
dependenciesNode.appendNode('dependency')
+def appendClassifier = { dep ->
+ dep.artifacts.each { art ->
+if (art.hasProperty('classifier')) {
+ dependencyNode.appendNode('classifier',
art.classifier)
+}
+ }
+}
if (it instanceof ProjectDependency) {
dependencyNode.appendNode('groupId',
it.getDependencyProject().mavenGroupId)
dependencyNode.appendNode('artifactId',
it.getDependencyProject().archivesBaseName)
dependencyNode.appendNode('version', it.version)
dependencyNode.appendNode('scope', param.scope)
+ appendClassifier(it)
} else {
dependencyNode.appendNode('groupId', it.group)
dependencyNode.appendNode('artifactId', it.name)
dependencyNode.appendNode('version', it.version)
dependencyNode.appendNode('scope', param.scope)
+ appendClassifier(it)
}
// Start with any exclusions that were added via
configuration exclude rules.
[beam] 01/01: Merge pull request #8882 from apache/revert-8822-bigquery-rowcount
This is an automated email from the ASF dual-hosted git repository. anton pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 35f99ba92ddc5728e767e14a3e7d7df0a8bc4bb5 Merge: 3c61ab8 7b7c215 Author: Anton Kedin <33067037+ake...@users.noreply.github.com> AuthorDate: Mon Jun 17 20:50:31 2019 -0700 Merge pull request #8882 from apache/revert-8822-bigquery-rowcount Revert "[BEAM-7513] Adding Row Count for Bigquery Table" .../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +- .../beam/sdk/extensions/sql/BeamSqlTable.java | 7 - .../beam/sdk/extensions/sql/SqlTransform.java | 2 - .../sql/impl/BeamCalciteSchemaFactory.java | 12 +- .../sdk/extensions/sql/impl/BeamCalciteTable.java | 23 --- .../sql/impl/BeamRowCountStatistics.java | 44 -- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 26 +--- .../beam/sdk/extensions/sql/impl/JdbcDriver.java | 34 + .../sql/meta/provider/bigquery/BigQueryTable.java | 39 - .../sdk/extensions/sql/impl/BeamSqlEnvTest.java| 7 +- .../sdk/extensions/sql/impl/JdbcDriverTest.java| 31 ++-- .../extensions/sql/impl/parser/BeamDDLTest.java| 7 +- ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 7 +- .../meta/provider/bigquery/BigQueryRowCountIT.java | 161 - .../meta/provider/bigquery/BigQueryTestTable.java | 45 -- .../bigquery/BigQueryTestTableProvider.java| 71 - .../sql/meta/provider/pubsub/PubsubJsonIT.java | 2 +- .../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 23 --- 18 files changed, 29 insertions(+), 519 deletions(-)
[beam] branch master updated (3c61ab8 -> 35f99ba)
This is an automated email from the ASF dual-hosted git repository. anton pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3c61ab8 Merge pull request #8867 from tvalentyn/classifiers add 7b7c215 Revert "[BEAM-7513] Adding Row Count for Bigquery Table" new 35f99ba Merge pull request #8882 from apache/revert-8822-bigquery-rowcount The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +- .../beam/sdk/extensions/sql/BeamSqlTable.java | 7 - .../beam/sdk/extensions/sql/SqlTransform.java | 2 - .../sql/impl/BeamCalciteSchemaFactory.java | 12 +- .../sdk/extensions/sql/impl/BeamCalciteTable.java | 23 --- .../sql/impl/BeamRowCountStatistics.java | 44 -- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 26 +--- .../beam/sdk/extensions/sql/impl/JdbcDriver.java | 34 + .../sql/meta/provider/bigquery/BigQueryTable.java | 39 - .../sdk/extensions/sql/impl/BeamSqlEnvTest.java| 7 +- .../sdk/extensions/sql/impl/JdbcDriverTest.java| 31 ++-- .../extensions/sql/impl/parser/BeamDDLTest.java| 7 +- ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 7 +- .../meta/provider/bigquery/BigQueryRowCountIT.java | 161 - .../meta/provider/bigquery/BigQueryTestTable.java | 45 -- .../bigquery/BigQueryTestTableProvider.java| 71 - .../sql/meta/provider/pubsub/PubsubJsonIT.java | 2 +- .../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 23 --- 18 files changed, 29 insertions(+), 519 deletions(-) delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamRowCountStatistics.java delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTable.java delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
[beam] branch revert-8822-bigquery-rowcount created (now 7b7c215)
This is an automated email from the ASF dual-hosted git repository. anton pushed a change to branch revert-8822-bigquery-rowcount in repository https://gitbox.apache.org/repos/asf/beam.git. at 7b7c215 Revert "[BEAM-7513] Adding Row Count for Bigquery Table" This branch includes the following new commits: new 7b7c215 Revert "[BEAM-7513] Adding Row Count for Bigquery Table" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Revert "[BEAM-7513] Adding Row Count for Bigquery Table"
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch revert-8822-bigquery-rowcount
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 7b7c21544b50b1c8ac96f6c88362a0e266b6d55e
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Mon Jun 17 16:48:30 2019 -0700
Revert "[BEAM-7513] Adding Row Count for Bigquery Table"
---
.../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +-
.../beam/sdk/extensions/sql/BeamSqlTable.java | 7 -
.../beam/sdk/extensions/sql/SqlTransform.java | 2 -
.../sql/impl/BeamCalciteSchemaFactory.java | 12 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 23 ---
.../sql/impl/BeamRowCountStatistics.java | 44 --
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 26 +---
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 34 +
.../sql/meta/provider/bigquery/BigQueryTable.java | 39 -
.../sdk/extensions/sql/impl/BeamSqlEnvTest.java| 7 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java| 31 ++--
.../extensions/sql/impl/parser/BeamDDLTest.java| 7 +-
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 7 +-
.../meta/provider/bigquery/BigQueryRowCountIT.java | 161 -
.../meta/provider/bigquery/BigQueryTestTable.java | 45 --
.../bigquery/BigQueryTestTableProvider.java| 71 -
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 23 ---
18 files changed, 29 insertions(+), 519 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
index 8bdb1bf..5e44c6c 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
@@ -25,7 +25,6 @@ import
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
/** {@link BeamSqlCli} provides methods to execute Beam SQL with an
interactive client. */
@Experimental
@@ -35,17 +34,15 @@ public class BeamSqlCli {
private MetaStore metaStore;
public BeamSqlCli metaStore(MetaStore metaStore) {
-return metaStore(metaStore, false, PipelineOptionsFactory.create());
+return metaStore(metaStore, false);
}
- public BeamSqlCli metaStore(
- MetaStore metaStore, boolean autoLoadUdfUdaf, PipelineOptions
pipelineOptions) {
+ public BeamSqlCli metaStore(MetaStore metaStore, boolean autoLoadUdfUdaf) {
this.metaStore = metaStore;
BeamSqlEnv.BeamSqlEnvBuilder builder = BeamSqlEnv.builder(metaStore);
if (autoLoadUdfUdaf) {
builder.autoLoadUserDefinedFunctions();
}
-builder.setPipelineOptions(pipelineOptions);
this.env = builder.build();
return this;
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
index 63f7158..14f1b80 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
@@ -17,8 +17,6 @@
*/
package org.apache.beam.sdk.extensions.sql;
-import org.apache.beam.sdk.extensions.sql.impl.BeamRowCountStatistics;
-import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -38,9 +36,4 @@ public interface BeamSqlTable {
/** Get the schema info of the table. */
Schema getSchema();
-
- /** Estimates the number of rows or returns null if there is no estimation.
*/
- default BeamRowCountStatistics getRowCount(PipelineOptions options) {
-return BeamRowCountStatistics.UNKNOWN;
- }
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index afa4438..e45daca 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
@@ -118,8 +118,6 @@ public abstract class SqlTransform extends
PTransform>
sqlEnvBuilder.setQueryPlannerClassName(
input.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getPlannerName());
-
[beam] branch master updated: Add Python 3.6, 3.7 to the list of supported version classifiers.
This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e295aa8 Add Python 3.6, 3.7 to the list of supported version
classifiers.
new 3c61ab8 Merge pull request #8867 from tvalentyn/classifiers
e295aa8 is described below
commit e295aa89652c79be24958bf456b4cdb3736bebd1
Author: Valentyn Tymofieiev
AuthorDate: Fri Jun 14 17:17:34 2019 -0700
Add Python 3.6, 3.7 to the list of supported version classifiers.
---
.../src/main/python-release/python_release_automation.sh | 16 +++-
sdks/python/setup.py | 6 --
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/release/src/main/python-release/python_release_automation.sh
b/release/src/main/python-release/python_release_automation.sh
index 8b305ae..4c52762 100755
--- a/release/src/main/python-release/python_release_automation.sh
+++ b/release/src/main/python-release/python_release_automation.sh
@@ -19,12 +19,10 @@
source
release/src/main/python-release/run_release_candidate_python_quickstart.sh
source
release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
-run_release_candidate_python_quickstart"tar" "python2.7"
-run_release_candidate_python_mobile_gaming "tar" "python2.7"
-run_release_candidate_python_quickstart"wheel" "python2.7"
-run_release_candidate_python_mobile_gaming "wheel" "python2.7"
-
-run_release_candidate_python_quickstart"tar" "python3.5"
-run_release_candidate_python_mobile_gaming "tar" "python3.5"
-run_release_candidate_python_quickstart"wheel" "python3.5"
-run_release_candidate_python_mobile_gaming "wheel" "python3.5"
+for version in 2.7 3.5 3.6 3.7
+do
+ run_release_candidate_python_quickstart"tar" "python${version}"
+ run_release_candidate_python_mobile_gaming "tar" "python${version}"
+ run_release_candidate_python_quickstart"wheel" "python${version}"
+ run_release_candidate_python_mobile_gaming "wheel" "python${version}"
+done
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 6ef9b7f..fc54702 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -173,8 +173,8 @@ python_requires =
'>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'
if sys.version_info[0] == 3:
warnings.warn(
- 'Python 3 support for the Apache Beam SDK is not yet fully supported. '
- 'You may encounter buggy behavior or missing features.')
+ 'Some syntactic constructs of Python 3 are not yet fully supported by '
+ 'Apache Beam.')
setuptools.setup(
name=PACKAGE_NAME,
@@ -218,6 +218,8 @@ setuptools.setup(
'Operating System :: POSIX :: Linux',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
+'Programming Language :: Python :: 3.6',
+'Programming Language :: Python :: 3.7',
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
],
[beam] branch asf-site updated: Publishing website 2019/06/17 20:36:53 at commit 96934f2
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 1e30ff1 Publishing website 2019/06/17 20:36:53 at commit 96934f2 1e30ff1 is described below commit 1e30ff1f7c0549d83b42ccc9e990a190ed859e17 Author: jenkins AuthorDate: Mon Jun 17 20:36:53 2019 + Publishing website 2019/06/17 20:36:53 at commit 96934f2 --- .../documentation/execution-model/index.html | 1 + website/generated-content/documentation/index.html | 1 + .../io/built-in/google-bigquery/index.html | 1 + .../documentation/io/built-in/hadoop/index.html| 1 + .../documentation/io/built-in/hcatalog/index.html | 1 + .../documentation/io/built-in/index.html | 1 + .../documentation/io/developing-io-java/index.html | 1 + .../io/developing-io-overview/index.html | 1 + .../io/developing-io-python/index.html | 1 + .../documentation/io/testing/index.html| 1 + .../patterns/custom-io-patterns/index.html | 1 + .../patterns/file-processing-patterns/index.html | 1 + .../documentation/patterns/overview/index.html | 1 + .../patterns/pipeline-option-patterns/index.html | 1 + .../patterns/side-input-patterns/index.html| 1 + .../pipelines/create-your-pipeline/index.html | 1 + .../pipelines/design-your-pipeline/index.html | 1 + .../pipelines/test-your-pipeline/index.html| 1 + .../documentation/programming-guide/index.html | 1 + .../resources/learning-resources/index.html| 51 -- .../resources/videos-and-podcasts/index.html | 1 + .../documentation/runners/jstorm/index.html| 1 + .../aggregation/approximatequantiles/index.html| 1 + .../aggregation/approximateunique/index.html | 1 + .../python/aggregation/cogroupbykey/index.html | 1 + .../python/aggregation/combineglobally/index.html | 1 + .../aggregation/combinewithcontext/index.html | 1 + .../transforms/python/aggregation/count/index.html | 1 + .../python/aggregation/distinct/index.html | 1 + .../python/aggregation/groupbykey/index.html | 1 + .../python/aggregation/groupintobatches/index.html | 1 + .../python/aggregation/latest/index.html | 1 + .../transforms/python/aggregation/max/index.html | 1 + .../transforms/python/aggregation/mean/index.html | 1 + .../transforms/python/aggregation/min/index.html | 1 + .../python/aggregation/sample/index.html | 1 + .../transforms/python/aggregation/sum/index.html | 1 + .../transforms/python/aggregation/top/index.html | 1 + .../python/elementwise/filter/index.html | 1 + .../python/elementwise/flatmap/index.html | 1 + .../transforms/python/elementwise/keys/index.html | 1 + .../python/elementwise/kvswap/index.html | 1 + .../transforms/python/elementwise/map/index.html | 1 + .../transforms/python/elementwise/pardo/index.html | 1 + .../python/elementwise/partition/index.html| 1 + .../transforms/python/elementwise/regex/index.html | 1 + .../transforms/python/elementwise/reify/index.html | 1 + .../python/elementwise/tostring/index.html | 1 + .../python/elementwise/values/index.html | 1 + .../python/elementwise/withkeys/index.html | 1 + .../python/elementwise/withtimestamps/index.html | 1 + .../transforms/python/other/create/index.html | 1 + .../transforms/python/other/flatten/index.html | 1 + .../transforms/python/other/passert/index.html | 1 + .../transforms/python/other/reshuffle/index.html | 1 + .../transforms/python/other/view/index.html| 1 + .../transforms/python/other/windowinto/index.html | 1 + .../transforms/python/overview/index.html | 1 + 58 files changed, 104 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/execution-model/index.html b/website/generated-content/documentation/execution-model/index.html index 505e368..62a8cb4 100644 --- a/website/generated-content/documentation/execution-model/index.html +++ b/website/generated-content/documentation/execution-model/index.html @@ -346,6 +346,7 @@ Getting Started Articles Interactive Labs +Beam Katas Code Examples API Reference Feedback and Suggestions diff --git a/website/generated-content/documentation/index.html b/website/generated-content/documentation/index.html index 19dc34a..d4748e0 100644 --- a/website/generated-content/documentation/index.html +++ b/website/generated-content/documentation/index.html @@ -346,6 +346,7 @@ Getting Started Articles Interactive Labs +Beam Katas Code Examples API Reference Feedback and Suggestions diff --git
[beam] branch master updated: Merge pull request #8836: Add Beam Katas to the website's "Learning Resources" page
This is an automated email from the ASF dual-hosted git repository.
melap pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 96934f2 Merge pull request #8836: Add Beam Katas to the website's
"Learning Resources" page
96934f2 is described below
commit 96934f2a40a5754e14d9cc0f93f4c7464ea45c1a
Author: Henry Suryawirawan
AuthorDate: Tue Jun 18 04:35:08 2019 +0800
Merge pull request #8836: Add Beam Katas to the website's "Learning
Resources" page
---
.../src/_includes/section-menu/documentation.html | 1 +
.../documentation/resources/learning-resources.md | 27 ++
2 files changed, 28 insertions(+)
diff --git a/website/src/_includes/section-menu/documentation.html
b/website/src/_includes/section-menu/documentation.html
index 30ac573..8c7347a 100644
--- a/website/src/_includes/section-menu/documentation.html
+++ b/website/src/_includes/section-menu/documentation.html
@@ -190,6 +190,7 @@
Getting
Started
Articles
Interactive
Labs
+Beam Katas
Code
Examples
API
Reference
Feedback
and Suggestions
diff --git a/website/src/documentation/resources/learning-resources.md
b/website/src/documentation/resources/learning-resources.md
index 47959e3..1cba690 100644
--- a/website/src/documentation/resources/learning-resources.md
+++ b/website/src/documentation/resources/learning-resources.md
@@ -95,6 +95,33 @@ If you have additional material that you would like to see
here, please let us k
* **[NDVI from Landsat
Images](https://qwiklabs.com/focuses/1849?locale=en=catalog)** (45m) -
Process Landsat satellite data in a distributed environment to compute the
[Normalized Difference Vegetation
Index](https://en.wikipedia.org/wiki/Normalized_difference_vegetation_index)
(NDVI).
* **[Simulate historic
flights](https://qwiklabs.com/focuses/1159?locale=en=catalog)** (60m) -
Simulate real-time historic internal flights in the United States and store the
resulting simulated data in BigQuery.
+## Beam Katas {#beam-katas}
+
+Beam Katas are interactive Beam coding exercises (i.e. [code
katas](http://codekata.com/))
+that can help you to learn Apache Beam concepts and programming model hands-on.
+Built based on [JetBrains Educational
Products](https://www.jetbrains.com/education/), Beam Katas
+objective is to provide a series of structured hands-on learning experiences
for learners
+to understand about Apache Beam and its SDKs by solving exercises with
gradually increasing
+complexity. Beam Katas are available for both Java and Python SDKs.
+
+### Java
+
+* Download [IntelliJ
Edu](https://www.jetbrains.com/education/download/#section=idea)
+* Upon opening the IDE, expand the "Learn and Teach" menu, then select
"Browse Courses"
+* Search for "Beam Katas - Java"
+* Expand the "Advanced Settings" and modify the "Location" and "Jdk"
appropriately
+* Click "Join"
+* [Learn
more](https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Java#explore_course)
about how to use the Education product
+
+### Python
+
+* Download [PyCharm
Edu](https://www.jetbrains.com/education/download/#section=pycharm-edu)
+* Upon opening the IDE, expand the "Learn and Teach" menu, then select
"Browse Courses"
+* Search for "Beam Katas - Python"
+* Expand the "Advanced Settings" and modify the "Location" and "Interpreter"
appropriately
+* Click "Join"
+* [Learn
more](https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Python#explore_course)
about how to use the Education product
+
## Code Examples {#code-examples}
### Java
[beam] branch master updated (2de7d07 -> 7e633f3)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2de7d07 Merge pull request #8822 from riazela/bigquery-rowcount add 8f300f9 Fixes filesystem_test for Windows. new 7e633f3 Merge pull request #8869: [BEAM-7569] Fixes filesystem_test for Windows The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/io/filesystem_test.py | 16 1 file changed, 8 insertions(+), 8 deletions(-)
[beam] 01/01: Merge pull request #8869: [BEAM-7569] Fixes filesystem_test for Windows
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7e633f34e3a8ee68274d2752382fb6a52b5e9b4a Merge: 2de7d07 8f300f9 Author: Chamikara Jayalath AuthorDate: Mon Jun 17 12:42:25 2019 -0700 Merge pull request #8869: [BEAM-7569] Fixes filesystem_test for Windows sdks/python/apache_beam/io/filesystem_test.py | 16 1 file changed, 8 insertions(+), 8 deletions(-)
[beam] branch master updated: [BEAM-7513] Implements row estimation for BigQuery.
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bc2c286 [BEAM-7513] Implements row estimation for BigQuery.
new 2de7d07 Merge pull request #8822 from riazela/bigquery-rowcount
bc2c286 is described below
commit bc2c2863e400f28a4ac14ffb07b03cfe8a312a23
Author: Alireza Samadian
AuthorDate: Tue Jun 11 09:09:23 2019 -0700
[BEAM-7513] Implements row estimation for BigQuery.
---
.../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +-
.../beam/sdk/extensions/sql/BeamSqlTable.java | 7 +
.../beam/sdk/extensions/sql/SqlTransform.java | 2 +
.../sql/impl/BeamCalciteSchemaFactory.java | 12 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 23 +++
.../BeamRowCountStatistics.java} | 37 +++--
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 26 +++-
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 34 -
.../sql/meta/provider/bigquery/BigQueryTable.java | 39 +
.../sdk/extensions/sql/impl/BeamSqlEnvTest.java| 7 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java| 31 ++--
.../extensions/sql/impl/parser/BeamDDLTest.java| 7 +-
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 7 +-
.../meta/provider/bigquery/BigQueryRowCountIT.java | 161 +
.../meta/provider/bigquery/BigQueryTestTable.java | 45 ++
.../bigquery/BigQueryTestTableProvider.java| 71 +
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 23 +++
18 files changed, 496 insertions(+), 45 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
index 5e44c6c..8bdb1bf 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
@@ -25,6 +25,7 @@ import
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
/** {@link BeamSqlCli} provides methods to execute Beam SQL with an
interactive client. */
@Experimental
@@ -34,15 +35,17 @@ public class BeamSqlCli {
private MetaStore metaStore;
public BeamSqlCli metaStore(MetaStore metaStore) {
-return metaStore(metaStore, false);
+return metaStore(metaStore, false, PipelineOptionsFactory.create());
}
- public BeamSqlCli metaStore(MetaStore metaStore, boolean autoLoadUdfUdaf) {
+ public BeamSqlCli metaStore(
+ MetaStore metaStore, boolean autoLoadUdfUdaf, PipelineOptions
pipelineOptions) {
this.metaStore = metaStore;
BeamSqlEnv.BeamSqlEnvBuilder builder = BeamSqlEnv.builder(metaStore);
if (autoLoadUdfUdaf) {
builder.autoLoadUserDefinedFunctions();
}
+builder.setPipelineOptions(pipelineOptions);
this.env = builder.build();
return this;
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
index 14f1b80..63f7158 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql;
+import org.apache.beam.sdk.extensions.sql.impl.BeamRowCountStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -36,4 +38,9 @@ public interface BeamSqlTable {
/** Get the schema info of the table. */
Schema getSchema();
+
+ /** Estimates the number of rows or returns null if there is no estimation.
*/
+ default BeamRowCountStatistics getRowCount(PipelineOptions options) {
+return BeamRowCountStatistics.UNKNOWN;
+ }
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index e45daca..afa4438 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
@@ -118,6 +118,8 @@ public abstract class SqlTransform extends
[beam] branch master updated (e432b96 -> 36f4f85)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e432b96 [BEAM-6777] Let HealthzServlet respond actual health information of SDK harnesses (#8827) add 18100ca Use code.jquery.com for jQuery CDN add ff89b2e use https new 36f4f85 [website] Use code.jquery.com for jQuery CDN The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/_includes/head.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: [website] Use code.jquery.com for jQuery CDN
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 36f4f85e60427dc953e949626207dbd7ad17f0a2 Merge: e432b96 ff89b2e Author: Lukasz Cwik AuthorDate: Mon Jun 17 09:52:48 2019 -0700 [website] Use code.jquery.com for jQuery CDN website/src/_includes/head.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (89c5abf -> e432b96)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 89c5abf Merge pull request #8850: [BEAM-7144] Fix for rescaling problem on Flink >= 1.6 add e432b96 [BEAM-6777] Let HealthzServlet respond actual health information of SDK harnesses (#8827) No new revisions were added by this update. Summary of changes: .../runners/dataflow/worker/BatchDataflowWorker.java | 4 +++- .../dataflow/worker/SdkHarnessRegistries.java| 17 + .../runners/dataflow/worker/SdkHarnessRegistry.java | 3 +++ .../dataflow/worker/StreamingDataflowWorker.java | 4 +++- .../dataflow/worker/status/HealthzServlet.java | 17 + .../dataflow/worker/status/WorkerStatusPages.java| 10 ++ .../worker/status/WorkerStatusPagesTest.java | 20 +--- 7 files changed, 62 insertions(+), 13 deletions(-)
