[
https://issues.apache.org/jira/browse/BEAM-6566?focusedWorklogId=193183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193183
]
ASF GitHub Bot logged work on BEAM-6566:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Feb/19 02:41
Start Date: 01/Feb/19 02:41
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7700:
[BEAM-6566] Add IWYU plugin, activate for Beam SQL, fix errors
URL: https://github.com/apache/beam/pull/7700
Beam SQL was successfully compiling and passing tests even though it had a
totally straightforward compile-time dependency on commons-codec, which it did
not declare.
I had thought that Gradle made this issue a thing of the past, but it is not
so. This adds the capability that we got from `mvn dependency:analyze` plugin.
It found the known error and a couple other issues, which may or may not be
spurious.
This needs to be cherrypicked for 2.10.0 as Beam SQL was nonfunctional for
2.9.0. We can advise on a workaround to simply add the necessary dependency. We
also need an end-to-end test that will ensure our shipped artifact works for
users.
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [x] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 193183)
Time Spent: 10m
Remaining Estimate: 0h
> SqlTransform does not work for beam version above 2.6.0
> -------------------------------------------------------
>
> Key: BEAM-6566
> URL: https://issues.apache.org/jira/browse/BEAM-6566
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
> Affects Versions: 2.7.0, 2.8.0, 2.9.0
> Reporter: Xuefeng Zhang
> Assignee: Kenneth Knowles
> Priority: Critical
> Fix For: 2.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Issue*:
> Beam versions above 2.6.0 do not work for SqlTransform. By looking at the
> code, those versions use PCollection.getSchema, this function never works
> even for 2.6.0
> *Details:*
> Beam 2.6.0, class BeamPcollectionTable which is used by SqlTransform:
> public BeamPCollectionTable(PCollection<Row> upstream) \{ super(((RowCoder)
> upstream.getCoder()).getSchema()); this.upstream = upstream; }
> But for Beam 2.7.0 and 2.8.0, it is changed to :
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
> So that got below errors after upgrading beam version from 2.6.0 to 2.9.0
> java.lang.IllegalStateException: Cannot call getSchema when there is no schema
> at org.apache.beam.sdk.values.PCollection.getSchema(PCollection.java:328)
> at
> org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable.<init>(BeamPCollectionTable.java:34)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.toTableMap(SqlTransform.java:111)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:91)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
> *Codes:*
> Schema schema = Schema.builder()
> .addStringField("weightMarketValue")
> .addStringField("ticker")
> .addStringField("ratingLongTermFitchRaw")
> .build();
> Row row = Row.withSchema(schema)
> .addValues("weightMarketValue 1", "ticker 1", "ratingLongTermFitchRaw 1")
> .build();
> Version 1:
>
> {color:#FF0000}PCollection<Row> input = p.apply(Create.of(row){color}
> {color:#FF0000} .withSchema(schema, SerializableFunctions.identity(),
> SerializableFunctions.identity()){color}
> {color:#FF0000} .withCoder(RowCoder.of(schema)));{color}
> PCollection<Row> output = input.apply(SqlTransform.query("select * from
> PCOLLECTION"));
> Version 2:
> {color:#FF0000}PCollection<Row> input = p.apply(Create.of(row){color}
> {color:#FF0000}.withRowSchema(schema){color}
> {color:#FF0000}.withCoder(RowCoder.of(schema)));{color}
> PCollection<Row> output = input.apply(SqlTransform.query("select * from
> PCOLLECTION"));
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
