junrushao1994 commented on a change in pull request #6162:
URL: https://github.com/apache/incubator-tvm/pull/6162#discussion_r463897900
##########
File path: src/parser/tokenizer.h
##########
@@ -32,13 +33,25 @@
#include <unordered_map>
#include <vector>
+#include "./meta_ref.h"
#include "./token.h"
namespace tvm {
namespace parser {
using namespace runtime;
+// trim from start (in place)
+static inline void ltrim(std::string& s) { // NOLINT(*)
+ s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](int ch) { return
!std::isspace(ch); }));
Review comment:
Do you prefer to use IsWhitespace we defined below?
##########
File path: include/tvm/ir/span.h
##########
@@ -85,16 +85,26 @@ class SpanNode : public Object {
int line;
Review comment:
Yeah I think I agree with @ANSHUMAN87. Maybe its better to use
"begin_line", "begin_column", "end_line", "end_column"?
##########
File path: include/tvm/parser/source_map.h
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#ifndef TVM_PARSER_SOURCE_MAP_H_
+#define TVM_PARSER_SOURCE_MAP_H_
+/*!
+ * \file source_map.h
+ * \brief A map from source names to source code.
+ */
Review comment:
We should move these 4 lines above the include guard
##########
File path: include/tvm/parser/source_map.h
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#ifndef TVM_PARSER_SOURCE_MAP_H_
+#define TVM_PARSER_SOURCE_MAP_H_
+/*!
+ * \file source_map.h
+ * \brief A map from source names to source code.
+ */
+#include <tvm/ir/span.h>
+#include <tvm/runtime/packed_func.h>
+#include <tvm/runtime/registry.h>
+
+#include <fstream>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace parser {
+
+/*! \brief A program source in any language.
+ *
+ * Could represent the source from an ML framework or the internal
+ * source of a TVM program.
+ */
+struct Source {
Review comment:
if it is not like POD aggregation, I personally prefer to use class
instead
```suggestion
class Source {
public:
```
##########
File path: src/parser/diagnostic.h
##########
@@ -138,8 +61,77 @@ struct Diagnostic {
/*! \brief The diagnostic message. */
std::string message;
+ /*! \brief A diagnostic for a single character token. */
Diagnostic(int line, int column, const std::string& message)
- : level(DiagnosticLevel::Error), span(SourceName(), line, column),
message(message) {}
+ : level(DiagnosticLevel::Error),
+ span(SourceName(), line, column, line, column + 1),
+ message(message) {}
+
+ Diagnostic(DiagnosticLevel level, Span span, const std::string& message)
+ : level(level), span(span), message(message) {}
+};
+
+/*!
+ * \brief A wrapper around std::stringstream to build a diagnostic.
+ *
+ * \code
+ *
+ * void ReportError(const Error& err);
+ *
+ * void Test(int number) {
+ * // Use error reporter to construct an error.
+ * ReportError(ErrorBuilder() << "This is an error number=" << number);
+ * }
+ *
+ * \endcode
+ */
+struct DiagnosticBuilder {
Review comment:
yeah i think there is no much difference between struct and class except
for default access rules, but just a nitpick, people usually use struct for
POD/aggregation like those without methods, and in other cases class is
preferred.
##########
File path: include/tvm/relay/adt.h
##########
@@ -309,8 +309,9 @@ class Match : public Expr {
* \param data the input being deconstructed.
* \param clauses The clauses for matching.
* \param complete Indicate if this match is complete.
+ * \param span The span of the expression.
*/
- TVM_DLL Match(Expr data, tvm::Array<Clause> clauses, bool complete = true);
+ TVM_DLL Match(Expr data, tvm::Array<Clause> clauses, bool complete = true,
Span span = Span());
Review comment:
Maybe we should think a bit about `Span(nullptr)` vs `Optional<Span>`?
##########
File path: include/tvm/parser/source_map.h
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#ifndef TVM_PARSER_SOURCE_MAP_H_
+#define TVM_PARSER_SOURCE_MAP_H_
+/*!
+ * \file source_map.h
+ * \brief A map from source names to source code.
+ */
+#include <tvm/ir/span.h>
+#include <tvm/runtime/packed_func.h>
+#include <tvm/runtime/registry.h>
+
+#include <fstream>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace parser {
+
+/*! \brief A program source in any language.
+ *
+ * Could represent the source from an ML framework or the internal
+ * source of a TVM program.
+ */
+struct Source {
+ /*! \brief The raw source. */
+ std::string source;
+ /*! \brief A mapping of line breaks into the raw source. */
+ std::vector<std::pair<int, int>> line_map;
+
+ /*! \brief An empty source. */
+ Source() : source(), line_map() {}
+
+ /*! \brief Construct a source from a string. */
+ TVM_DLL explicit Source(const std::string& source);
+
+ TVM_DLL Source(const Source& source) : source(source.source),
line_map(source.line_map) {}
Review comment:
Shall we just use `= default`?
##########
File path: src/parser/diagnostic.h
##########
@@ -42,84 +43,6 @@
namespace tvm {
namespace parser {
-/*! \brief A program source in any language.
- *
- * Could represent the source from an ML framework or the internal
- * source of a TVM program.
- */
-struct Source {
- /*! \brief The raw source. */
- std::string source;
- /*! \brief A mapping of line breaks into the raw source. */
- std::vector<std::pair<int, int>> line_map;
-
- /*! \brief An empty source. */
- Source() : source(), line_map() {}
-
- /*! \brief Construct a source from a string. */
- explicit Source(const std::string& source) : source(source) {
- int index = 0;
- int length = 0;
- line_map.push_back({index, length});
- for (auto c : source) {
- if (c == '\n') {
- // Record the length of the line.
- line_map.back().second = length;
- // Bump past the newline.
- index += 1;
- // Record the start of the next line, and put placeholder for length.
- line_map.push_back({index, 0});
- // Reset length to zero.
- length = 0;
- } else {
- length += 1;
- index += 1;
- }
- }
- line_map.back().second = length;
- }
-
- Source(const Source& source) : source(source.source),
line_map(source.line_map) {}
-
- /*! \brief Generate an error message at a specific line and column with the
- * annotated message.
- *
- * The error is written directly to the `out` std::ostream.
- *
- * \param out The output ostream.
- * \param line The line at which to report a diagnostic.
- * \param line The column at which to report a diagnostic.
- * \param msg The message to attach.
- */
- void ReportAt(std::ostream& out, int line, int column, const std::string&
msg) const {
- CHECK(line - 1 <= static_cast<int64_t>(line_map.size()))
- << "requested line: " << (line - 1) << "line_map size: " <<
line_map.size()
- << "source: " << source;
-
- // Adjust for zero indexing, now have (line_start, line_length);
- auto range = line_map.at(line - 1);
- int line_start = range.first;
- int line_length = range.second;
- out << "file:" << line << ":" << column << ": parse error: " << msg <<
std::endl;
- out << " " << source.substr(line_start, line_length) << std::endl;
- out << " ";
- std::stringstream marker;
- for (int i = 1; i <= line_length; i++) {
- if (i == column) {
- marker << "^";
- } else if ((column - i) < 3) {
- marker << "~";
- } else if ((i - column) < 3) {
- marker << "~";
- } else {
- marker << " ";
- }
- }
- out << marker.str();
- out << std::endl;
- }
-};
-
/*! \brief The diagnostic level, controls the printing of the message. */
enum DiagnosticLevel {
Review comment:
Use enum class if possible
##########
File path: include/tvm/parser/source_map.h
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#ifndef TVM_PARSER_SOURCE_MAP_H_
+#define TVM_PARSER_SOURCE_MAP_H_
+/*!
+ * \file source_map.h
+ * \brief A map from source names to source code.
+ */
+#include <tvm/ir/span.h>
+#include <tvm/runtime/packed_func.h>
+#include <tvm/runtime/registry.h>
+
+#include <fstream>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace parser {
+
+/*! \brief A program source in any language.
+ *
+ * Could represent the source from an ML framework or the internal
+ * source of a TVM program.
+ */
+struct Source {
+ /*! \brief The raw source. */
+ std::string source;
+ /*! \brief A mapping of line breaks into the raw source. */
+ std::vector<std::pair<int, int>> line_map;
+
+ /*! \brief An empty source. */
+ Source() : source(), line_map() {}
+
+ /*! \brief Construct a source from a string. */
+ TVM_DLL explicit Source(const std::string& source);
+
+ TVM_DLL Source(const Source& source) : source(source.source),
line_map(source.line_map) {}
+
+ /*! \brief Generate an error message at a specific line and column with the
+ * annotated message.
+ *
+ * The error is written directly to the `out` std::ostream.
+ *
+ * \param out The output ostream.
+ * \param span The span to report the error at.
+ * \param msg The message to attach.
+ *
+ */
+ // TODO(@jroesch): replace the ostream with an interface for rendering
errors.
+ TVM_DLL void ReportAt(std::ostream& out, const Span& span, const
std::string& msg) const;
+};
+
+/*!
+ * \brief A mapping from a unique source name to source fragment.
+ */
+class SourceMap;
+/*!
+ * \brief Stores locations in frontend source that generated a node.
+ */
+class SourceMapNode : public Object {
+ public:
+ /*! \brief The source mapping. */
+ Map<SourceName, tvm::String> source_map;
Review comment:
nit: not much difference tho
```suggestion
Map<SourceName, tvm::runtime::String> source_map;
```
##########
File path: src/parser/token.h
##########
@@ -85,6 +86,9 @@ enum TokenType {
Extern,
Match,
PartialMatch,
+ Metadata,
Review comment:
Use enum class if possible. Also, I think in Google style people prefix
constants with a "k", e.g. kMetadata, kVersion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
